Introduction Model Examples and Discussion
Bayesian Modelling of a Human MMORPG Player Gabriel Synnaeve, Pierre Bessi`ere University of Grenoble / INRIA, CNRS
July 5, 2010
1/23
Introduction Model Examples and Discussion
1
Introduction MMORPG Existing NPC AI Bayesian Programming
2
Model Target Selection Skill Selection Identification and Questions
3
Examples and Discussion Example Setup Results Pespectives
2/23
Introduction Model Examples and Discussion
MMORPG Existing NPC AI Bayesian Programming
MMORPG: Massively Multiplayer Online Role Playing Game
3/23
Introduction Model Examples and Discussion
MMORPG Existing NPC AI Bayesian Programming
FSM in Quake III and Warcraft III ⇒ analysis and control are complex, no learning, either robust or good level HTN in Killzone 2 (Champandard 2009) ⇒ good for planning, not for reacting, no learning Behavior Trees (Isla 2005) in Halo(s) and Left4Dead ⇒ limited for collaboration without hierarchy, no learning Behavior Multi-queues (Cutumisu 2009) ⇒ real-time, parallel and collaborative behaviors, no learning ⇒ all as good as the programmer’s efforts put into their scripting
4/23
{
Description
Bayesian Program
Introduction Model Examples and Discussion
{
MMORPG Existing NPC AI Bayesian Programming
Variables Decomposition Parametric Forms
Identification Questions
P P(Searched|Known) = P(Searched|Known) =
Free
P(Searched ∧ Free ∧ Known) P(Known)
1X P(Searched ∧ Free ∧ Known) Z Free
5/23
Introduction Model Examples and Discussion
MMORPG Existing NPC AI Bayesian Programming
Modelling
Perceptions Infered/interm.
Actions
6/23
Introduction Model Examples and Discussion
MMORPG Existing NPC AI Bayesian Programming
The Idea
Learn from human observation Introduce clever composite variables (functionally composed of inter-dependant perception variables) Assume that looking only one time-step backwards will allow for some “sequence learning” (Bayesian filtering) ⇒ Reactive model
7/23
Introduction Model Examples and Discussion
Target Selection Skill Selection Identification and Questions
What do we want?
8/23
Introduction Model Examples and Discussion
Target Selection Skill Selection Identification and Questions
Target: T ∈ {t1 . . . tn } Hit Points: HP1 . . . HPn |HPi ∈ [0 . . . 9] Distance: D1 . . . Dn |Di ∈ {contact, close, far , veryfar } Delta Hit Points: ∆HP1 . . . ∆HPn |∆HPi ∈ {−, o, +} Imminent Death: ID1 . . . IDn |IDi ∈ {false, true} Class: C1 . . . Cn |Ci ∈ {tank, contact, ranged, heal}
9/23
Introduction Model Examples and Discussion
Target Selection Skill Selection Identification and Questions
Joint Distribution: P(T ∧ T t−1 ∧ HP1:n ∧ D1:n ∧ A1:n ∧ ∆HP1:n ∧ ID1:n ∧ C1:n ) = P(T t−1 ).P(T |T t−1 ) n Y × [P(HPi |Ai ∧ Ci ∧ T ) i=1
× P(Di |Ai ∧ T ) × P(Ai |T ) × P(∆HPi |Ai ∧ Ci ∧ T ) × P(IDi |T ) × P(Ci |Ai ∧ T )]
10/23
Introduction Model Examples and Discussion
Target Selection Skill Selection Identification and Questions
P(T t−1 ) is unknown and so unspecified (uniform) P(T |T t−1 ) prevents switching targets too often, can be learnt others P(Left|Right): learnt by counting or EM
11/23
Introduction Model Examples and Discussion
Target Selection Skill Selection Identification and Questions
Target: T ∈ {t1 . . . tn } Hit Points: HP1 . . . HPn |HPi ∈ [0 . . . 9] Distance: D1 . . . Dn |Di ∈ {contact, close, far , veryfar } Delta Hit Points: ∆HP1 . . . ∆HPn |∆HPi ∈ {−, o, +} Imminent Death: ID1 . . . IDn |IDi ∈ {false, true} Class: C1 . . . Cn |Ci ∈ {tank, contact, ranged, heal}
Resists: R1 . . . Rn |Ri ∈ {Nothing , Fire, Ice, Nature, FireIce, IceNat, FireNat, All} Skill: S ∈ {Skill1 . . . Skillm }
12/23
Introduction Model Examples and Discussion
Target Selection Skill Selection Identification and Questions
Joint Distribution: P(S ∧T ∧T t−1 ∧HP1:n ∧D1:n ∧A1:n ∧∆HP1:n ∧ID1:n ∧C1:n ∧R1:n ) = P(S).P(T ) n Y × [P(HPi |Ai ∧ Ci ∧ T ∧ S) i=1
× P(Di |Ai ∧ T ∧ S) × P(Ai |T ∧ S) × P(∆HPi |Ai ∧ Ci ∧ T ∧ S) × P(IDi |T ∧ S) × P(Ci |Ai ∧ T ∧ S) × P(Ri |Ci ∧ S ∧ T )]
13/23
Introduction Model Examples and Discussion
Target Selection Skill Selection Identification and Questions
P(T ) submodel call others P(Left|Right): learnt by counting or EM P(S) specify the prior of which spell/skill the player is more likely to cast/use
14/23
Introduction Model Examples and Discussion
Target Selection Skill Selection Identification and Questions
Identification
Not only perceived variables Some probability tables can be learnt by counting and averaging When using composite variables → Expectation Maximisation or “composing while learning”.
15/23
Introduction Model Examples and Discussion
Target Selection Skill Selection Identification and Questions
Questions
P(S ∧ T |hp1:n ∧ d1:n ∧ a1:n ∧ ∆hp1:n ∧id1:n ∧ c1:n ∧ r1:n ) will be computed by: X
P(S|T ∧ hp1:n ∧ d1:n ∧ a1:n ∧ ∆hp1:n ∧ id1:n ∧ c1:n ∧ r1:n )
T
and the subquestion: P(T |hp1:n ∧ d1:n ∧ a1:n ∧ ∆hp1:n ∧ id1:n ∧ c1:n )
16/23
Introduction Model Examples and Discussion
Example Setup Results Pespectives
2 Setups: with and without “dying add”
Lich
Add
Lich
Add
MT
Tank
MT
Tank
Priest
Druid
Mage
Priest
Druid
Mage
17/23
Introduction Model Examples and Discussion
Example Setup Results Pespectives
Target distribution with and without dying add
18/23
Introduction Model Examples and Discussion
Example Setup Results Pespectives
Skills distribution with and without dying add
19/23
Introduction Model Examples and Discussion
Example Setup Results Pespectives
Skills and Target distribution without dying add
with P(ID|Target) = 0.9, P(A|Target) = 0.6
20/23
Introduction Model Examples and Discussion
Example Setup Results Pespectives
complete the model for other tasks scale with high number of values for T and S → we propose clusterizing skills by type (intermediate variable) apply this model to others for prediction learn the behavior from human players
21/23
Introduction Model Examples and Discussion
Example Setup Results Pespectives
Bibliography The AI for Killzone 2’s multiplayer (2009) [Champandard A. & al.] An Architecture for Game Behavior AI: Behavior Multi-Queues (2009) [Cutumisu M. & Szafron D.] Bayesian Programming and Hierarchical Learning in Robotics (2000) [Diard J. & Lebeltel O.] Handling Complexity in the Halo 2 AI (2005) [Isla D.] It knows what you’re going to do (2001) [Laird J.E.] Bayesian Robot Programming (2004) [Lebeltel O. & al.] Teaching Bayesian Behaviours to Video Game Characters (2004) [Le Hy R. & al.] Probability Theory: The Logic of Science (2003) [Jaynes E.T.]
22/23
Introduction Model Examples and Discussion
Example Setup Results Pespectives
Thanks
Thank you for your attention.
23/23
