USOO7636701 B2

(12) United States Patent (10) Patent No.: US 7.636,701 B2

Funge et al. (45) Date of Patent: Dec. 22, 2009
(54) QUERY CONTROLLED BEHAVIOR MODELS 6.424,356 B2 * 7/2002 Chang et al. ................ 715/701
AS COMPONENTS OF INTELLIGENT 6.425,582 B1 7, 2002 Rosi
AGENTS 6,427,063 B1 7/2002 Cook et al.
(75) Inventors: John Funge, Sunnyvale, CA (US); Ron 6,561,811 B2 5/2003 Rapoza et al.
Musick, Belmont, CA (US); Daniel 6,709,335 B2 * 3/2004 Bates et al. ................... 463/42
Dobson, Atherton, CA (US); Nigel 6,892,349 B2 5/2005 Shizuka et al.
Duffy, San Francisco, CA (US); Michael 7,054,928 B2 5/2006 Segan et al.
McNally, San Jose, CA (US); Xiaoyuan
Tu, Sunnyvale, CA (US); Ian Wright,
Mountain View, CA (US); Wei Yen, Los
Altos Hills, CA (US); Brian Cabral, San (Continued)
J CA (US
ose, CA (US) FOREIGN PATENT DOCUMENTS
(73) Assignee: AiLive, Inc., Mountain View, CA (US) WO 2006014560 A2 2/2006
(*) Notice: Subject to any disclaimer, the term of this
patent is extended or adjusted under 35
U.S.C. 154(b) bV 146 davs.
(b) by ayS Primary Examiner Wilbert L Starks, Jr.
(21) Appl. No.: 11/929, 170 (74) Attorney, Agent, or Firm—Joe Zheng
(22) Filed: Oct. 30, 2007 (57) ABSTRACT
(65) Prior Publication Data
US 2008/OO97948A1 Apr. 24, 2008 Providing dynamic learning for software agents in a simula
pr. Z4, tion is described. The software agents with learners are
Related U.S. Application Data capable of learning from examples. When a non-player char
acter queries the learner, it can provide a next action similar to
(63) Continuation of application No. 10/885,495, filed on a player character. A game designer provides program code,
Jul. 6, 2004, now Pat. No. 7,296,007. from which compile-time steps determine a set of raw fea
(51) Int. Cl tures. The code may identify a function (like computing dis
Goon s/00 (2006.01) tances). At compile-time steps, determining these raw fea
tures in response to a scripting language, so the designer can
. F.c c - - - - - ificati- - - - - -s - - - - - - - h- - - - - - - - - 7647. specify which code should be referenced. A set of derived
(58) Field of Classification Search ................... 706747 features, responsive to the raw features, may be relatively
S lication file f 1 hhi simple, more complex, or determined in response to a learner.
ee application file for complete search history. The set of such raw and derived features form a context for a
(56) References Cited learner. Learners might be responsive to (more basic) learn
ers, to results of State machines, to calculated derived fea
U.S. PATENT DOCUMENTS tures, or to raw features. The learner includes a machine
6,072,466 A 6, 2000 Shah et al. learning technique.
6,152,821 A * 1 1/2000 Nakagawa et al. ............. 463/4
6,201.948 B1 3, 2001 Cook et al. 17 Claims, 3 Drawing Sheets

Response (e.g., action) s Contexts

a? 200S contextN learning
training signal element(s)

OOO
 US 7,636,701 B2
Page 2

U.S. PATENT DOCUMENTS 2008/0097.948 A1 4/2008 Funge et al.

7,296,007 B1 1 1/2007 Funge et al.
2007/0260567 A1 1 1/2007 Funge et al. * cited by examiner
U.S. Patent Dec. 22, 2009 Sheet 1 of 3 US 7.636,701 B2

1
D
|

y
09
U.S. Patent Dec. 22, 2009 Sheet 2 of 3 US 7.636,701 B2

150
Re
Raw Features
-160
Derived Features

Relative motion
Location of agent and
of agent obstacleslopposition

Speed and direction

of agent

Location Impact of weather

of goal on agent's motion

Value or rank
of goal

impact of weather on
Weather conditions obstacleslopposition

Location of
obstacleslopposition

Impact of weather on
Nature of obstacleslopposition
obstacleslopposition

Speed and direction of

obstacleslopposition Contexts 200
NY
O
O
O 170
185 180
220- > Leraning
What should Queries Decision Predictions Element(s)/
next action be? Model(s) Behavior
Model(s)
230
>
Responses
Next action

NFF/
Game Code
2-120
U.S. Patent Dec. 22, 2009 Sheet 3 of 3 US 7.636,701 B2

301

3OO
302
N Run/Continue
Game
307
303 Update Behavior
Model(s)
NO Use Pre-Load
Model(s)? 3O8
Receive Queries
YES
304
Load 309

305 310
NO NPC Uses
Make Decisions
Learner?

311
306
Couple to Generate Responses
Learner
312

End Game?

YES
C End d 313
 US 7,636,701 B2
1. 2
QUERY CONTROLLED BEHAVIOR MODELS playing style of the player characters generally, or may be
AS COMPONENTS OF INTELLIGENT distinct, such as when the non-player characters are distrib
AGENTS uted into teams, each team of which learns the playing style of
a specific player character during a specific time frame.
CROSS-REFERENCE TO RELATED PATENT 5 For a first example, if non-player characters could learn
APPLICATION from a particular player character, that player may see their
tactics and strategies, and possibly their personality, repli
This application is a continuation of prior U.S. patent cated and possibly amplified by a set of non-player characters
application Ser. No. 10/885,495, titled “Real Time Context (whether friendly non-player characters or enemy non-player
Learning by Software Agents, filed Jul. 6, 2004 now U.S. 10 characters). For a second example, if distinct teams of non
Pat. No. 7,296,007, by John Funge, et al., which is hereby player characters could learn from multiple distinct player
incorporated by reference. characters, those distinct player characters may each develop
BACKGROUND OF THE INVENTION
their own team of non-player characters, each team with its
own particularized behavior and each team with its own lead
15 ership and cooperative strategies, in response to the ability or
1. Field of the Invention initiative of those player characters in training or re-training
The invention relates to context learning by software their respective teams.
agents, for example, it may be used by embedded game agents In a first set of known systems, non-player characters have
in a gaming System during play of the game. a set of pre-programmed behavior, which they follow no
2. Related Art 2O
In a computer simulation, especially in a computer game matter who the player is, how the player acts, or how the game
that attempts to present a fantasy world (i.e., to create and progresses. (1) The set of possible actions can be pre-pro
simulate a variant world in which there are at least some grammed by the game designer, with a known set of actions to
fantasy or fictional aspects), there may be numerous agents be performed in known possible circumstances. This is some
within the game, each of which has independent actions it can 25 times called a “script.” (2) More generally, a script may
perform. One set of those agents is controlled by players include random or pseudorandom selection of actions, in
(herein sometimes called “player characters, and herein response to parameters selected by the game designer. (3)
More generally, the parameters selected by the game designer
sometimes called “human users”), while other sets of those can be adjusted to cause the global apparent behavior of the
agents are controlled by the game software (herein sometimes non-player character to change with time.
called “non-player characters'). It is also possible that the 30
selection of which of those agents is player characters and the While these known methods generally achieve the goal of
selection of which of those agents are non-player characters providing non-player characters with actions to take in each
may change from time to time as the game progresses, per context expressed within the game, they have drawbacks. In
haps as the player chooses to control different characters. general, the non-player characters do not learn, in real-time,
One problem with such systems is that it is difficult to make 35 to behave like a player character, and with the result that they
non-player characters learn about the game in real-time (that typically remainless like player characters and with the effect
is, as it is played) or in response to an exemplar Such as a that they typically remain less imaginative, intelligent, real
player character (that is, in response to choices made by a istic, and robust than nearly all player characters. Lack of
human user). There are several possible sources of potential ability to learn has the effect that non-player characters are
learning, including for example, from current player charac- 40 clearly inferior to, and do not share any salient characteristics
ters, from records of play of player characters, from the non of the personality of any individual user.
player characters’ own experiences, and from records of play In a second set of systems, known in some academic
of other non-player characters. It would be advantageous if circles, a non-player character may maintain a model of how
non-player characters may become more like the player char to behave (sometimes herein called a “character model of
acters in their behavior, as this would lead to a richer and 45 behavior” or a “behavior model), and attempt to update that
possibly more challenging game environment. For a first model in response to aspects of the game environment. (In
example, if player characters are imaginative, intelligent, this context, a “model” refers to a set of internal state indica
realistic, or robust, it would be advantageous for non-player tive of information or techniques possibly learned by the
characters to exhibit at least some of these qualities, or to non-player character, for example, if the non-player character
exhibit these qualities to at least Some of the degree that player 50 were using a neural network to learn behavior, that model
characters do. For a second example, if player characters would include the connectivity and weights assigned to nodes
exhibit one or more elements of oddity or quirkiness, it would and edges in that neural network, as well as the particular
be advantageous for non-player characters to exhibit at least topology of that neural network, and methods of using the
some of those elements, or similar elements of oddity or neural network to generate responses to queries.) For
quirkiness, or other aspects of the players personality, as 55 example, a non-player character may update that model in
expressed through those players playing style. response to actions taken by a player character (for the pur
It would also be advantageous if non-player characters pose attempting to imitate that player character), or may
could learn in real time (that is, could modify their behavior update that model in response to experiences in the game
and their response to various stimuli during actual play) from environment (for the purpose of attempting to learn from its
at least some of these possible sources of potential learning, 60 mistakes).
including learning from player characters, from other non These known systems have two substantial drawbacks if
player characters, or from their interaction with the game they are to be used at run-time with known game systems: (1)
environment. More specifically, it would also be advanta they consume far too much in the way of computing and
geous if each distinct player character can “teach' (for memory resources, in response to the very large number of
example, serve to model behavior for) its own set of non- 65 possible contexts, and (2) models being learned in real-time
player characters. Those sets of non-player characters may are typically not sufficiently robust to be created and used in
overlap. Such as when the non-player characters learn the the course of actual game play.
 US 7,636,701 B2
3 4
A further drawback of known methods is that they gener programming, multidimensional clustering, neural networks,
ally involve substantial knowledge about both (1) machine and weighted majority techniques. In addition, the learner
learning techniques, and (2) sets of derived features likely to may include a weighted average of a set of relatively simpler
be valuable for machine learning, each of which has the elements, updated in real time during actual game play using
effect, and in combination which have the effect, of involving an exponential gradient technique, or some other machine
Substantial effort, time, and possibly monetary resources for learning technique.
the game designer to implement. It would be advantageous to In one embodiment, there are one or more learners associ
provide a system in which non-player characters, or other ated with actions to be taken by selected software agents. For
actors, can learn from player characters, without requiring example, for a non-player characterina computer game, there
Substantial investment of effort, money, or time by the game 10 may be separate learners for direction of movement and style
designer. of movement, and a separate learner for combat actions.
Accordingly, it would be advantageous to provide a Similarly, in a computer game in which Such movement is
method and system not subject to drawbacks of known sys possible, there may be a separate learner for direction of
temS. movement, and a separate learner for hopping,jumping, som
15 ersaulting, or otherwise indulging in acrobatics. Where
SUMMARY OF THE INVENTION appropriate, multiple actions may be separated into control by
different learners, or where alternatively appropriate, mul
The invention provides a method and system capable of tiple actions may be consolidated into control by one learner.
providing dynamic learning for software agents in a simula After reading this application, those skilled in the art would
tion, Such as non-player characters in a computer game. The recognize that the systems described herein provide an
method and system provide Software agents with one or more enabling technology, with the effect that heretofore advanta
learners, that is, Software elements capable of learning from geous features can be provided that heretofore were substan
examples, capable of receiving requests from non-player tially infeasible.
characters and capable of responding to those requests with
responses from which the non-player characters can readily 25 BRIEF DESCRIPTION OF THE FIGURES
derive one or more actions to take. In one embodiment, the
learner gleans behavior from actions taken in context by a FIG. 1 shows a block diagram of a representation of a game
player character. This has the effect that when a non-player system including context learning by Software agents;
character queries that learner for a next action, the learner can FIG. 2 shows a block diagram of an example of context
provide a next action similar to one that may be taken by the 30 learning by Software agents; and
player character (if the player character were directing the FIG.3 shows a process flow diagram of a method of oper
actions of that non-player character). ating in a System for context learning by Software agents.
In one embodiment, the invention includes a method and
system for adding a set of learners to the game structure, and DETAILED DESCRIPTION OF THE PREFERRED
a computer game with those learners added (that is, a method 35 EMBODIMENT
of playing a computer game, and a system for playing that
computer game). Preferred embodiments of the invention are described
In one embodiment, the method and system include a set of herein, including preferred device coupling, device function
derived features, responsive to raw features. Raw features ality, and process steps. After reading this application, those
include raw data that indicate information about a game envi 40 skilled in theart would realize that embodiments of the inven
ronment, for example locations of game objects, distances tion can be implemented using a variety of other techniques
between game objects, a direction of a most valuable visible not specifically described herein, without undue experimen
object, measures of wounded status of player characters and tation or further invention, and that Such other techniques
non-player characters, measures of damage to non-living would be within the scope and spirit of the invention.
objects such as walls, and the like. Derived features may be 45
relatively simply related to raw features, such as an indicator Definitions
of a direction to move to get to a nearest enemy or treasure, or
a measure of a distance thereto. Derived features may be more The general meaning of each of these terms is intended to
complex, for example: (1) an expert System in which at least be illustrative and in no way limiting.
Some state information is retained from time to time, or (2) a 50 The phrase 'game system' generally describes any com
set of Scripted instructions, or (3) a state machine operated in puting device or other electronic system on which games
response to other raw features or derived features. Derived may be played. Game systems include those units that
features may be still more complex, Such as those determined are standalone consoles, portable devices, and general
in response to a query to, and a response from, a learner. The purpose computers capable of supporting game Software
set of such raw features and derived features, in combination, 55 (for example laptop and desktop computers). Known
collectively form a context to which a learner is responsive. examples of standalone consoles include the Sony
This has the effect that learners may be responsive to (more “PlayStation 2, the Nintendo “Game Cube,” and the
basic) learners, or to a set of results of (expert) state machines, Microsoft X-Box.’
or to a set of calculated derived features, or to a set of raw The phrase “computer game' or 'game' generally
features, or to some combination or conjunction thereof. 60 describes Software or instructions, or rules describing an
In one embodiment, the learner includes one or more interactive system, performed by a computing device or
machine learning techniques, including for example, decision other electronic system and intended for use by a user.
trees, hidden Markov models, Bayesian learning techniques, Computer games include character “fighting games,
and other alternatives. Several machine learning techniques hand-eye coordination or 'shooter' games, mazes and
are known in the arts of artificial intelligence and machine 65 puzzles, problem-solving games, racing or piloting
learning. Among the many alternatives include techniques games, role-playing games, simulation games, and other
related to evolution strategies, genetic algorithms, genetic games. In the context of the invention, there is no par
 US 7,636,701 B2
5 6
ticular requirement that computer games are designed to within the game environment), “learning.” “learning by
be played by children (games with adult-oriented con example.” “learning from a model.” “machine learning.”
tent are known), or that computer games are designed to and “real-time learning generally describe alteration of
be played for fun (games with educational intent are parameters of a behavior model, with the effect that the
known), or that computer games have a “winning or 5 software agent controlled by that behavior model exhib
“losing state (games with simulation effect, for its different behavior in the future compared with the
example, “Sim City 3000, are known). This concept is past. In one embodiment, dynamic learning is performed
intended to broadly encompass any system in which a by non-player characters in response to exemplary
user interacts with a computing device, or with other actions performed by player characters.
users, according to rules (typically described and imple 10 The phrases “behavior strategies.” “cooperation.” “leader
mented by the software or instructions), even if those ship,” “modeling behavior,” “training and “retraining
rules are be changed. generally describe types of exemplary behavior that may
The phrases “game actors.” “game agents.” “software be presented, for example by a player character.
agents, and “game characters' (including both player” The phrases “player,” or “user generally describe aperson
characters and “non-player characters) generally 15 (or a team of persons) controlling a player character.
describe active software within the computer game, The scope and spirit of the invention is not limited to any of
capable of making decisions (to anthropomorphize a bit) these definitions, or to specific examples mentioned therein,
in response to contexts in which that active Software but is intended to include the most general concepts embodied
responds. In the case of game characters, a player char by these and other terms.
acter would be primarily controlled by a user of the game System Elements
(although there may be times when the game limits FIG. 1 shows a block diagram of a representation of a game
decisions for that character or imposes decisions on that system including context learning by Software agents.
character) and a non-player character would be prima A game 100 on a game system 105 includes a game engine
rily controlled by the game itself (although there may be
times when the game allows a user to take all or partial 25 110, which in turn includes game code 120, game-state 130,
control for that character). and one or more software agents 140.
The phrases 'game environment,” “game experiences.” The game code 120 includes a set of software processes for
'game play.” “game-state' or “state information, and controlling the presentation of a computer game. Game code
“simulation' generally describe the state of the game designers120 typically is provided or coded by one or more game
system and the computer game, including the raw fea 30 and programmers.
tures, derived features, and contexts to which software The game code preferably includes program structures
agents respond. Such as data structures, data types, function entry points,
The phrases “machine learning.” “artificial intelligence.” function pointers, object data types, object methods, and the
and the like interchangeably refer to processes and tech like. Some, all, none or different structure may be present
niques by which computers or other devices emulate 35 depending on the way in which game 100 is designed.
learning and learned behavior, for example to emulate or Although the game code 120 is shown in FIG. 1 as a
to predict actions of humans or other actors. (Techni separate box, the other elements of the game engine 110
cally, machine learning is a Subset of artificial intelli typically are formed from portions of game code 120.
gence; however, the terms are generally used in a The game-state 130 includes a set of state information,
broader and more interchange sense in this application.) 40 preferably used during operation of the game engine 110.
The phrases “context,” “actions taken in context gener Although the game-state 130 is shown in FIG. 1 as a separate
ally describe a set of features, which commonly taken box, the game-state can encompass variables and data struc
together, indicate a state of the game system. A context tures used by the other element of the game engine 110.
can be generically thought of as a point (or a set of The one or more software agents 140 collectively include
points, for example, a cluster) selected from R", where n 45 one or more player characters 141 and non-player characters
is the number of possible features and R represents the 142. Preferably, a player character 141 includes a video object
set of real numbers; that is, each context describes a set controlled by a human user 143 (or a team of such human
of possible values for all variables available in the game users 143). For one example, not intended to be limiting in
State. any way, “Lara Croft' (of the “Tomb Raider Series) and
The phrase “decision model” generally describes any 50 “Jak' (of the “Jak and Daxter' game) are examples of player
instructions, model, or technique for receiving queries characters 141. Preferably, a non-player character 142
and providing responses (in context). For one example, includes a video object controlled by the system running the
not intended to be limiting in any way, a decision model application software. For one example, not intended to be
may include a technique for inference in a belief net limiting in any way, the four roaming ghosts ("Blinky.”
work. 55 “Pinky.” “Inky,” and “Clyde') in the classic “PacMan” game
The phrase “behavior model generally describes any are examples of non-player characters 142. In some embodi
information or other representation for describing ments, it is possible to use game controllers or other input
behavior (in context) of an exemplar actor. For one devices to affect the actions of at least some of the non-player
example, not intended to be limiting in any way, a behav characters 142, for example, the second “Ice Climbers' char
ior model may include a belief network. 60 acter in “Super Smash Brothers Melee' (for the Nintendo
The phrases 'game designer” and “game software gener GameCube).
ally describe, respectively, the entity providing the origi The program code for the game engine 110 includes ele
nal game software and its concepts, and the original ments capable of specifying relevant information to other
Software capable of being compiled and executed as the elements in the game 100.
game System or computer game. 65 For a first example, not intended to be limiting in any way,
The phrases “behavior modification.” “dynamic learning.” game engine 110 may include program structures having
“learned behavior (for example, responses to stimuli information relating to raw features 150.
 US 7,636,701 B2
7 8
The raw features 150 indicate important information about determining) a useful property of the game-state 130.
the game 100, for example, a location of a game objects, a For one example, not intended to be limiting in any way,
point value of a game object, a measure of a wounded status a raw feature 150 includes a function “is Visible (p1,
of a player character 141 or non-player character 142, and the p2), identified by a callback pointer or entry point,
like. In Such cases, the program structures for raw features which provides a value describing whether player p2 is
150 may include C language code (or C++ language code, or visible to player p1 (in response to game-state 130 for
code in another language) describing those data structures example, whether player p2 has an invisibility cloak,
and how they can be referenced (for example a word offset player p1 is temporarily blinded, or can “see invisible.”
within a data structure). in addition to considerations of line-of-sight, and the
For a second example, not intended to be limiting in any 10 like). While it is possible in many cases for a raw feature
way, game engine 110 may include references to object meth 150 of this type to be determined by the game engine 110
ods capable of computing raw features 150, for example, a as a derived feature 160, it may be advantageous for the
distance between game objects, a direction of the most valu game designer to provide that descriptor or function as a
able object visible, and the like. In such cases, the program raw feature 150, for example, to take advantage of spe
structures for raw features 150 may include C language code 15 cial knowledge of the game-state 130 or particular struc
(or C++ language code, or code in another language) describ ture of the data structures used with that game-state 130.
ing locations of those object methods or instructions for per As noted herein, there may be a very great many raw
forming them, and how they can be referenced (for example a features 150, for example, a location of a game objects, a
function-call address). point value of a game object, a measure of a wounded status
For example, not intended to be limiting in any way, a raw of a player character 141 or non-player character 142, and the
feature 150 may include one or more of the following data: like.
A raw feature 150 may include a data structure, for The derived features 160 include a set offeatures derived in
example from game-state 130, used by the game; that is, response to the raw features 150, for example, by calculating
used by the simulation to determine what happens in the the values of those derived features 160 using the values of the
context of the game 100. In one embodiment, the actual 25 raw features 150. A derived feature 160 may include a
data structures used by the game are exposed to the weighted sum of a set of raw features 150, a function calcu
learner 165, so that those data structures may be directly lation in response to a set of raw features 150, and the like. For
read from and written into, without any special require one example, not intended to be limiting in any way, a derived
ment for a data marshalling step (or similar step) or a feature 160 may include a “best path' from a non-player
buffer (or similar data structure) in which data is main 30 character 142 to a target object, with the distance and direc
tained after reading from that data structure or before tion for that best path pre-computed for each point thereon.
writing into that data structure. For one example, not A derived feature 160 may be determined using one or
intended to be limiting in any way, there may be a data more predefined calculations or operations performed on one
structure “playerInfo' including fields for that particular or more raw features, raw features and derived features, or
player's position, Velocity, allegiance, power, health, 35 both raw features and derived features. Some of the following
whether that player is a player character or a non-player examples are based on Such pre-defined calculations or
character, and the like. operations.
A raw feature 150 may include a list or an array of indi A derived feature 160 may also or instead be determined
vidual data structures, for example from game-state 130, using one or more learned or derived calculations or opera
used by the game; that is, used by the simulation to 40 tions performed on one or more raw features, raw features and
determine what happens in the context of the game 100. derived features, or both raw features and derived features.
For one such example, not intended to be limiting in any Some of the following examples are based on such predefined
way, there may be an array of data structures playerInfo calculations or operations.
II including fields similar to those described above, and For example, not intended to be limiting in any way, a
for a plurality of players, whether player characters or 45 derived feature 160 may include one or more of the following
non-player characters. The structure of the raw feature data:
150 as a list or an array has an advantage in that related A derived feature 160 may include a Boolean value or a
information can be determined about the game-state 130 predicate operator, Such as a determination whether
for one or more functions serving as inputs to derived Some condition is true at the moment with regard to the
features 160 (described below) or to one or more learn 50 game 100. For one such example, not intended to be
ers 165 (described below). For one such example, not limiting in any way, there may be a predicate “p1->is
intended to be limiting in any way, an array of Such data Healthy” including a determination of whether a par
structures allows determination of information about ticular character p1 is considered healthy or not. In one
player-to-player relationships, for example (for each embodiment, this particular derived feature 160 may be
player) which other player is the nearest enemy or 55 defined using the assignment “p1->is Healthy=IF
friend, and the like. (healthd80% of maxPossibleHealth), where “health'
A raw feature 150 may include a list or an array of indi and “maxPossibleHealth are raw features 150. Similar
vidual data structures, the individual data structures derived features 160 may be constructed for concepts
being similar to those described above, with the differ such as “islfarAway,” “is Friend, and “is Enemy.”
ence that the array relates to a set of objects other than 60 A derived feature 160 may include a Boolean value or a
players, such as an array of obstacles, an array of power predicate operator defined in response to already-known
ups, and the like. For one such example, an array “near predicates. For one such example, not intended to be
est-BlueTarget may include an identifier, location or limiting in any way, there may be a predicate “p1->is
pointer to another character (either player character or Healthy Friend' defined using the assignment “p1->is
non-player character) who is the nearest blue target. 65 HealthyFriend=IF (p1->is Healthy AND p1->is
A raw feature 150 may include a pointer to a descriptor or Friend), where is Healthy and is Friend are themselves
a function with the property of computing (or otherwise derived features 150.
 US 7,636,701 B2
10
A derived feature 160 may include an iterator defined in Third, this has the effect of allowing the game designer to
response to an array of already-known raw features 150 not have to think about any learning aspects of the game
or derived features 160. For one such example, not simulation, and not having to design any code or data
intended to be limiting in any way, there may be an array structures therefore.
playerInfo II' as described above, including informa Fourth, this has the effect of allowing the game designer to
tion relating to each player, and there may be an iterator call upon the learner 165 provided by the invention to
“bluePlayerInfo II defined as the subset of all players in ask for actions to be taken by non player characters (or
the array playerInfo II also meeting the predicate opera any other game element that may learn, for example, an
tor is Blue. Similarly, using the iterator “bluePlayerInfo intelligent device).
another derived feature 160 such as “friendlyBlue 10 After reading this application, other and further advantages
PlayerInfo II' may be defined as the subset of all players of providing a selected set of derived features will be clear to
in the array bluePlayerInfo II also meeting the predicate those skilled in the art. Moreover, these other and further
operator is Friendly. Alternatively, the derived feature advantages will be clear to be within the scope and spirit of the
160 such as “friendlyBluePlayerInfo II' may be defined invention, and not to involve any undue experimentation or
15 further invention.
in response to only raw features 150 as the subset of the Learner 165 is capable of providing dynamic learning for
array playerInfo meeting the predicate operator (is
Blue AND is Friendly). software agents in game 100. Learner 165 is capable of learn
ing from examples, capable of receiving requests from non
A derived feature 160 may include an element of an itera player characters and capable of responding to those requests
tive operator by selecting one or more objects from that with responses from which the non-player characters can
iterative operator. For one Such example, not intended to readily derive one or more actions to take. In one embodi
be limiting in any way, there may be an iterative operator ment, the learner 165 gleans behavior from actions taken in
“bluePlayerInfo II as described above, with a selector context by a player character. This has the effect that when a
nearestOne applied thereto to select the nearest blue 25
non-player character queries that learner for a next action, the
player. Similarly, if the iterative operator were named learner can provide a next action similar to one that may be
“enemyPlayerInfo II and defined using a predicate taken by the player character (if the player character were
operator is Enemy, there may be applied thereto the directing the actions of that non-player character).
same selector nearestOne to select the nearest enemy In one embodiment, learner 165 makes predictions and
player. 30
generates responses to queries for given contexts based on
learned behavior. Learner 165 can also be referred to as a
A derived feature 160 may be defined in response to other simulation engine.
raw features 150, other derived features 160, or recur
sively defined in response to a variant form of the same In FIG. 1, learner 165 includes learning element(s) 170,
derived feature 160 (for example a sorted version of an behavior model(s) 180 and decision model(s) 190. The inven
tion is not limited to learners constructed as shown in FIG.1.
array or list). For one example, not intended to be lim 35
Inputs to leaning element(s) 170 include contexts 200 and
iting in any way, there may be a predicate operator
“safe Atlast’ defined in response to either raw features training signal 210. Each context 200 includes a set of values
150 or derived features 160, as for example using the for the derived features 160 and possibly for the raw features
assignment “safeAtlast=IF (nearestEnemy==Null). 150. The number of possible contexts may be quite large. For
40 example, not intended to be limiting in any way, if there are
A derived feature 160 may be defined in response to a several hundred raw features 150, each of which may take on
learner 165, with the effect that an output of a defined one of ten possible values, and there are several hundred
learner 165 may be used as an input for defining a derived features 160 determined in response to those raw
derived feature 160. For one example, not intended to be features 150, the number of possible contexts 170 may be
limiting in any way, a learner 165 may be defined to 45 larger than 10 raised to the power of hundreds.
provide an array or list of the 10 choices recently made Training signal 210 includes decisions made by one or
by a player character and the 10 choices predicted as more of Software agents 140. Training signal 210 can also
most likely to be made by that player character for a include comparisons between actions predicted by learner
“next move.” In such cases, a derived feature 160 may 165 and actions actually taken by a Software agent. Thus,
easily be defined in response to that learner 165, for 50 learning element(s) 170 are trained based on what decisions
example, using the assignment "surpriseMove-IF (ac one or more software agents 140 make in various contexts.
tualMove - predictedMove). In FIG. 1, learning element(s) 170 are the portions of
One aspect of the invention is the provision of a selected set learner 165 that are primarily responsible for generating sig
of derived features needed for the learner 165 to be able to nals that result in machine learning. In particular, learning
learn effectively. The inventors have found that it is non 55 element(s) 170 generate updates 175 for behavior model(s)
obvious and unexpectedly useful to provide such a selected 180.
set of derived features. A behavior model 180 attempts to model behaviors exhib
First, this has the effect of allowing the game designer to ited by software agents 140 in various contexts 200. Behavior
not have to think about which derived features are going model(s) 180 preferably generate predictions based on con
to be derived from the raw features (or other system state 60 text information. The predictions from behavior model(s) 180
variables) of the game simulation, when designing or attempt to approximate what a player character or trained
implementing that game simulation. non-player character would do in the given context.
Second, this has the effect of allowing the game designer to In one embodiment, a behavior model 180 includes a set of
not have to design any code or data structures for pro characteristics that (at least partially) predict the interaction
viding those derived features, and also allowing the 65 of a non-player character 142 in various contexts during
game designer to not have to design any code or data game-play. These characteristic could take the form of coef
structures for computing those derived features. ficients, weights, or other values
 US 7,636,701 B2
11 12
In the arrangement shown in FIG. 1, the context informa A query could be “what is a ranking for a set of possibili
tion for behavior model(s) 180 is contexts 200 from derived ties.” These possibilities could include possible player
features 160. Alternatively, the same or modified context character or non-player character actions, possible cur
information can be supplied to the behavior model(s) 180 rent or future contexts, or the like. The rankings could be
from decision model(s) 190. For example, the decision model based on importance, likelihood of Success, amount of
(s) could ask for a prediction about a context that may occur time required, or the like, relative to some player char
based on a current context but that is not actually occurring. acter or non-player character goal or some game context.
A decision model 190 uses one or more behavior model(s) A query could be for a sample from a distribution over a set
180 to respond to queries 220 from game engine 110, given of actual or possible results. In one embodiment, it is not
contexts 200. Accordingly, a decision model 190 is respon 10 necessary for the query maker to know the actual prob
sive to a set of contexts 200, and is capable of providing a ability distribution to obtain a response.
decision for an action by a non-player character 142. For A query could be “what is the probability, or conditional
example, a decision model 190 may provide a decision for a probability of a given event.” In one embodiment, it is
non-player character 142 regarding whether, in a particular not necessary for the query maker to know the actual
context, fight or flight is better for that non-player character 15 probability distribution to obtain a response.
142. A query could be “what is the probability distribution (in
The decision model(s) 190 can be viewed as wrapping the whole or in part) for a set of events.” In one embodiment,
behavior model(s) 180, in the sense that a decision model it is not necessary for the query maker to know the actual
takes a prediction from one or more behavior model(s) and probability distribution for each individual event to
can choose to adapt or modify the prediction to answer a obtain a response.
query. Alternatively, the decision model(s) and behavior A query could be “what is the location of similar objects
model(s) can be viewed as separate entities, in the sense that within some set given criteria by which the grouping is
the behavior model(s) are more involved in updating and made.
learning, whereas the decision model(s) are more involved in A query could be “what are the top N most important
predicting behavior and responding to queries. 25 factors in making a given decision possibly with
The learner 165 embodies one or more machine learning respect to a given phenomena, for example that explain
techniques. Accordingly, the learner 165 would be ultimately the differences between 2 different answers to some
responsive to the raw features 150 and derived features 160 in other query.
each of contexts 200, and thus responsive to each context 200 A query could be “how certain or uncertain is the model
as it occurs, to provide a response to a query from the game 30 about the query result being returned. In one embodi
engine 110. ment, it is not necessary for the query maker to know
Multiple learning elements 170, behavior models 180 and how the query result is actually determined to obtain a
decision models 190 may be combined into one or more response.
learners 165, each for responding to queries in different situ A query could be “what is the most critical information
ations. These multiple learners, learning elements, and/or 35 (input samples) that can be collected to reduce the uncer
models may control distinct aspects of the actions taken by a tainty of a given query or class of queries. In one
non-player character 142 (in a particular context 170). For embodiment, it is not necessary for the query maker to
one example, not intended to be limiting in any way, a first know how the query result is actually determined to
learning element 180 may determine the fight-or-flight obtain a response.
response (as described above) for the non-player character 40 A query could be “what is the most/least common result
142, a second learner 165 may determine what direction and over a distribution of results.” In one embodiment, it is
how best to conduct flight when that response is chosen by the not necessary for the query maker to know the actual
first learner 165, and a third learner 165 may determine how probability distribution for each individual result, or
to conduct combat when that response is chosen by the first how the query result is actually computed, to obtain a
learner 165. 45 response.
For example, not intended to be limiting in any way, a A query could be “why was a given action predicted. In
learner 165 may be responsive to include one or more of the one embodiment, it is not necessary for the query maker
following queries and provide one more of the following to know the actual probability distribution for each indi
responses: vidual predicted action.
A query could be “what action should be taken next given 50 A query could be “what actions should be taken' to achieve
a past, current, or predicted context.” The action could be a given goal in a given context.
a direction vector to travel, a speed to travel, a discrete A query could be “what actions would one player or non
action choice (e.g. “punch”, “shoot', 'jump'), a force to player character be likely to perform in place of another
apply, or any other action you can imagine a player player or non-player character as well as a listing or
character or NPC having to choose in a game. The action 55 ranking of probabilities for those actions.
could also be a “high-level action that is translated Other queries could include (from the perspective of a
down into game-level action by some other component. character) “what would I do if I were you.” “what may I
For example, not intended to be limiting in any way, the do in this other situation.” “what are my top N choices.”
learner could respond with a goal location, a path plan and the like.
ner then plans a path to the goal, and a path follower 60 As learner165 is providing responses 230, the game engine
generates the low-level game actions to follow the path. 110 compares those responses with the actions of a target
A query could be “what event is likely or possible to hap whose behavior the non-player character 142 seeks to imitate.
pen next (for example, predicting a selected next event For one example, not intended to be limiting in any way, the
or a selected action by another character). The response target may include a particular player character 141, with the
could be an action that a player character or non-player 65 effect that the decisions made by the learner 165 (as requested
character is likely take in a given context, a predicted by the game engine 110) are compared with actual decisions
change in context, or the like. made by the player character 141.
 US 7,636,701 B2
13 14
In the foregoing description of FIG.1, learner 165 learns by sian classifier, a neural network, an expert System, a hidden
having learning element(s) 170 update behavior model(s) Markov model, a partially observable Markov decision pro
180, which in turn affect decision model(s) 190. The inven cess model, etc.
tion is not limited to this particular arrangement. Other After reading this application, those of ordinary skill in the
designs of elements and modules that result in machine learn art would recognize that many variations on the learner 165
ing or other forms of artificial intelligence can be used with are within the scope and spirit of the invention, and that those
the invention. In those other designs, each of learning element variations would not require undue experimentation or new
(s) 170, behavior model(s) 180, and decision model(s) 190 invention. Some examples, not intended to be limiting in any
may or may not be present, may have different responsibili way, include:
ties, or may be replaced with one or more other elements or 10 training an individual non-player character 142 in response
modules. to an individual target player character 141, for example,
Learning to imitate that player character 141, to do the opposite of
A human player 143 generally makes better decisions the that player character 141, to act favorably toward that
next time they encounter a situation based on the Success of player character 141, to actinimically toward that player
15
prior decisions and their outcomes. Simple Artificial Intelli character 141, to act neither like nor dislike (that is,
gence in computer games often does not learn because it "orthogonal to') that player character 141, or in general
generally retains no prior knowledge of past experiences, and any transformation of the above, for example, to act like
when it does, it does not learn from that experience so as to act that player character 141 but more slowly;
differently in that context a next time. training a group of non-player characters 142 in response
Enabling a game that learns to interact more like human to an individual target player character 141, for example,
beings keeps the game fresh for the human user 143. For to (imitate, do the opposite of favor, disfavor, act
example, "PacMan” was a very popular game in the early orthogonally to, or some transform of the above) with
eighties. After publication of known “play patterns” for its regard to decisions made by that player character 141—
non-player characters 142, anyone who knew the patterns 25
this would include training a group of “followers” for
could potentially score an unlimited number of points. Essen that player character 141:
tially, the game was beaten and gamers looked to other games training one or more non-player characters 142 in response
that presented a challenge. to a group of target player characters 141, where each
The learner 165 enables techniques whereby a game decision or action made by an individual player charac
designer or programmer unskilled in artificial intelligence 30
ter 141 may be considered in the training process, with
can take advantage of more powerful learning techniques, the effect that one or more of the non-player characters
overcoming these short-falls of prior games that do not learn 142 act in response to a statistical measure of the actions
or that learn through simple AI techniques implemented by taken by those player characters 141 (for one such
game designers and programmers. example, the training signal may be considered to be
In one embodiment, decisions made by the learner 165 (as 35
modal action taken by the set of individual player char
requested by the game engine 110) are compared with actual acter 141 in that context, that is, whether most of them
decisions made by the player character 141. Training signal run away or stay to fight);
210 can represent the actual decisions, a comparison of training one or more non-player characters 142 in response
responses 230 to those decisions, or some other measure of to a set of results (short-term, long-term, or otherwise)
actual decisions. Learning element 170 of learner 165 can use 40 produced by a target, where that target may be one or
this training signal to update behavior model 180. more player characters 141, one or more non-player
After reading this application, those of ordinary skill in the characters 142 (possibly including the ones being
art would recognize that learner 165 can implement a predic trained), or some combination thereof this would
tive technique in which learner 165 attempts to predict the include a technique for unsupervised learning by one or
training signal 210 for each context 200. When the learner 45 more non-player characters 142 in the game 100:
165 is sufficiently good at predicting that training signal, it training one or more non-player characters 142 in response
may be said that the learner 165 “learned to imitate' a target to an aggregate, average, or some other statistical Sam
player character 141. pling of actions taken by a group of player characters or
In one embodiment, the behavior model(s) 180 of learner non-player characters, for example to train the non
165 may use weighted averages, median calculations, or other 50 player characters based on a collective strategy used by
statistical measures to make predictions. Learning element(s) a group of player or non-player characters (for one such
170 could update the coefficients or other values used to example, the training signal for a first set of non-player
implement the statistical measures based on contexts 200 and characters may include the actions of a player character
training signal 210. for a first time duration, while the training signal for a
Preferably, the weights or coefficients for a behavior model 55 second set of non-player characters may include the
180 are real-valued coefficients or other values as precise as actions of a player character for a second time duration,
available using machine arithmetic. However, in the context with the effect that the second set of non-player charac
of the invention, there is no particular requirement for select ters acts differently from the first set of non-player char
ing the coefficients or statistical measure from any particular acters if the player characteracted differently during the
number field. 60 second time duration than during the first time duration);
The inventors have found that an exponential gradient tech training one or more Subsets of non-player characters 142
nique for machine learning provides a reasonably quick, Suf at different times, with the effect that one or more non
ficient method by which the coefficients of learning element player characters in a Subset can later be designated to
170 can be updated. perform a certain action, to be used in a selected context,
Other techniques can be used by learner 165 instead oforin 65 or the like, either by the game system or by a player;
addition to the exponential gradient technique. Examples of retraining or repartitioning one or more Subsets of non
such other techniques include but are not limited to a Baye player characters that were trained at different times.
 US 7,636,701 B2
15 16
In Some embodiments, the game engine 110 may provide a Storage
human user 143 with access to attributes within a non-player The storage element 240 includes a device for storing
character 142 generated by the learner 165, for example, to electronic data including but not limited to a hard disk drive,
correct any flaws, or for example, to add flaws, such as when an optical drive, a silicon chip storage element (Smart media
the non-player character 142 plays the game too well and is 5 card, video game external storage card, random access
hard to beat. memory). Code and data for the game 100 can be saved to the
In various embodiments, non-player characters 142 may storage 240.
start with any one of the following: In one embodiment, data that is generated by the learner
a blank slate behavior model 180 that is, no important 165 may be saved and loaded, either on demand, or in
information about the game 100: 10 response to particular inputs (for example, a game controller
a basic behavior model 180 that is, some basic informa button). This data may be used as an initial behavior model
tion about the game 100, including possibly some rules 180, that is, pre-training, or may be used as a static behavior
to prevent the non-player character 142 from doing any model 180, that is, post-training. The behavior model 180
thing exceedingly stupid, may be incorporated into one or more software agents 140,
an advanced behavior model 180 that is, significant 15 for example, non-player characters 142. In some embodi
information about the game 100, possibly ranked by ments, new non-player characters may be generated with
“experience' or “level” or some other measure of knowl their initial behavior model 180 at least partially specified by
edge about the game 100. the human player 143.
In various embodiments, non-player characters 142 may be Applications
coupled to a training signal for learning, Such as any one of the
following: The invention enables a multitude of new modalities,
no training signal mechanisms and options (collectively referred to as tech
a player character 141 as the training signal, for example, to niques below) for game designers. Examples, not intended to
learn to imitate the knowledge and style of that player be limiting in any way, are described below. Application of
character 141; 25 the invention to these modalities, mechanisms and options
reinforcement from the game 100 as the target, for would not require further invention or undue experimenta
example, a form of unsupervised learning for the non tion. These applications include techniques that Support
player character 142, as described above: learning behavior either during actual game play, in a batch
reinforcement from another non-player character 142 as mode, or a combination thereof.
the training signal, for example, a set of "experienced 30 A technique that Supports a “train by playing modality
non-player characters 142 as described below—in such whereby a player character's activities while playing the
embodiments, non-player characters 142 may alterna game is used to train non-player-characters. Entities
tively use each other as target training signals in a simu Such as followers (i.e., non-player characters) can learn
lated contest among or between those non-player char not only the actions the player tends to take, but the
acters 142, providing a second form of unsupervised 35 contexts in which to apply them. Learning can continue
learning for the non-player characters 142. over the course of the game.
In various embodiments, non-player characters 142 may be A technique that Supports creating teams wherein indi
coupled to a learner 165 for learning, such as any one of the vidual members of the team are trained individually by a
following: player using the “train by playing modality.
no learning 40 A technique that Supports in-play controlling and/or influ
learning for an initial duration, after which the non-player encing actions of other non-player-entities in a game.
character 142 has acquired some of the knowledge and Entities include but are not limited to non-player-char
style of the target, but not so much as to look like a copy acters, and the active control elements of the virtual
thereof the inventors have found that even a few sec game world. Entities can be followers, friends or
onds of regular play can bring a non-player character 45 enemies.
142 to a point of looking very much like its target player A technique that Supports controlling and/or influencing
character 141; the actions of a group of followers that learn both the
learning for a Substantial duration, after which the non actions the player takes, as well as the context in which
player character 142 has acquired a Substantial amount the player takes those actions in 2D and 3D tactical/
of the knowledge and style of the target, possibly enough 50 Strategic action and fighting games.
to directly challenge the knowledge and style of that A technique that Supports controlling and/or influencing
player character 141: entities that can recognize millions of different contexts
continual or periodic learning, in which the non-player and behave appropriately according to how they have
character 142 continues to acquire more of the knowl observed the player behaving.
edge and style of the target. 55 A technique that Supports controlling and/or influencing
A production run of a game 100 or application within the entities that can adapt and evolve their behavior on-the
game 100 may include at least some non-player characters fly and over time as the player's behavior changes.
142 for each degree of development in their behavior model A technique that Supports controlling and/or influencing
180. entities that can learn tactical or high-level strategic play
In one embodiment, behavior models for the non-player 60 from the player.
character 142 may be saved for later use, either in the same A technique that Supports controlling and/or influencing
game 100 or in another copy of that game 100 (that is, behav the actions of teammates in sports games. The sport
ior models may be communicated or carried on physical maybe a depiction of a known sport, or a sport with rules
media from one game 100 to another game 100). However, in concocted for the video game.
the context of the invention, there is no particular requirement 65 A technique that Supports controlling and/or influencing
for actually saving any behavior models for any or all of the entities that learn quickly and for whom learning can
non-player characters 142. continue indefinitely.
 US 7,636,701 B2
17 18
A technique that Supports controlling and/or influencing A technique that Supports training several generals at once
entities that can learn individual skills and/or how to on the battlefield, either by example or by swapping into
co-operate in large and Small groups. control them.
A technique that Supports controlling and/or influencing A technique that Supports allowing a player to create ava
entities either during a single critical decision, or all the tars as opponents for other players to compete with.
time, or anything in-between. A technique that Supports creating and evolving a knowl
A technique that Supports creating behavior for entities that edge base per player, for example to be loaded up for
can be saved, restored, replayed and/or traded. competitive reasons.
A technique that Supports personalizing a player's experi Moreover, in the context of the invention, there is no par
ence of the game. 10
ticular requirement that the player themself cannot learn from
A technique that Supports controlling and/or influencing non-player characters. For example the invention enables a
entities that can learn to make tactical decisions like, technique that Supports training a new player character by
Such as but not limited to: timing, spacing, approach, providing hints or indicators of what other player characters
positioning and selection of opening moves and follow (or Scripts designed for non-player characters would do in a
up moves for attacks, blocks, and backing off target 15
similar context. Application of the invention to these tech
Selection, including double-teaming; direction and style niques would not require further invention or undue experi
of movement, where and when to dodge and flee; selec mentation.
tion of a level of aggression; ordering and timing of
decisions; deciding to flee or fight. Dead Reckoning
A technique that Supports controlling and/or influencing Another technique enabled by the invention includes “deal
target selection decisions, such as but not limited to: reckoning in which a game (or other virtual simulation) may
aiming for the heart of a cluster, aiming at Stragglers; be performed on more than one computing device. For one
ignoring mobs and go for a leader or Vice-versa; select example, not intended to be limiting in any way, a game may
ing which power-ups to go after, which crates to open, be played using two logically or physically remote computing
with what priority, and the like: selecting which rooms to 25 devices A (including controlling a first player character X at
in, which doors to try, and the like. computing device A) and B (including controlling a second
A technique that Supports controlling and/or influencing player character Y at computing device B). Each computing
entities that initially have either no knowledge, very device may use a prediction function predictLocation (char
little knowledge, lots of knowledge or anything in-be acter, futureTime) to predict a location of that selected char
tween and learn from the player. 30 acter at that selected futureTime.
A technique that Supports controlling and/or influencing a If each computing device has access to a substantially
single companion, a core group of followers, whole identical function predictLocation (), the computing devices
armies, or anything in between. may exchange only differences between actual values and
A technique that Supports controlling and/or influencing predicted values. For one example, not intended to be limiting
entities in single player mode, or in head-to-head events 35
in any way, if computing device A predicts character X's
that show off training and leadership skills. future location at a time t1 as XA predictLocation (X, t1),
A technique that Supports controlling and/or influencing while computing device B predicts character X’s future loca
entities whose errors can be corrected by demonstration. tion at the time t1 as XB-predictLocation (X, t1), the two
A technique that Supports controlling and/or influencing computing devices each know approximately where charac
entities in different groups in different ways. That is, 40
ter X should be, that is, XAXB. If computing device Aknows
Some user interface element, or pre-defined condition, is that character X is in a substantially different location, that is,
used to specify to which group the training signal is computing device A's computation of XA-actualDocation
applied and when. (X, t1)>Some minimum threshold tolerance, computing
A technique that Supports controlling and/or influencing device A knows it needs to send information to computing
entities whose behavior can evolve over multiple sce 45
device B to update the latter's value of actuallocation (X, t1).
narios, and multiple games. In one embodiment, the predictor predictLocation () can
A technique that Supports controlling and/or influencing be improved by learning in context, either at computing
entities at the formation level. For example, not intended to be device A, computing device B, or both. In Such embodiments,
limiting in any way, a group of entities can learn: What is the computing devices A and B may need (but alternatively may
shape of a given Squad—a line, a wedge, a square'? When does 50
dispense with) sending information to update each others
a given Squad choose to engage enemies, and when does it decision model for predictLocaction ( ), but the inventors
flee? Does every soldier attack for himself, or should he hold expect that total communication traffic between by comput
formation and coordinate attacks?
A technique that Supports optionally controlling and/or ing devices A and B would be substantially reduced.
influencing entities that can learn from the player with 55 After reading this application, those skilled in the art would
commands like: "Attention,” “Dismissed, "Scatter.” recognize that predictLocation () has a clear training signal,
or “Form up!” as whenever it is wrong by more than Some minimum thresh
A technique that Supports creating non-player-entities that old tolerance, it is corrected to provide the actual location
can explain their behavior. value.
A technique that Supports creating non-player-entities that 60
can negotiate. EXAMPLE
A technique that Supports sharing trained avatars (i.e.,
characters), for example to permit later selection of a FIG. 2 shows a block diagram of an example of context
best-trained avatar for each head-to-head encounter. learning by Software agents. The invention is not limited in
A technique that Supports allowing a non-player character 65 any way to the details of this example.
to emulate behavior of expert or even world-champion Raw features 150 are shown in FIG. 2 as including various
players. values and indications for game code 120. A learner accord
 US 7,636,701 B2
19 20
ing to the invention generates the derived features 160 from coupled to a learner 165 for learning. If so, the method per
the raw features 150 and possibly from other derived features forms the next step (the step 306), and continues with the step
160. thereafter (the step 307). If not, the method 300 skips the next
In the example shown in FIG. 2, the game code is for a step (the step 306), and continues with the step thereafter (the
simulator Such as a flight simulator. The various raw features step 307).
150 relate to information about the flight simulation. The At a step 306, the method 300 couples one or more non
particular raw features shown in FIG. 2 are meant to be player characters 142 to learners 165, as determined in the
illustrative only; actual code for a flight simulator may use all, previous step (the step 305).
Some, none, or different raw features. At a step 307, the method 300 updates the behavior models
The derived features 160 in FIG. 2 represent relationships 10 180, for example using learning elements 170 acting in
between some of raw features 150. Derived features 160 response to contexts 200, which in turn are based on raw
according to the invention are not limited to Such relation features 150 and derived features 160, and in response to
ships. Furthermore, the particular relationships shown in FIG. training signal(s) 210.
2 are meant to be illustrative only; actual derived features for In one embodiment, the contexts may include a set of
a flight simulator may use all, some, none, or different derived 15 changes from earlier recent contexts. However, in the context
features. of the invention, there is no particular requirement for any
In order to generate the derived features, the learner must special format for this information. Either all of contexts 200,
know where in the game code 120 to find those features. This or parts thereof, or a set of differences, or some of them, may
information preferably is extracted from the code, preferably be used.
based on information Supplied by the game's developer or In one embodiment, the training signals could be actions
designer, and based on templates Supplied for this purpose. actually taken by a player or non-player character, a compari
The derived features 160 form one or more contexts 200. son of actions taken with earlier responses from leaner(s) 165
These contexts are supplied to learning element(s) 170 and or predictions by behavior model(s) 180, or some other type
behavior element(s) 180, where they are used to generate of feedback. Learning elements 170 update behavior
predictions 185. 25 model(s) 180 responsive to these training signals, for example
The contexts also are supplied to decision model(s) 1909, by updating coefficients, weights, or other values used by the
where they are used in conjunction with predictions 185 to behavior model(s).
respond to queries 220 with responses 230. These queries and At a step 308, the method 300 receives queries, for example
responses are used by game code 120 to incorporate the for what action a non-player character should take, from one
learning capabilities of the invention into the code. 30 or more of the software agents 140.
The game code Supplies training signals 210 to learning At a step 309, the behavior model(s) 180 make predictions
element(s) 170, which use the signals to update behavior about what action a player character or some other character
element(s) 180. The behavior element(s) in turn affect the would take in the current context.
operation of decision model(s) 190. At a step 310, the decision model(s) 190 make decisions
Method of Operation 35 responsive to the predictions.
FIG.3 shows a process flow diagram of a method of oper At a step 311, a response to the query is sent to the querying
ating in a system for context learning by Software agents. Software agent responsive to the decisions.
A method 300 is performed by the game 100. Although the In a training mode, less than all of these steps may be
method 300 is described serially, the flow points and steps of performed. For example, steps 310 and 311 may be skipped in
40 a training mode. Conversely, if the learner(s) are considered
the method 300 can be performed by separate elements in
conjunction or in parallel, whether asynchronously or Syn to be fully trained, step 307 may be skipped. Other variations
chronously, in a pipelined manner, or otherwise. There is no are possible.
particular requirement that the method 300 must be per Those of steps 307 to 311 that are performed preferably are
formed in the same order in which this description lists flow repeatedly executed so as to facilitate learning by the Software
45
points or steps, except where explicitly so indicated. agents. As raw features 150 and derived features 160 change,
contexts 200 change, permitting the agents to learn behaviors
Associating, Updating and Using Behavior Models for different contexts. This continues until the game ends in
At a flow point 301, the method 300 is ready to begin steps 312 and 313.
associating and updating behavior model(s) 180 with soft
ware agents 140. 50 Interrelation of Method Parts
At a step 302, the game 100 is activated, and the method After reading this application, those skilled in the art will
300 begins to operate in conjunction or combination with the recognize that many variations are possible that would be
game 100. workable, that such variations are within the scope and sprit
At a step 303, the method 300 determines if there should be of the invention, and that such variations would not require
any non-player characters 142 with pre-loaded behavioral 55 undue experimentation or further invention.
and decision model(s). If so, the method 300 performs the For example, not intended to be limiting in any way, the
next step (the step 304), and continues with the step thereafter following aspects of the method 300 would be clear to those
(the step 305). If not, the method 300 skips the next step (the skilled in the art:
step 304), and continues with the step thereafter (the step Each of steps 307 to 311 can be, and preferably are, per
305). 60 formed in parallel. This has the effect that the method
At a step 304, the method 300 receives one or more behav 300 can update the learners 165 concurrently with using
ioral models 180 and decision models 190 from the storage those learners 165 to make decisions on behalf of non
element 240, and incorporates or associates those models to player characters 142.
non-player characters 142 as determined in the previous step In the context of the invention, there is no particular
(the step 303). 65 requirement for learning to start and stop when the game
At a step 305, the method 300 determines if any non-player starts or stops. For example, learning may be conducted
characters 142 (or other software agents 140) should be for only a short duration after the game starts, or at
 US 7,636,701 B2
21 22
pre-selected time durations during the game, or at peri learn the stratagems of the player character 141 and
odic time durations, or at time durations responsive to rapidly learn to neutralize simple tricks or quirks of the
the game itself (for example, performing learning for game 100.
combat skills only during intense combat). ALTERNATIVE EMBODIMENTS
In the context of the invention, it is possible for learning to
be conducted in context at one or more levels of gener Although preferred embodiments are disclosed herein,
ality. For one example, not intended to be limiting in any many variations are possible which remain within the con
way, non-player characters 142 can learn to imitate cept, scope, and spirit of the invention. These variations
player characters 142 skill at identifying the most dan 10 would become clear to those skilled in the art after perusal of
gerous enemy (e.g., not always the one that is hardest to this application.
defeat or does the most damage). Another example, also As noted above, the invention enables a multitude of new
not intended to be limiting in any way, non-player char modalities, mechanisms and options (collectively referred to
acters 142 can learn to imitate player characters 142 as techniques below) for game designers. Examples, not
skill at determining when to initiate and when to break 15 intended to be limiting in any is way, are described below.
off combat (e.g., when health is too low for the remain Application of the invention to these modalities, mechanisms
ing enemies, or when the remaining enemies appear too and options would not require further invention or undue
powerful to defeat without substantial risk). Another experimentation.
example, also not intended to be limiting in any way, For example, not intended to be limiting in any way, the
non-player characters 142 can learn to imitate player invention Supports techniques by which multiple non
characters 142 degree of daring or risk-taking (e.g., player characters can learn from multiple player charac
whether particular risks are “too risky” or not). Another ters, for example by (a) designating one of the player
example, also not intended to be limiting in any way, characters as a target signal to be imitated by one or more
non-player characters 142 can learn to imitate player non-player characters, (b) designating a statistical value
characters 142 goal preferences (e.g., whether to 25 in response to a set of player characters as a target signal
improve health or to find treasure to win the game, to be imitated by one or more non-player characters.
whether to pre-pare defensively against damage or For example, not intended to be limiting in any way, the
offensively to defeat enemies, and the like). Another invention Supports providing generalized artificial intel
example, also not intended to be limiting in any way, ligence or machine learning techniques that are respon
non-player characters 142 can learn to imitate player 30 sive to an exemplar, for example, by generating a model
character's 142 approach to goal-making or degree of of desired behavior without explicitly programming that
behavior.
intentionality of play (for example, whether to go out After reading this application, those skilled in the art would
“looking for trouble.” or to focus planning and effort on recognize that the techniques described herein provide an
a specific enemy or prize). After reading this application,
hose skilled in the art would recognize that all of these 35 enabling technology, with the effect that heretofore advanta
concepts are within the scope and spirit of the invention, geous features can be provided that heretofore were substan
would be workable, and would not require undue experi tially infeasible.
mentation or further invention. After reading this application, those skilled in the art will
The more the game is played, the more experiential learn recognize that these alternative embodiments and variations
ing non-player characters 142 gain. This enhances over
40 are illustrative and are intended to be in no way limiting.
all game play and rewards the human user 143 for time The invention claimed is:
invested in playing the game. 1. In a computer system including a simulation engine
Most of the currently popular game consoles promote col having a set of state variables, and wherein a collection of
laborative or competitive play. For example, the Sony 45 possible values for at least some of those state variables
PlayStation 2 console allows players to connect over a defines a context, a method including steps of
network to play as a team against other teams or to play defining a set of behavior models, each capable of receiv
against each other as enemies. Using the invention, a ing queries from the simulation engine and generating
host of scenarios become possible. For example, but responses to those queries;
without limitation, a single human user can be a team 50 defining a set of software agents, each being responsive to
that includes a player character 141 and one or more one or more of those behavior models, and each capable
non-player characters 142. of being updated in response to changes from a first
In addition to collaborative or competitive games, there are context to a second context;
also games representative of group activities, such as presenting a sequence of states, each possible Such state
sports teams, corporate activity, or military units. For 55 defining a context, the steps of presenting including
one example, not intended to be limiting in any way, a operating a set of software agents within a set of rules for
first possible use of the invention is for rapidly providing the steps of presenting; and
teams of non-player characters 142 with particular char generating a response to a query for decision by at least one
acteristics, for example, a platoon of extremely loyal of a set of decision models;
Soldiers. For another example, again not intended to be 60 wherein at least one of the Software agents is responsive to
limiting in any way, a second possible use of the inven one or more decision models, and is capable of being
tion is for providing teams of non-player characters 142 updated in response to changes from a first context to a
who imitate the player character 141 and thus allow the second context.
player character 141 to demonstrate (or learn) leader 2. A method as in claim 1, wherein
ship. For another example, again not intended to be 65 the steps of presenting collectively include operating a
limiting in any way, a third possible use of the invention simulation engine having a set of state variables, and
is for providing non-player characters 142 who rapidly having a set of rules describing permitted changes in
 US 7,636,701 B2
23 24
value of those state variables, wherein substantially each determining a probability distribution for a set of possible
collection of possible values for the set of state variables events;
defines a context; and determining a set of clusters of similar objects within a
the set of decision models collectively include a set of selected set;
behavior models, each capable of responding to a query 5 providing a set of criteria by which determining a set of
from the simulation engine and capable of generating a clusters of similar objects within a selected set can be
response to that query. performed:
3. A method as in claim 2, wherein the simulation engine selecting a set of relevant factors in making a game deci
includes a set of features, each either defined by a current sion;
context or associated with a set of information from outside 10 determining how certain or uncertain the decision model is
the simulation engine. regarding its response to a selected query for decision;
4. A method as in claim 2, including steps of constructing determining a set of relevant factors for retrieval to make
at least one of said behavior models in response to an action the decision model more certain or less uncertain about
taken by one or more of those software agents; whereby a first its response to a selected query for decision;
one of those Software agents is capable of learning behavior in 15 determining a most common result from a possible distri
context from a second one of those Software agents. bution thereof
5. A method as in claim 2, wherein the simulation engine determining a least common result from a possible distri
includes at least one of: bution thereof
a self-learning mode including at least Some of those providing an explanation of a reason for predicting a
behavior models and Software agents having initial val selected possible future action;
ues, and reinforcing those behavior models and Software providing a sequence of actions to advance a game agent
agents which are successful according to an evaluator toward a selected goal.
for the simulation engine; 8. A method as in claim 2, including steps of
a demonstration mode including at least some of those in a computer game having at least one decision model, the
behavior models and Software agents demonstrating 25 decision model being capable of generating a response
their learning to an observer, to a query for decision, generating a response to a query
a real-time mode including substantially all of those behav for decision including at least one of:
ior models and Software agents being updated with each training one or more non-player characters in real time in
change in context; response to actual play by a human user of one or more
a stochastic mode including introduction of random or 30 player characters, whereby those one or more non
pseudorandom values for at least Some of those state player characters learn during play;
variables. training a group or team of non-player characters in
6. A method as in claim 2, wherein the simulation engine response to an individual player character, whereby that
defines a computer game including at least one player or group or team of non-player characters learn the knowl
non-player character, and wherein the learning includes at 35 edge or style of that player character during play;
least one of: training one or more non-player characters in response to
a non-player character learning behavior from a player decisions made by each of a plurality of player charac
character, ters in context, whereby those one or more non-player
a plurality of non-player characters learning behavior from characters act in response to a statistical measure of the
a player character; 40 actions taken by those player characters;
a plurality of non-player characters learning behavior from training one or more non-player characters in response to a
a plurality of player characters; set of game results, wherein those game results are pro
a plurality of non-player characters each learning behavior duced by at least one of one or more player characters,
from a distinct one of a plurality of player characters; one or more non-player characters that are already
a first non-player character learning behavior from a sec 45 trained, one or more non-player characters being
ond non-player character; trained.
a non-player character learning to produce similar behav 9. A method as in claim 2, including steps of
ior in context as another character, in a computer game having at least one decision model, the
a non-player character learning to produce opposite behav decision model being capable of generating a response
ior in context as another character, 50 to a query for decision, generating a response to a query
a non-player character learning to produce behavior in for decision including training one or more non-player
context friendly to another character; characters in response to one or more player characters,
a non-player character learning to produce behavior in to perform at least one of:
context inimical to another character. imitating that player character,
7. A method as in claim 2, including steps of 55 doing an opposite of that player character;
in a computer game having at least one decision model, the act favorably toward that player character;
decision model being capable of generating a response act unfavorable toward that player character;
to a query for decision, generating a query for decision act orthogonally to that player character;
including at least one of act in a manner that is a particular transformation of how
Selecting which action to take in a selected context, the 60 that player character acts.
Selected context being a possibly empty context; 10. A method as in claim 2, including steps of
predicting a future event; in a computer game having at least one decision model, the
predicting what action another game agent will take in a decision model being capable of generating a response
Selected context; to a query for decision, generating a response to a query
ranking a set of possibilities in an order according to a 65 for decision including training one or more non-player
selected criterion; characters in response to a set of game results, wherein
selecting a result from a possible distribution thereof; those game results are produced by at least one of one or
 US 7,636,701 B2
25 26
more non-player characters that are already trained, one Supporting train by playing, whereby activities by a player
or more non-player characters being trained; character while playing the game are used to train non
whereby no human user is required for training, whereby player characters;
training of those one or more non-player characters is Supporting train by playing, whereby activities by a player
effectively unsupervised training. 5 character while playing the game are used to train non
11. A method as in claim 2, including steps of player characters, whereby non-player characters can
in a computer game having at least one decision model, the learn at least one of (1) actions a player character tends
decision model being capable of generating a response to take, (2) at least one context in which to apply those
to a query for decision, generating a response to a query actions, (3) continuing learning over a time duration
for decision including training one or more non-player 10 within the game;
characters in response to a set of game results, wherein Supporting creating teams wherein individual members of
those game results are produced by at least one of one or the team are trained individually by a player using train
more non-player characters or at least one of one or more by playing:
player characters; controlling or influencing actions of other non-player-en
whereby changes in strategy by a human user are adapted 15
tities in a game, wherein those entities include at least
to by the one or more non-player characters in response one of non-player characters including followers,
to those changes, whereby those one or more non-player friends or enemies; or active control elements of the
characters effectively learn from collaboration or com virtual game;
petition with human users. controlling or influencing the actions of a group of follow
12. A method as in claim 2, including steps of ers that learn at least one of (1) actions a player character
in a computer game having at least one decision model, the takes, (2) at least one context in which to apply those
decision model being capable of generating a response actions are taken in 2D and 3D tactical/strategic action
to a query for decision, generating a response to a query and fighting games;
for decision including training one or more non-player
characters in response to a set of game results, 25 controlling or influencing entities that can recognize mil
wherein those game results are produced by at least one of lions of different contexts and behave appropriately
one or more non-player characters or at least one of one according to how they have observed the player behav
or more player characters; ing:
wherein group actions by those one or more non-player controlling or influencing entities that can adapt and evolve
30
characters are responsive to the knowledge and style of their behavior on-the-fly and overtime as behavior of the
at least one human user. player character changes;
13. A method as in claim 12, wherein those group actions controlling or influencing entities that can learn tactical or
include at least one of business actions, military actions, high-level strategic play from player characters;
political actions, Social actions, sports team actions. 35 controlling or influencing the actions of teammates in
14. A method as in claim 2, including steps of sports games, wherein the sport might be a depiction of
in a computer game having at least one decision model, the a known sport, or a sport with rules concocted for the
decision model being capable of generating a response video game;
to a query for decision, generating a response to a query controlling or influencing entities that learn quickly and for
for decision including training one or more non-player 40 whom learning can continue indefinitely;
characters in response to a set of game results, wherein controlling or influencing entities that can learn individual
those game results are produced by at least one of one or skills or how to cooperate in large or Small groups;
more non-player characters or at least one of one or more controlling or influencing entities during any one of (1) a
player characters;
wherein actions by those one or more non-player charac single critical decision, (2) all the time, (3) Some fraction
45 of decisions therebetween;
ters are responsive for at least one human user, to at least
one of demographic information, player history, player creating behavior for entities that can be saved, restored,
preferences, player skill, player style. replayed or traded;
15. A method as in claim 14, including at least two human controlling or influencing entities that can learn to make
users, wherein actions by those one or more non-player char 50 tactical decisions including at least one of timing, spac
acters are responsive for at least one pair of human users, to a ing, approach, positioning and selection of opening
comparison or interaction between that at least one pair of moves and follow-up moves for attacks, blocks, and
human users, of at least one for each human user of demo backing off target selection, including double-teaming;
graphic information, player history, player preferences, direction and style of movement, where and when to
player skill, player style. 55 dodge and flee; selection of a level of aggression; order
16. A method as in claim 2, including steps of ing and timing of decisions; deciding to flee or fight;
a computer game having at least one decision model, the controlling or influencing target selection decisions includ
decision model being capable of generating a response ing at least one of aiming for the heart of a cluster;
to a query for decision, generating a response to a query aiming at Stragglers; ignoring mobs to attack a leader or
for decision including training one or more non-player 60 Vice-versa; selecting which power-ups to try to obtain,
characters in response to at least one of one or more Selecting which crates to open or with what priority;
non-player characters or at least one of one or more Selecting which rooms to enter or with what priority,
player characters; selecting which doors to try or with what priority;
wherein those trained one or more non-player characters controlling or influencing entities that initially at least one
are substantially able to negotiate with human users. 65 of (1) no knowledge, (2) very little knowledge, (3) lots of
17. A method as in claim 2, including at least one of the knowledge, or (4) Some degree of knowledge therebe
following steps: tween, and thereafter learn from player characters;
 US 7,636,701 B2
27 28
controlling or influencing at least one of (1) a single com every soldier attacking individually or holding forma
panion, (2) a core group of followers, (3) whole armies, tion or coordinating attacks;
or (4) anything therebetween, in response to at least one optionally controlling or influencing entities that can learn
player character; commands from the player character,
controlling or influencing entities in a single player mode creating non-player entities that can explain their behavior;
or in head-to-head events that show off training and creating non-player entities that can negotiate;
leadership skills; sharing trained characters, whereby later selection of a
choice of avatars is possible for a selected encounter;
controlling or influencing entities whose errors can be cor allowing a player to create avatars as opponents for other
rected by demonstration; 10 players to compete with:
controlling or influencing entities, wherein some user allowing a non-player character to emulate behavior of an
interface element, or predefined condition, specifies to expert or a world-champion player character;
which group the training signal is applied and when: creating and evolving a knowledge base per player,
controlling or influencing entities whose behavior can wherein that knowledge base can be loaded for a
evolve over multiple scenarios or multiple games; Selected encounter;
controlling or influencing entities with regard to type of training a new player character by providing hints or indi
organization or formation, including at least one of unit cators of what other player characters or Scripts designed
shape, when a unit chooses to engage enemies, when a for non-player characters would do in a similar context.
unit chooses to flee enemies, whether a unit involves k k k k k