AI for Dynamic Difficulty Adjustment in Games

Robin Hunicke, Vernell Chapman

Northwestern University
Computer Science Department
1890 Maple - Evanston, IL 60201
hunicke@cs.northwestern.edu, vernell@northwestern.edu

Abstract Game developers iteratively refine these systems based on

Video Games are boring when they are too easy and play testing feedback – tweaking behaviors and settings
frustrating when they are too hard. While most single- until the game is balanced. While balancing, they often
player games allow players to adjust basic difficulty (easy, analyze systems intuitively by tracking specific identifiable
medium, hard, insane), their overall level of challenge is patterns or types of dynamic activity. It is a difficult and
often static in the face of individual player input. This lack time consuming process [Rollings and Adams, 2003].
of flexibility can lead to mismatches between player ability
and overall game difficulty.
While game balancing and tuning can’t be automated,
In this paper, we explore the computational and design directed mathematical analysis can reveal deeper structures
requirements for a dynamic difficulty adjustment system. and relationships within a game system. With the right
We present a probabilistic method (drawn predominantly tools, researchers and developers can calculate
from Inventory Theory) for representing and reasoning relationships in less time, with greater accuracy.
about uncertainty in games. We describe the
implementation of these techniques, and discuss how the In this paper, we describe a first step towards such tools.
resulting system can be applied to create flexible interactive Hamlet is a Dynamic Difficulty Adjustment (DDA) system
experiences that adjust on the fly.
built using Valve’s Half Life game engine. Using
techniques drawn from Inventory Theory and Operations
Introduction Research, Hamlet analyzes and adjust the supply and
demand of game inventory in order to control overall game
Video games are designed to generate engaging difficulty.
experiences: suspenseful horrors, whimsical amusements,
fantastic adventures. But unlike films, books, or televised
media – which often have similar experiential goals – Prior Work
video games are interactive. Players create meaning by
interacting with the game’s internal systems.
Commercial DDA
One such system is inventory – the stock of items that a While it has potential benefits, DDA does not come
player collects and carries throughout the game world.1 cheaply. Ultimately, DDA systems take control away from
The relative abundance or scarcity of inventory items has a the designer and put it in the hands of code, which has
direct impact on the player’s experience.2 As such, games obvious drawbacks.
are explicitly designed to manipulate the exchange of
resources between world and player. [Simpson, 2001] Few commercial developers have implemented adjustment
systems for their games, and even fewer have shipped
This network of producer-consumer relationships can be them. For a discussion of commercial DDA system designs
viewed as an economy – or more broadly, as a dynamic and failures, see [Arey, Wells, 2001].
system [Castronova, 2000, Luenberger, 79].
To the extent that commercial systems have been formally
1
Inventory items for “first-person shooters” include health, proposed or published, there are two basic approaches. The
weapons, ammunition and power-ups like shielding or temporary first is predominantly manual; designers annotate game
invincibility. tasks and obstacles with information about their difficulty
2
A surplus of ammunition affords experimentation and “shoot prior to evaluation [Pfeiffer, 2003]. The second champions
first” tactics, while limited access to recovery items (like health a combination of data mining and off-line analysis
packs) will promote a more cautious approach to threatening [Kennerly, 2003].
situations.
We propose a probabilistic technique that dynamically
evaluates the difficulty of given obstacles based on user Flow
performance, as the game is running. Game adjustment is not as simple as adding health when
the player is in trouble; it is a design problem that involves
Genres estimating when and how to intervene. Keeping the player
challenged and interested is especially difficult in
Difficulty adjustment can be applied in almost any game
interactive contexts. [LeBlanc et al, 2001-2004].
genre, but is often discussed within the context of
Massively Multiplayer Online Games. Due to the size of
One common approach to player investment is the flow
these games, and their largely stats-based nature, even
model developed by M. Csikszentmihalyi. Here our goal is
slight imbalances can have an extremely negative impact
to keep the player in the flow channel – away from states
on the overall player experience – but tweaking the live
where the game is far too challenging, or way too easy.
game is also disruptive.

As a result, MMOG developers want to know which

elements should be tuned, and how that can be achieved High

Increasing challenge
Too Difficult Skill
successfully in a live game. Specifically – when is it
necessary to tweak an in-game system? Which adjustments
will result in direct, predictable and desirable changes to Flow
the system dynamics? [Carpenter, 2003] Channel

Low
FPS game economies are relatively simple compared to Too Easy
Skill
online game economies. But the combat economics of FPS
games are often repeated in sub-economies of MMOG and Increasing Skill
other game spaces. We investigate the FPS environment
as a first step towards understanding the dynamics of Flow state as it transitions over the course of a dynamic
larger, more complex game economies. experience. Challenge and pacing must ramp to match skill,
in order to support continued engagement.

Hamlet System Looking at the FPS environment, we can abstract

gameplay with a relatively simple state transition diagram.
Players engage in loops of searching, retrieving, solving
Architecture and fighting. With each new level, new enemies and
The Hamlet system is primarily a set of libraries embedded obstacles are introduced. Overall, difficulty increases with
in the Half Life game engine. This includes functions for time, as does skill acquisition.

• Monitoring game statistics according to pre-

defined metrics. Die
• Defining adjustment actions and policies. Win Enemy
(or retreat)
• Executing those actions and policies.
• Displaying data and system control settings. Fight

• Generating play session traces. Search Solve

Spawn
(or not)
As the player moves throughout the game world, Hamlet Find
Find
uses statistical metrics to monitor incoming game data. Obstacle
Over time, Hamlet estimates the player’s future state from Get
Object
this data. When an undesirable but avoidable state is (or not)
predicted, the system intervenes and adjusts game settings
as necessary. A simplified state transition diagram for the First Person
Shooter game genre.
Basically, we are trying to predict when the player is
Hamlet is designed to keep the player in the flow channel
“flailing” – repeatedly edging towards a state where her
by encouraging certain states, and discouraging others.
current means can no longer accomplish necessary and
Effectively, our goal is to keep the player in engaging
immediate ends. When we detect flailing behavior, we
interaction loops, for the most appropriate period of time,
want to intervene – helping the player progress through the
given their level of overall skill and game-specific
game.
experience.
By conducting a cheap, abstract simulation of the player’s Because we are summing the distributions
progression through state space, we hope to predict when
the player is repeatedly entering an undesirable loop, and t
help them get out of it. ∑ x(i)
i =1
Goals
The mean µ of the resulting sum is the sum of the means
There are many takes on whether this type of adjustment is
“desirable” or “necessary” for a good game experience.
Without delving into a philosophical discussion of play  t

aesthetics or game design, we can parameterize our goals µ = Ε  ∑ x(i )  = t Ε( x(t )) = t µ d
with respect to adjustment as follows.  i =1 

We want Hamlet’s modifications to encourage both local And the variance is the sum of the variances
and long-term success and engagement. We want the
system to support the player – without eliminating
 t

negative feedback and making everything very easy and
predictable We want Hamlet to intervene just enough – so
σ 2 = V  ∑ x(i)  = tσ d 2
that system behavior is relatively stable and predictable
 i =1 
over time [Rollings, Morris, 2000]. As such, we aim to: σ = tσ d
• Assess when adjustment is necessary The distribution of damage then becomes
• Determine which changes should be made
• Execute changes as seamlessly as possible 1 2
2σ 2
p ( x) = e−( x−µ )
We will now discuss assessment and adjustment in light of σ 2π
these goals, state our observations regarding overall system
behavior, and close with preliminary conclusions. We can now state the cumulative distribution function of
the Gaussian distribution – also known as the error
integral. Here, F(d) represents the probability of receiving
Assessment d or less damage on a given tick:
Our goal is to determine when a player is flailing. In many
d d
FPS games, this is characterized by repeated inventory
∫ p ( x)dx = 1 σ 2π  ∫ e( x − µ )
2
(2σ 2 )
shortfalls – places where the player’s available resources F (d ) = dx
fail to meet the immediate demands. −∞ −∞

By observing trends in the player’s inventory expenditure, Obviously, this is not a closed form equation. But lucky
we can watch for potential shortfalls and thus, pinpoint for us, the error integral is implemented as erf(x) in the
potential adjustment opportunities. Our first task is to standard C and C++ math libraries.
establish metrics for assessing statistical data within the
game world. Let us start with the direct observation of Now that we know what the distribution of damage looks
player health – and more specifically, the damage a player like, we can start to approximate inventory levels for
takes over time. player health.

Damage Inventory
We begin by assuming a sequence of random In any given system, the level of inventory will depend on
measurements of damage x(t), each of which has some the rates of flow in and out. This is often discussed in
terms of supply (input flow) and demand (output flow)
probability distribution pd .
[Mankiw, 1997].

The total damage at a given point in time is then the sum of

these random variables. By the Central Limit Theorem, is constant, and so the process is stationary. We also assume that
this sum converges to a Gaussian distribution.3 the probability of being hit at one moment is independent of
whether you were hit before, so the process is uncorrelated, or
3
In practice, these probabilities will vary depending on the “white”.
application domain. Here, we assume the probability of being hit
 Adjustment
We are currently experimenting with a number of
adjustment protocols in the Hamlet system. Adjustment
0
actions, when combined with cost estimations, will form
t adjustment policies. What follows is a description of
0 action, cost evaluation and policy designs, and an example
Inventory level as it fluctuates over the period of time t. A
illustration of the final goals for player-system interaction.
shortfall occurs when demand surpasses supply.
Actions
When completed, Hamlet will support two types of
In the FPS domain, player inventory continually shifts. adjustment actions.
Items are supplied as the player explores the environment
(found in the open, discovered inside crates, or field-
Reactive actions will adjust elements that are “in
stripped from the bodies of foes), and depleted by various
play” or “on stage” (i.e. entities that have noticed the
interactions within it (ammunition and health spent in
player and are attacking). This includes directly
combat, damage taken from falls, exposure to poison and
manipulating the accuracy or damage of attacks,
so on).
strength of weapons, level of health, and so on.
Using inventory theory equations, we can model the
player’s overall inventory as well as the flow if specific Proactive actions will adjust “off stage” elements (i.e.
inventory items in and out of the system. entities that are spawned but inactive or waiting to
spawn). This includes changes to the type, spawning
Shortfalls order, health, accuracy, damage and packing
properties of entities that have yet to appear on screen.
To predict an inventory shortfall, let z denote the inventory
level at the given time. The expected shortfall is the
cumulative probability of damage exceeding the initial While proactive changes give us more power over the
level or Ps, calculated as follows: game’s behavior, they happen at a greater distance from
the point of action – which can introduce uncertainty as to
their effectiveness. As a result, they are harder to evaluate.
Ps = P( x(t ) > z ) = 1 − P ( x(t ) < z ) = 1 − F ( z )
In the worst case, they may actually require subsequent
∞ reactive adjustments, causing a spiraling loop of change,
1
∫e
− ( t − µ )2 2σ 2
= 1− dt resulting in erratic and unpredictable behavior.
σ 2π z
Reactive adjustments map more directly to changes in the
gameplay experience. They are simple to execute, and
With this equation, and our handy erf functions, shortfall happen closer to the point of intersection/interaction
can be computed as a function of initial health, mean and between the player and game system. Changing the
standard deviation of loss of health over time.4 strength or accuracy of an entity that is under attack has a
straightforward impact on its life expectancy – and upon
We can use this calculation of eminent health shortfalls as on the life expectancy of the player.
a crude indicator of when the player is in need.5 Then, we
can move beyond assessment to adjustment – analyzing the In the end, it is a tradeoff. Reactive changes run the risk of
systems that impact the inventory item in question, and disrupting the player’s sense of disbelief. They can make it
adjusting the game accordingly. difficult to interpret the immediate behavior of in-game
obstacles, causing the game to appear schizophrenic.6
4
This approach is best at predicting events at times in the not- Proactive changes, because they happen “in the wings”,
too-distant future. With small sample sizes, the CLT is less are less likely to interrupt the player’s suspension of
accurate and standard deviation more likely to fluctuate. disbelief.
5
In application, these computations can be modified according to
the specific game under consideration. Different games may
require more complicated distributions, exhibiting a significantly 6
A problem that plagues many interactive AI research projects
different mean or standard deviation. [Sengers, 1998].
 engaged. This policy will be characterized by scarce but
Cost goal-oriented supply and sporadic but high demand.
Determining the cost of a given action, then, is clearly
critical to the process of gradual but effective adjustment. Example
If the system intervenes in the same way every time, the A player reaches the second engagement of Case Closed.
game might start to feel trivial, repetitive or boring. If it There are four enemies, and she has 45% health. Hamlet
continually makes interventions without considering the observes her inventory stats, to see if she is flailing – in
player’s perception of those interactions, it may interrupt this case, repeatedly dying before completing the
their overall suspension of disbelief. engagement, often with a majority of enemies standing.

Individual adjustment actions will be offset by

observations about

! The player’s current location in the level.

! How far along the player is in the game.
! How often they’ve died (at this location, in the
level, and in the game)
! How often they’ve repeated the current encounter
! How many times we’ve intervened in the past.

By calculating a cost for each intervention, and modifying

it as our understanding of the player’s performance
changes, we hope to dynamically adjust the game in more
responsive and responsible fashion. The second enemy encounter of Case Closed.

Policies
In immediate response to observed flailing, Hamlet may
Hamlet can combine these actions and cost calculations to
create “modes of control” – overall adjustment policies ! Donate a health pack somewhere in the scene.
that control the supply and demand of goods according to ! Upgrade the strength of her ammo.
overall player experience goals. Two examples follow. ! Reduce the accuracy or strength of enemy attacks.
Comfort Zone: This policy will keep players within a Depending on the success of these individual actions,
mean range of health points over time. Entities will be Hamlet can intervene again. If, over time, the player
tuned to shoot less often and less accurately, and requires significant adjustments to the initial levels set by
health is readily available. The goal is to keep the the game designer, Hamlet can also
player at about 50% health, occasionally dipping near
25% or cresting to 75%. ! Reduce initial strength and health of pending
enemies.
Trial and error are important in this policy – enemies will ! Pack more health and ammunition on the bodies
be tuned only if they repeatedly overwhelm the player. of vanquished foes.
Much like a benevolent babysitter, it intervenes frequently, ! Replace pending enemies with a type the user has
but leaves room for mistakes. Overall, the policy will be had more success with in the past.
characterized by steady demand and predictable supply.
The key here is that Hamlet will intervene iteratively,
Discomfort Zone: This policy is designed for more allowing for trial and error. The game will gradually
experienced players. The entities in a DZ game are change to accommodate the current player.
increasingly accurate, ammo and health relatively
scarce. The goal here is to keep the player “on the
edge of her seat” – constantly on the alert for enemies, Preliminary Conclusions
and fighting back from 15 or 20% health most of the
time.
Simulation
This policy is much more like an aggressive drill sergeant It’s clear that work of this nature involves tradeoffs with
or boxing coach. It sets high standards, but delivers respect to the type, number and frequency of adjustments
enough positive feedback to keep the player energized and performed. There is also a performance tradeoff here
between accuracy and generality/speed.
If our goal was to do a “correct” simulation of each Csikszentmhalyi, M.1990. Flow: The Psychology of
individual user’s performance, we would need to perform Optimal Experience. NY, NY: Harper Collins
several trials per user. However, as we are trying for a Berkely, California: Osborne/McGraw-Hill.
“best fit” given current information, this is not necessary.
Crawford, C. 1984. The Art of Computer Game Design.
In fact, we assume that a perfect simulation of the player’s Berkely, California: Osborne/McGraw-Hill.
progress through the environment would actually be less
useful due to the time constraints of real-time gameplay Jensen, P. A., Bard, J. F., 2002. Operations Research
and the limitations of commercial gaming hardware. Models and Methods. Hoboken, NJ, Wiley.

By conducting an abstract simulation of the user’s Kennerly, D., 2003. Better Game Design through Data
trajectory, we essentially generate an envelope of possible Mining. Gamasutra.com.
locations within the state space, and construct our system
to direct that envelope away from undesirable areas of Khoo, A., Hunicke, R., et al, 2002. FlexBot, Groo, Patton
gameplay. and Hamlet: Research using Computer Games as a
Platform. Technical content paper for Intelligent Systems
Evaluation Demonstration, Proceedings of the Eighteenth National
Conference on Artificial Intelligence.
While more advanced techniques can map out many of the
finer relationships in game systems, it is unclear to us at
LeBlanc, M., Hunicke, R., et al 2001-2004 – Game Tuning
this time whether this is necessary, or even desirable.
and Design Workshop. Game Developers Conference:
Future work may comment on the viability and value of
2001, San Jose.
alternative metrics, policies and so on.
Luenberger, D. G., 1979. Introduction to Dynamic
In the meantime, we plan to evaluate a broad sample of
Systems: Theory, Models, and Applications. New York:
users interacting with the Hamlet system. We will attach a
John Wiley and Sons, Inc.
simple heart-rate monitor to a variety of players, have them
play the game on both settings, and then map out the
Mankiw, N. G., 1997. Principles of Microeconomics. Fort
results.
Worth, Texas: Dryden.
If the adjusted game is able to keep test subjects equally
Pfeiffer, B., 2003. AI to Control Pacing in Games.
aroused over time, while dynamically adjusting to keep
Proceedings, IC2 GameDev Workshop, Univeristy of
them alive longer, then we will consider the work a
Texas, Austin.
success. 7
Rabin, S., 2002. editor, AI Programming Wisdom,
References Hingham, Massachusetts: Charles River Media.

Rollings, A., Adams, E., 2003. On Game Design.

Adams, E., 2002. Balancing Games with Positive Indianapolis: New Riders.
Feedback. Gamasutra.com.
Arey, D., Wells, E., 2001. Balancing Act: The Art and Rollings, A., Morris, D., 2000. Game Architecture and
Science of Dynamic Difficulty Adjustment. 2001 Game Design. Scottsdale, Arizona: Corliolis.
Developers Conference, San Jose
Rouse, R., 2001. Game Design: Theory and Practic,
Bethke, E., 2003. Game Development and Production. Plano, Texas: Wordware.
Plano, Texas: Wordware.
Stengel, R. F., 1994. Optimial Control and Estimation.
Castronova, E., 2001. Virtual Worlds: A First-Hand New York: Dover.
Account of Market and Society on the Cyberien Fronier.
Working Paper, Center for Economic Studies and IFO Sengers, P., 1998. Anti-Boxology: Agent Design in
Institute for Economic Research: Munich, Germany. Cultural Context. PhD thesis, Carnigie Melon University,
Pittsburg.
Carpenter, A., 2003. Applying Risk Analysis to Play-
Balance RPGs. Gamasutra.com. Simpson, Z., 1999. The In-game Economics of Ultima
Online. Game Developers Conference: 2000, San Jose.
7
And then - we’ll try it on our parents!