Economics 2010c: Lecture 1

Introduction to Dynamic Programming

David Laibson

9/02/2014
Outline of my half-semester course:

1. Discrete time methods (Bellman Equation, Contraction Mapping Theorem,

and Blackwell’s Sufficient Conditions, Numerical methods)

• Applications to growth, search, consumption, asset pricing

2. Continuous time methods (Bellman Equation, Brownian Motion, Ito Process,

and Ito’s Lemma)

• Applications to search, consumption, price-setting, investment, indus-

trial organization, asset-pricing
Outline of today’s lecture:

1. Introduction to dynamic programming

2. The Bellman Equation

3. Three ways to solve the Bellman Equation

4. Application: Search and stopping problem

1 Introduction to dynamic programming.

• Course emphasizes methodological techniques and illustrates them through

applications.

• We start with discrete-time dynamic optimization.

• Is optimization a ridiculous model of human behavior? Why or why not?

• Today we’ll start with an ∞-horizon stationary problem:

The Sequence Problem (cf. Stokey and Lucas)
Notation:

 is the state vector at date 

 ( +1) is the flow payoff at date  ( is ‘stationary’)

  is the exponential discount function

 is referred to as the exponential discount factor

The discount rate is the rate of decline of the discount function, so

h i
   
 ≡ − ln  = − 


Note that exp(−) =  and exp(−) =  

Definition of Sequence Problem: Find () such that
∞
X
(0) = sup   ( +1)
{+1 }∞
=0 =0

subject to +1 ∈ Γ() with 0 given.

Remark 1.1 When I omit time subscripts, this implies that an equation holds
for all relevant values of . In the statement above, +1 ∈ Γ() implies,
+1 ∈ Γ() for all  = 0 1 2 
Example 1.1 Optimal growth with log utility and Cobb-Douglas technology:
∞
X
sup   ln()
{}∞
=0 =0
subject to the constraints,   ≥ 0  + +1 =  and 0 given.

Translate this problem into Sequence Problem notation by (1) eliminating re-
dundant variables and (2) introducing constraint correspondence Γ

Example 1.2 Optimal growth translated into Sequence Problem notation:

∞
X
(0) = sup   ln( − +1)
{+1}∞
=0 =0

such that +1 ∈ [0 ] ≡ Γ() and 0 given.

2 Bellman Equation

Compare Sequence Problem and Bellman Equation.

Definition: Bellman Equation expresses the value function as a combination

of a flow payoff and a discounted continuation payoff:
() = sup { ( +1) + (+1)} ∀
+1∈Γ()

• Flow payoff is  ( +1)

• Current value function is () Continuation value function is (+1)

• Equation holds for all (feasible) values of 

• We call (·) the solution to the Bellman Equation.

• Note that any old function won’t solve the Bellman Equation.

• We haven’t yet demonstrated that there exists even one function (·) that
will satisfy the Bellman equation.

• We will show that the (unique) value function defined by the Sequence
Problem is also the unique solution to the Bellman Equation.
A solution to the Sequence Problem is also a solution to the Bellman Equation.
∞
X
(0) = sup   ( +1)
+1∈Γ() =0
⎧ ⎫
⎨ ∞
X ⎬
= sup  (0 1) +   ( +1)
+1∈Γ()
⎩ ⎭
=1
⎧ ⎫
⎨ ∞
X ⎬
= sup  (0 1) +   −1 ( +1)
+1∈Γ()
⎩ ⎭
=1
⎧ ⎫
⎨ ∞
X ⎬
= sup  (0 1) +  sup   (+1 +2)
1∈Γ(0)
⎩ +1 ∈Γ() =0
⎭

= sup { (0 1) + (1)}

1∈Γ(0)
A solution to the Bellman Equation is also a solution to the Sequence Problem.
(0) = sup { (0 1) + (1)}
1∈Γ(0)
= sup { (0 1) +  [ (1 2) + (2)]}
+1∈Γ()
..
n o
= sup  (0 1) + · · · +  −1 
 (−1 ) +  ()
+1∈Γ()
∞
X
= sup   ( +1)
+1∈Γ() =0

Sufficient condition: lim→∞  () = 0 ∀ feasible  sequences (Stokey

and Lucas Thm. 4.3).

In summary, a solution to the Bellman Equation will also be a solution to the

Sequence Problem and vice versa.
Example 2.1 Optimal growth in Sequence Problem notation:
∞
X
(0) = sup   ln( − +1)
{+1}∞
=0 =0

such that +1 ∈ [0 ] ≡ Γ() and 0 given.

Optimal growth in Bellman Equation notation:

() = sup {ln( − +1) + (+1)} ∀

+1∈Γ()
3 Solving the Bellman Equation

• Three methods

1. guess a solution (that’s no typo)

2. iterate functional operator analytically (what’s a functional operator?)

3. iterate functional operator numerically

• Method 1 today.

• Guess a function (), and then check to see that this function satisfies
the Bellman Equation at all possible values of 
• For our growth example, guess that the solution of the growth problem
takes the form:
() =  +  ln()
where  and  are constants for which we need to find solutions.

• Here value function inherits functional form of utility function (ln).

• To solve for constants rewrite Bellman Equation:

() = sup {ln( − +1) + (+1)} ∀

+1∈Γ()

 +  ln() = sup {ln( − +1) +  [ +  ln(+1)]} ∀

+1∈Γ()
First order condition (FOC) on the right-hand-side of the Bellman Equation:
 ( +1)
+  0(+1) = 0
+1
Envelope Theorem:
0  ( +1)
 () = 

Heuristic Proof of Envelope Theorem:
 ( +1)  ( +1) +1 
 0() = + +  0(+1) +1
 +1  
" #
 ( +1)  ( +1) +1
= + + 0(+1)
 +1 
 ( +1)
= 

Problem Set 1 asks you to use the FOC and the Envelope Theorem to solve for
 and . You will also confirm that

() =  +  ln()
is a solution to the Bellman Equation.
4 Search and optimal stopping

Example 4.1 An agent draws an offer,  from a uniform distribution with

support in the unit interval. The agent can either accept the offer and realize
net present value  (ending the game), or the agent can reject the offer and
draw again a period later. All draws are independent. Rejections are costly
because the agent discounts the future exponentially with discount factor .
This game continues until the agent receives an offer that she is willing to
accept.

• The Bellman equation for this problem is (relatively) easy to write:

() = max{  [(+1)]} (1)

Our problem is to find the value function (·) that solves equation (1). We’ll
also want to find the associated policy rule.

Definition: A policy is a function that maps  to the action space.

Definition: An optimal policy achieves payoff () for all feasible .

Proposition: In the search and optimal stopping problem, the threshold pol-
icy with cutoff ∗ is a best response to any continuation value function, b if
and only if (iff)
∗ =  [b(+1)] 

Proof: Optimization generates the following policy:

ACCEPT iff   ∗ =  [b(+1)]
REJECT iff   ∗ =  [b(+1)]
If  = ∗ =  [b(+1)]  then ACCEPT and REJECT generate the same
payoff. ¥
• Find threshold ∗ so that the associated value function,
( )
 if  ≥ ∗
() =  (2)
∗ if  ≤ ∗
satisfies the Bellman Equation.

• In other words, find the value of ∗ so that () (defined in equation 2)

solves the Bellman Equation (equation 1).
If  = ∗ you should be indifferent between stopping and continuing.
(∗) = ∗
= (+1)
Z =∗ Z =1
=  ∗ ()  +    () 
=0 =∗
1 ∗ 2 1
=  ( ) + 
2 2

So the final result is

∗ h ∗ 2 i
 = ( ) + 1
2
which has solution
µ q ¶
∗ =  −1 1 − 1 − 2 

Always think about comparative statics and sensibility of the answer.

 Optimal threshold in stopping problem
1

0.9
converges to 1 as
discount rate goes to 0
0.8

0.7
optimal threshold

0.6

0.5

0.4

0.3
converges to 0 as
discount rate goes to ∞
0.2

0.1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
discount rate = -ln(delta)
Outline of today’s lecture:

1. Introduction to dynamic programming

2. The Bellman Equation

3. Three ways to solve the Bellman Equation

4. Application: Search and stopping problem