ECON7020: MACROECONOMIC THEORY I Martin Boileau

A CHILD'S GUIDE TO OPTIMAL CONTROL THEORY

1. Introduction

This is a simple guide to optimal control theory. In what follows, I borrow freely from
Kamien and Shwartz (1981) and King (1986).

2. The Finite Horizon Problem

Optimal control theory is useful to solve continuous time optimization problems of the
following form:
Z T
max F (x(t); u(t); t) dt (P )
0

subject to

x_ i = Qi (x(t); u(t); t); i = 1; : : : ; n; (1)

xi (0) = xi0 ; i = 1; : : : ; n; (2)
xi (T ) ¸ 0; i = 1; : : : ; n; (3)
u(t) 2 ¨; ¨ 2 IRm : (4)

where,
x(t) is a n-vector of state variables (xi (t)). These describes the state of the system at any
point in time. Also, note that dxi =dt = x_ i .
u(t) is a m-vector of control variables. These are the choice variables in the optimization
problem.
F (¢) is a twice continuously di®erentiable objective function. Throughout, we assume that
this function is time additive.
Qi (¢) are twice continuously di®erentiable transition functions for each state variable.

1
 The di®erent constraints are:
i. Equation (1) de¯nes the transition equations for each state variables. It describes how
the state variables evolve over time.
ii. Equation (2) show the initial conditions for each state variable. In this, xi0 are
constants.
iii. Equation (3) show the terminal conditions for each state variable.
iv. Equation (4) de¯nes the feasible set for control variables.
We can now discuss the Pontryagin maximum principle. This principle says that we
can solve the optimization problem P using a Hamiltonian function H over one period. A
version of the Theorem is (see Kamien and Schwartz):
Theorem In order that x¤ (t), u¤ (t) be optimal for the problem P , it is necessary that
there exist a constant ¸0 and continuous functions ¤(t) = ( ¸1 (t); : : : ; ¸n (t) ), where for all
0 · t · T we have ¸0 6= 0 and ¤(t) 6= 0 such that for every 0 · t · T

H (x¤ (t); u; ¤(t); t) · H (x¤ (t); u¤ (t); ¤(t); t) ;

where the Hamiltonian function H is de¯ned by

n
X
H (x; u; ¤; t) = ¸0 F (x; u; t) + ¸i Qi (x; u; t):
i=1

Except at points of discontinuity of u¤t ,

@H(¢)
= ¡¸_ i (t); i = 1; : : : ; n:
@xi

Furthermore, ¸0 = 0 or ¸0 = 1 and, ¯nally, the following transversality conditions are

satis¯ed:
¸i (T ) ¸ 0; ¸i (T )x¤i (T ) = 0; i = 1; : : : ; n:

3. The Present Value Hamiltonian

The previous theorem suggests that one can solve problem P by simply setting up the
following present value Hamiltonian:
n
X
H (x; u; ¤; t) = F (x; u; t) + ¸i Qi (x; u; t):
i=1

2
In the above, we call ¸i (t) co-state variables. They are analogous to Lagrange multipliers.
The necessary conditions for a maximum are:

@H(¢)
= 0; k = 1; : : : ; m;
@uk
@H(¢)
= ¡¸_ i (t); i = 1; : : : ; n;
@xi
¸i (T ) ¸ 0; ¸i (T )x¤i (T ) = 0; i = 1; : : : ; n:

The su±cient conditions for a maximum are that the Hamiltonian function be concave in
x and u.
Finally, in some cases, our optimization problem will also include other constraints:
Z T
max F (x(t); u(t); t) dt (P 1)
0

subject to
x_ i = Qi (x(t); u(t); t); i = 1; : : : ; n;
Gj (x(t); u(t); t) ¸ 0; j = 1; : : : ; q;
xi (0) = xi0 ; i = 1; : : : ; n;
xi (T ) ¸ 0; i = 1; : : : ; n;
u(t) 2 ¨; ¨ 2 IRm :
In that case, we can write problem P 1 as a Lagrangian:
q
X
L(x; u; ¤; ©; t) = H(x; u; ¤; t) + Áj Gj (x; u; t)
j=1
n
X q
X
= F (x; u; t) + ¸i Qi (x; u; t) + Áj Gj (x; u; t);
i=1 j=1

where © = ( Á1 ; : : : ; Áq ). In this case, the necessary conditions for a maximum are:

@L(¢)
= 0; k = 1; : : : ; m;
@uk
@L(¢)
= ¡¸_ i (t); i = 1; : : : ; n;
@xi
¸i (T ) ¸ 0; ¸i (T )x¤i (T ) = 0; i = 1; : : : ; n:
Áj (t) ¸ 0; Áj (t)Q(x(t); u(t); t) = 0; j = 1; : : : ; q:

The su±cient conditions are as before.

3
4. An Economic Example

Consider the following problem:

Z T
max e¡½t u (c(t)) dt
0
subject to
a(t)
_ = ra(t) ¡ c(t);
a(0) = a0 ;
a(T ) ¸ aT :
The present value Hamiltonian is

H(a; c; ¸; t) = e¡½t u (c) + ¸ [ra ¡ c] :

The ¯rst-order conditions are

@H
= 0 =) e¡½t u0 (c(t)) ¡ ¸(t) = 0;
@c
@H _ _
= ¡¸(t) =) ¸(t)r = ¡¸(t);
@a
¸(T ) ¸ 0; ¸(T )a(T ) = 0:

A simple interpretation of these ¯rst-order conditions is to relate the growth rate of

consumption, gc (t), to the elasticity of intertemporal substitution, ¾(c(t)), and the market
and subjective discount rates, r and ½. To do this, ¯rst note that

_
¸(t) = ¡r¸(t) = ¡re¡½t u0 (c(t)):

Then, totally di®erentiating the ¯rst ¯rst-order condition yields

¡½e¡½t u0 (c(t)) + e¡½t u00 (c(t)) c(t) _

_ ¡ ¸(t) = 0:

De¯ne the growth rate of consumption as gc = (1=c)dc=dt:

c(t)
_ u0 (c(t))
gc (t) = = ¡(r ¡ ½) ;
c(t) c(t)u00 (c(t))
such that
gc (t) = ¾(c(t))(r ¡ ½);

where ¾(c(t)) = ¡u0 (c(t))=[c(t)u00 (c(t))] > 0. Thus, consumption grows whenever the
market discount rate is larger than the subjective discount rate. In that case, the consumer
is less impatient than the market.

4
5. The In¯nite Horizon Problem

In this section we consider an extension of the ¯nite horizon problem to the case of in¯nite
horizon. To simplify the exposition, we will focus our attention to stationary problems.
The assumption of stationarity implies that the main functions of the problem are
time invariant:
F (x(t); u(t); t) = F (x(t); u(t));
Gi (x(t); u(t); t) = Gi (x(t); u(t));
Qj (x(t); u(t); t) = Qj (x(t); u(t)):
In general, to obtain an interior solution to our optimization problem, we require the
objective function to be bounded away from in¯nity. One way to achieve boundedness is
to assume discounting. However, the existence of a discount factor is neither necessary nor
su±cient to ensure boundedness. Thus, we de¯ne

F (x(t); u(t)) = e¡½t f (x(t); u(t)):

Our general problem becomes:

Z 1
max e¡½t f (x(t); u(t)) dt (P 2)
0
subject to

x_ i = Qi (x(t); u(t)); i = 1; : : : ; n; (1)

Gj (x(t); u(t)) ¸ 0; j = 1; : : : ; q;
xi (0) = xi0 ; i = 1; : : : ; n; (2)

The present value Hamiltonian is

n
X
¡½t
H (x; u; ¤) = e f (x; u) + ¸i Qi (x; u):
i=1

The Lagrangian is
n q
X X
L(x; u; ¤; ©) = e¡½t f (x; u) + ¸i Qi (x; u) + Áj Gj (x; u):
i=1 j=1

The ¯rst-order conditions are:

@L(¢)
= 0; k = 1; : : : ; m;
@uk
@L(¢)
= ¡¸_ i (t); i = 1; : : : ; n;
@xi
lim ¸i (T ) ¸ 0; lim ¸i (T )x¤i (T ) = 0; i = 1; : : : ; n:
T !1 T !1
Áj (t) ¸ 0; Áj (t)Q(x(t); u(t)) = 0; j = 1; : : : ; q:

5
The su±cient conditions are as before.

6. The Current Value Hamiltonian

Any discounted optimal control problem can be solved using both a present value Hamil-
tonian or a current value Hamiltonian. We sometime prefer the current value Hamiltonian
for its simplicity.
We obtain the current value Hamiltonian as follows:

H(x; u; ª) = e½t H(x; u; ¤);

where Ãi = e½t ¸i . The current value Hamiltonian is thus

n
X
H (x; u; ª) = f(x; u) + Ãi Qi (x; u):
i=1

The transformation from present value to current value a®ects the ¯rst-order condi-
tions. To see this, let us restate our problem P 2 using the current value Hamiltonian. The
problem is
Z 1
max e¡½t f (x(t); u(t)) dt (P 2)
0

subject to

x_ i = Qi (x(t); u(t)); i = 1; : : : ; n; (1)

Gj (x; u) ¸ 0; j = 1; : : : ; q;
xi (0) = xi0 ; i = 1; : : : ; n; (2)

The current value Hamiltonian is

n
X
H (x; u; ª) = f(x; u) + Ãi Qi (x; u):
i=1

The Lagrangian is

n
X q
X
L(x; u; ª; ©) = f (x; u) + Ãi Qi (x; u) + Áj Gj (x; u):
i=1 j=1

6
The ¯rst-order conditions are:
@L(¢)
= 0; k = 1; : : : ; m;
@uk
@L(¢) h i
= ¡ Ã_ i (t) ¡ ½Ãi (t) ; i = 1; : : : ; n;
@xi
lim e¡½T Ãi (T ) ¸ 0; lim e¡½T Ãi (T )x¤i (T ) = 0; i = 1; : : : ; n:
T !1 T !1
Áj (t) ¸ 0; Áj (t)Q(x(t); u(t)) = 0; j = 1; : : : ; q:

The su±cient conditions are as before.

7. The Economic Example in In¯nite Horizon

Consider the following problem:

Z 1
max e¡½t u (c(t)) dt
0

subject to
a(t)
_ = ra(t) ¡ c(t);
a(0) = a0 :
Abstracting from the time subscript, the current value Hamiltonian is

H(a; c; Ã) = u(c) + Ã [ra ¡ c] :

Its ¯rst-order conditions are

@H
= 0 =) u0 (c) ¡ Ã = 0;
@c
@H h i h i
= ¡ Ã_ ¡ ½Ã =) Ãr = ¡ Ã_ ¡ ½Ã ;
@a
lim e¡½t Ã(T ) ¸ 0; lim e¡½t Ã(T )a(T ) = 0:
T !1 T !1

Once again, the interpretation of these ¯rst-order conditions relates consumption

growth to the elasticity of intertemporal substitution, the market discount rate, and the
subjective discount rate, r and ½. To show this, ¯rst note that

Ã_ = ¡(r ¡ ½)Ã = ¡(r ¡ ½)u0 (c):

Then, totally di®erentiating the ¯rst ¯rst-order condition yields

u00 (c)c_ = Ã_ = ¡(r ¡ ½)u0 (c);

7
such that
c(t)
_ u0 (c(t))
gc (t) = = ¡(r ¡ ½)
c(t) c(t)u00 (c(t))
or
gc (t) = ¾(c(t))(r ¡ ½);

where ¾(c(t)) = ¡u0 (c(t))=[c(t)u00 (c(t))] > 0. Thus, consumption grows whenever the mar-
ket discount rate is larger than the subjective discount factor. In that case, the consumer
is less impatient than the market.

8
8. Summary

To summarize, here is the simple cookbook. Assume that you face the following problem:
Z 1
max e¡½t f (x(t); u(t)) dt
0

subject to
x_ i = Qi (x(t); u(t)); i = 1; : : : ; n;
xi (0) = xi0 ; i = 1; : : : ; n:
There are two ways to proceed:
1. Present Value Hamiltonian
Step 1 Write the present value Hamiltonian:
n
X
¡½t
H (x; u; ¤) = e f(x; u) + ¸i Qi (x; u):
i=1

Step 2 Find the ¯rst-order conditions:

@H(¢)
= 0; k = 1; : : : ; m;
@uk
@H(¢)
= ¡¸_ i (t); i = 1; : : : ; n;
@xi
lim ¸i (T ) ¸ 0; lim ¸i (T )x¤i (T ) = 0; i = 1; : : : ; n:
T !1 T !1

2. Current Value Hamiltonian

Step 1 Write the current value Hamiltonian:
n
X
H (x; u; ª) = f (x; u) + Ãi Qi (x; u):
i=1

Step 2 Find the ¯rst-order conditions:

@H(¢)
= 0; k = 1; : : : ; m;
@uk
@H(¢) h i
= ¡ Ã_ i (t) ¡ ½Ãi (t) ; i = 1; : : : ; n;
@xi
lim e¡½T Ãi (T ) ¸ 0; lim e¡½T Ãi (T )x¤i (T ) = 0; i = 1; : : : ; n:
T !1 T !1

9
References

Kamien, Morton I. and Nancy L. Schwartz, 1981, Dynamic Optimization: The Calculus
of Variations and Optimal Control in Economics and Management, New York: North
Holland.

King, Ian P., 1986, A Cranker's Guide to Optimal Control Theory, mimeo Queen's Uni-
versity.

10