Slides Prepared by

JOHN S. LOUCKS
St. Edward’s University

© 2003 Thomson/South-Western 1
Slide
 Chapter 18
Dynamic Programming

 Dynamic Programming Overview

 Dynamic Programming Notation
 Backwards Recursion
 3 Applications of Dynamic Programming
 A Production and Inventory Control Problem

© 2003 Thomson/South-Western 2
Slide
 Dynamic Programming

 Dynamic programming (DP) is an approach to problem

solving which permits decomposing of the original
problem into a series of several smaller subproblems.
 To successfully apply DP, the original problem must be
viewed as a multistage decision problem.
 Defining the stages of a DP problem is sometimes
obvious, but at other times this requires subtle
reasoning.

© 2003 Thomson/South-Western 3
Slide
 Dynamic Programming

 The power of DP is that one need solve only a small

portion of all subproblems, due to Bellman's principle of
optimality.
 Bellman’s principle states that regardless of what
decisions were made at previous stages, if the decision to
be made at stage n is to be part of an overall optimal
solution, then the decision made at stage n must be
optimal for all remaining stages.

© 2003 Thomson/South-Western 4
Slide
 Dynamic Programming Notation

 At each stage, n, of the dynamic program, there is:

•a state variable, xn
•an optimal decision variable, dn
 For each value of xn and dn at stage n, there is:
•a return function value, rn(xn,dn)
 The output of the process at stage n is:
•the state variable for stage n-1, xn-1
•xn-1 is calculated by a stage transformation function,
tn(xn,dn)
 The optimal value function, fn(xn), is the cumulative
return starting at stage n in state xn and proceeding to
stage 1 under an optimal strategy.

© 2003 Thomson/South-Western 5
Slide
 Backwards Recursion

 Generally, a dynamic programming problem is solved

by starting at the final stage and working backwards to
the initial stage, a process called backwards recursion.
 The following recursion relation can be used to
operationalize the principle of optimality:

fn(xn) = MAX {rn(xn,dn) + fn -1(tn(xn , dn))}

dn
 A problem is solved beginning at stage 0 with the
boundary condition f0(x0) = 0, and working backwards
to the last stage, N.

© 2003 Thomson/South-Western 6
Slide
 Three Applications of Dynamic
Programming Problems
 Shortest Route Problem
In solving a shortest route problem using
dynamic programming, one should consider the
network as a series of stages with a unique subset of
nodes corresponding to each stage. The state
variables correspond to the different nodes at each
stage.

© 2003 Thomson/South-Western 7
Slide
 Three Applications of Dynamic
Programming Problems
 Knapsack or Cargo Loading Problem
The knapsack problem seeks to determine the
optimal number of each of N items (which must not be
fractional) to select in order to maximize profit subject to
an overall capacity constraint. In solving a knapsack
problem using dynamic programming, the stages
correspond to the different items being placed into a
knapsack. The state variables correspond to the capacity
available at the stage.

© 2003 Thomson/South-Western 8
Slide
 Three Applications of Dynamic
Programming Problems
 Production and Inventory Control Problems
In production and inventory control problems, the
stages correspond to time periods and the state
variables generally will refer to the amounts of
inventory on hand at the beginning of each stage.

© 2003 Thomson/South-Western 9
Slide
 Example: Dicom Corporation

 Production and Inventory Control Problem

Dicom Corporation wishes to determine a
production schedule for its new Model 44/12 robotic
welder. Because of differences in parts availability
and spare production capacity, the cost of producing
the machines will vary from month to month.
The holding cost for each unsold welder still in
inventory at the end of the month is $500,000.
Corporate policy dictates that the maximum number
of welders allowed in inventory at the end of any
month is 8.
Additional data appears on the next slide.

© 2003 Thomson/South-Western 10
Slide
 Example: Dicom Corporation

Cost of Maximum
Production Production Sales
Per Machine Level Demand
Month (in $100,000's) for Month (in Units)
August 32 4 2
September 18 3 1
October 26 4 5
November 45 5 3

Determine an optimal 4-month production

schedule for the Model 44/12 welder.

© 2003 Thomson/South-Western 11
Slide
 Example: Dicom Corporation

 4-Stage Dynamic Programming Problem

Working backwards, let stage 1 correspond to
November, stage 2 to October, etc. The following
data (costs in $100,000's) can be inferred:
Product. Holding
Product. Storage Cost Cost
Month Dem. Capacity Capacity Per Unit Per unit
(n) Dn Pn Wn Cn Hn
1 3 5 8 45 5
2 5 4 8 26 5
3 1 3 8 18 5
4 2 4 8 32 5

© 2003 Thomson/South-Western 12
Slide
 Example: Dicom Corporation

 State Variable Defined

xn = number of computers in inventory at the
beginning of month n.
x4 = 0 (Since the computer is new, there will be no
inventory at the start of August.)
 Decision Variable Defined
dn = production quantity for month n.

© 2003 Thomson/South-Western 13
Slide
 Example: Dicom Corporation

 Stage Transformation Function

Then the stage transformations for months 0
through 4 can be defined by:
(Previous month's inventory)
+ (Production this month)
 (Demand this month)

xn-1 = xn + dn - Dn

© 2003 Thomson/South-Western 14
Slide
 Example: Dicom Corporation

 Return Function
rn(xn,dn) = (Production cost) + (Holding cost) for month n
Production cost = (Production cost per unit)
x (Number of units produced in month n)
Holding cost = (Holding cost per unit)
x (Ending inventory for month n)

Hence, rn(xn,dn) = Cn dn + Hn(xn + dn - Dn)

© 2003 Thomson/South-Western 15
Slide
 Example: Dicom Corporation

 Return Function (continued)

rn(xn,dn) = Cn dn + Hn(xn + dn - Dn)

This gives: r1(x1,d1) = 50d1 + 5x1 - 15

r2(x2,d2) = 31d2 + 5x2 - 25
r3(x3,d3) = 23d3 + 5x3 - 5
r4(x4,d4) = 37d4 + 5x4 – 10

© 2003 Thomson/South-Western 16
Slide
 Example: Dicom Corporation

 Restrictions on xn and dn
•Restriction (1)
Because backordering is not allowed, we must be
able to meet the sales demand in any month.

xn + dn > Dn

© 2003 Thomson/South-Western 17
Slide
 Example: Dicom Corporation

 Restrictions on xn and dn (continued)

•Restriction (2)
Because there is a maximum storage of Wn at each
stage n, the total inventory at the end of any month
cannot exceed Wn .

xn + dn - Dn < Wn
or
xn + dn < Wn + Dn

© 2003 Thomson/South-Western 18
Slide
 Example: Dicom Corporation

 Restrictions on xn and dn (continued)

•Restriction (3)
The amount produced in any given month cannot
exceed the production capacity for that month.

dn < Pn

© 2003 Thomson/South-Western 19
Slide
 Example: Dicom Corporation

 Optimal Value Function

fn(xn) = optimal return (minimal cost) for stages 1
through n given one starts stage n with xn
welders in inventory

fn(xn) = Min {rn(xn ,dn) + fn -1(xn -1)}

dn

where dn is constrained by restrictions (1), (2), and (3).

© 2003 Thomson/South-Western 20
Slide
 Example: Dicom Corporation

 Stage 1 (November)
Because f0(x0) = 0 is a boundary condition:

f1(x1) = Min r1(x1, d1)

d1
Using the expression for r1(x1, d1) and restrictions (1), (2), (3):

f1(x1) = Min 5x1 + 50d1 - 15

s.t. x1 + d1 > 3 (1)
x1 + d1 < 11 (2)
d1 < 5 (3)
and, d1 > 0

© 2003 Thomson/South-Western 21
Slide
 Example: Dicom Corporation

 Tabulated Values for 50d1 +5x1 –15

d1
x1 0 1 2 3 4 5 d1* f1(x1)
0 135 185 235 3 135
1 90 140 190 240 2 90
2 45 95 145 195 245 1 45
3 0 50 100 150 200 250 0 0

Note: Having x1 > 3 would result in having welders in

inventory at the end of November which is undesirable.

© 2003 Thomson/South-Western 22
Slide
 Example: Dicom Corporation

 Stage 2 (October)

f2(x2) = Min 5x2 + 31d2 -25 + f1(x1)

d2

Given the restrictions, the subproblem is:

f2(x2) = Min 5x2+ 31d2 -25 + f1(x2+ d2 - 5)

s.t. x2 +d2 > 5 (1)
x2 +d2 < 13 (2)
d2 < 4 (3)
and d2 > 0

© 2003 Thomson/South-Western 23
Slide
 Example: Dicom Corporation

 Tabulated Values for 5x2 + 31d2 -25 + f1(x1)

d2
x2 0 1 2 3 4 d2* f2(x2) x2+d2*-5 = x1
1 239 4 239 0
2 213 199 4 199 1
3 187 173 159 4 159 2
4 161 147 133 119 4 119 3
5 135 121 107 93 3 93 3
6 95 81 67 2 67 3
7 55 41 1 41 3
8 15 0 15 3
Note: x2= 0 is infeasible.
© 2003 Thomson/South-Western 24
Slide
 Example: Dicom Corporation

 Stage 3 (September)

f3(x3) = Min 5x3 + 23d3 - 5 + f2(x2)

d3

Given the restrictions, the subproblem is:

f3(x3) = Min 5x3 + 23d3 - 5 + f2(x3 + d3 - 1)

s.t. x3 + d3 > 1 (1)
x3 + d3 < 9 (2)
d3 < 3 (3)
and d3 > 0

© 2003 Thomson/South-Western 25
Slide
 Example: Dicom Corporation

 Tabulated Values for 5x3 + 23d3 - 5 + f2(x3 + d3 -1)

d3
x3 0 1 2 3 d3* f3(x3) x3 + d3*-1 = x2
0 280 263 3 263 2
1 263 245 228 3 228 3
2 244 227 210 193 3 193 4
3 209 192 175 172 3 172 5
4 174 157 154 151 3 151 6
5 139 136 133 130 3 130 7
6 118 115 112 109 3 109 8
7 97 94 91 1 91 8
8 76 73 1 73 8

© 2003 Thomson/South-Western 26
Slide
 Example: Dicom Corporation

 Stage 4 (August)
f4(x4) = Min 5x4 + 37d4 - 10 + f3(x3)
d4

Given the restrictions, the subproblem is:

f4(x4) = Min 5x4 + 37d4 - 10 + f3(x4 + d4 - 2)

s.t. x4 + d4 > 2 (1)
x4 + d4 < 10 (2)
d4 < 4 (3)
and d4 > 0

© 2003 Thomson/South-Western 27
Slide
 Example: Dicom Corporation

 Tabulated Values for 5x4 + 37d4 - 10 + f3(x4 + d4 - 2)

d4
x4 0 1 2 3 4 d4* f4 (x4 ) x4+d4*-2 = x3

0 327 329 331 2 327 0

Note: Because August starts with x4 = 0 inventory on

hand, compute table only for x4 = 0.

© 2003 Thomson/South-Western 28
Slide
 Example: Dicom Corporation

 Solution Summary
Working backwards through the tables, one can
determine the optimal solution that gives the minimum
cost of $327 x 100,000 = $32,700,000.

Inventory On-
Hand Beginning
Stage Month Produce (dn*) Next Month xn-1
4 August 2 0
3 September 3 2
2 October 4 1
1 November 2 0

© 2003 Thomson/South-Western 29
Slide
 End of Chapter 18

© 2003 Thomson/South-Western 30
Slide