Z00_REND1011_11_SE_MOD2 PP2.

QXD

2/22/11

12:18 PM

Page 1

MODULE

Dynamic Programming

LEARNING OBJECTIVES
After completing this module, students will be able to:
1. Understand the overall approach of dynamic
programming.
2. Use dynamic programming to solve the shortestroute problem.
3. Develop dynamic programming stages.

4. Describe important dynamic programming

terminology.
5. Describe the use of dynamic programming in solving
knapsack problems.

MODULE OUTLINE
M2.1 Introduction
M2.2 Shortest-Route Problem Solved Using Dynamic
Programming
M2.3 Dynamic Programming Terminology
M2.4 Dynamic Programming Notation
M2.5 Knapsack Problem

Summary Glossary Key Equations Solved Problem Self-Test Discussion Questions and Problems Case
Study: United Trucking Internet Case Study Bibliography

M2-1

Z00_REND1011_11_SE_MOD2 PP2.QXD

M2-2

M2.1

2/22/11

12:18 PM

Page 2

MODULE 2 DYNAMIC PROGRAMMING

Introduction

Dynamic programming
breaks a difficult problem
into subproblems.

Dynamic programming is a quantitative analysis technique that has been applied to large, complex problems that have sequences of decisions to be made. Dynamic programming divides
problems into a number of decision stages; the outcome of a decision at one stage affects the decision at each of the next stages. The technique is useful in a large number of multiperiod business problems, such as smoothing production employment, allocating capital funds, allocating
salespeople to marketing areas, and evaluating investment opportunities.
Dynamic programming differs from linear programming in two ways. First, there is no algorithm (as in the simplex method) that can be programmed to solve all problems. Dynamic programming is, instead, a technique that allows us to break up difficult problems into a sequence
of easier subproblems, which are then evaluated by stages. Second, linear programming is a
method that gives single-stage (one time period) solutions. Dynamic programming has the
power to determine the optimal solution over a one-year time horizon by breaking the problem
into 12 smaller one-month horizon problems and to solve each of these optimally. Hence, it uses
a multistage approach.
Solving problems with dynamic programming involves four steps:
Four Steps of Dynamic Programming
1. Divide the original problem into subproblems called stages.
2. Solve the last stage of the problem for all possible conditions or states.
3. Working backward from the last stage, solve each intermediate stage. This is done by
determining optimal policies from that stage to the end of the problem (last stage).
4. Obtain the optimal solution for the original problem by solving all stages sequentially.
In this module we show how to solve two types of dynamic programming problems: network and nonnetwork. The shortest-route problem is a network problem that can be solved by
dynamic programming. The knapsack problem is an example of a nonnetwork problem that can
be solved using dynamic programming.

M2.2

Shortest-Route Problem Solved Using Dynamic Programming

George Yates is about to make a trip from Rice, Georgia (1), to Dixieville, Georgia (7). George
would like to find the shortest route. Unfortunately, there are a number of small towns between
Rice and Dixieville. His road map is shown in Figure M2.1.
The circles on the map, called nodes, represent cities such as Rice, Dixieville, Brown, and
so on. The arrows, called arcs, represent highways between the cities. The distance in miles is
indicated along each arc. This problem can, of course, be solved by inspection. But seeing how
dynamic programming can be used on this simple problem will teach you how to solve larger
and more complex problems.

The first step is to divide the

problem into subproblems or
stages.

Step 1: The first step is to divide the problem into subproblems or stages. Figure M2.2 reveals

The second step is to solve the

last stagestage 1.

Step 2: We next solve stage 1, the last part of the network. Usually, this is trivial. We find the

the stages of this problem. In dynamic programming, we usually start with the last part of the
problem, stage 1, and work backward to the beginning of the problem or network, which is stage
3 in this problem. Table M2.1 summarizes the arcs and arc distances for each stage.

shortest path to the end of the network, node 7 in this problem. At stage 1, the shortest paths,

Z00_REND1011_11_SE_MOD2 PP2.QXD

2/22/11

12:19 PM

Page 3

M2.2

SHORTEST-ROUTE PROBLEM SOLVED USING DYNAMIC PROGRAMMING

FIGURE M2.1
Highway Map between
Rice and Dixieville

Lake City

Athens
10 Miles

5
14

Mi

2M

ile
s

7 Dixieville

5 Miles

Brown

Mi

les

s
ile

les

6M

ile

ile

2M

ile
s

Rice

10 Miles

Hope

FIGURE M2.2
Three Stages for the
George Yates Problem

Georgetown

A Node
A Branch

10

5 14

4
5
2

12
3 6

7
2

10

2
Stage 3

TABLE M2.1
Distance Along Each Arc

M2-3

Stage 2

Stage 1

STAGE

ARC

ARC DISTANCE

57

14

67

45

10

35

12

36

25

26

10

14

13

12

Z00_REND1011_11_SE_MOD2 PP2.QXD

M2-4

2/22/11

12:19 PM

Page 4

MODULE 2 DYNAMIC PROGRAMMING

FIGURE M2.3
Solution for the OneStage Problem

5 14

4
1

Minimum Distance
to Node 7 from
Node 5

14

10

5
2

12
3 6

7
2

10

Minimum Distance
to Node 7 from
Node 6

2
2

from node 5 to node 6, to node 7 are the only paths. Also note in Figure M2.3 that the minimum
distances are enclosed in boxes by the entering nodes to stage 1, node 5 and node 6. The objective is to find the shortest distance to node 7. The following table summarizes this procedure for
stage 1. As mentioned previously, the shortest distance is the only distance at stage 1.

STAGE 1

Step 3 involves moving backward

solving intermediate stages.

FIGURE M2.4
Solution for the TwoStage Problem

BEGINNING
NODE

SHORTEST DISTANCE
TO NODE 7

ARCS ALONG
THIS PATH

14

57

67

Step 3: Moving backward, we now solve for stages 2 and 3. At stage 2 we will use Figure M2.4.

Minimum Distance
to Node 7 from
Node 4

24

4
10
Minimum Distance
to Node 7 from
Node 3

14

8
12

5
2

6
2
4

10

2
2
Minimum Distance
to Node 7 from
Node 2

14

12

Z00_REND1011_11_SE_MOD2 PP2.QXD

2/22/11

12:19 PM

Page 5

M2.2

SHORTEST-ROUTE PROBLEM SOLVED USING DYNAMIC PROGRAMMING

M2-5

If we are at node 4, the shortest and only route to node 7 is arcs 45 and 57. At node 3, the
shortest route is arcs 36 and 67 with a total minimum distance of 8 miles. If we are at node 2,
the shortest route is arcs 26 and 67 with a minimum total distance of 12 miles. This information is summarized in the stage 2 table:
STAGE 2
BEGINNING
NODE

SHORTEST DISTANCE
TO NODE 7

24

ARCS ALONG
THIS PATH
45
57

36
67

12

26
67

The solution to stage 3 can be completed using the following table and the network in
Figure M2.5:
STAGE 3
BEGINNING
NODE

SHORTEST DISTANCE
TO NODE 7

ARCS ALONG
THIS PATH

13

13
36
67

FIGURE M2.5
Solution for the ThreeStage Problem

24
Minimum Distance
to Node 7 from
Node 1

13

10
14

5 14

4
1

5
2

12
3 6

7
2

10

2
2
12

Z00_REND1011_11_SE_MOD2 PP2.QXD

M2-6

12:19 PM

Page 6

MODULE 2 DYNAMIC PROGRAMMING

The fourth and final step is to

find the optimal solution after all
stages have been solved.

M2.3

2/22/11

Step 4: To obtain the optimal solution at any stage, all we consider are the arcs to the next stage

and the optimal solution at the next stage. For stage 3, we only have to consider the three arcs to
stage 2 (12, 13, and 14) and the optimal policies at stage 2, given in a previous table. This is
how we arrived at the preceding solution. When the procedure is understood, we can perform all
the calculations on one network. You may want to study the relationship between the networks
and the tables because more complex problems are usually solved by using tables only.

Dynamic Programming Terminology

Regardless of the type or size of a dynamic programming problem, there are some important
terms and concepts that are inherent in every problem. Some of the more important of them
follow:
1. Stage: a period or a logical subproblem.
2. State variables: possible beginning situations or conditions of a stage. These have also
been called the input variables.
3. Decision variables: alternatives or possible decisions that exist at each stage.
4. Decision criterion: a statement concerning the objective of the problem.
5. Optimal policy: a set of decision rules, developed as a result of the decision criteria, that
gives optimal decisions for any entering condition at any stage.
6. Transformation: normally, an algebraic statement that reveals the relationship between
stages.
In the shortest-route problem, the following transformation can be given:
Distance from the
Distance from the beginning Distance from the
beginning of a given =
of the previous stage
+ given stage to the
stage to the last node
to the last node
previous stage
This relationship shows how we are able to go from one stage to the next in solving for the
optimal solution to the shortest-route problem. In more complex problems, we can use symbols
to show the relationship between stages.
State variables, decision variables, the decision criterion, and the optimal policy can be determined for any stage of a dynamic programming problem. This is done here for stage 2 of the
George Yates shortest-route problem.

IN ACTION

Dynamic Programming in Nursery

Production Decisions

anaging a nursery that produces ornamental plants is difficult. In most cases, ornamental plants increase in value with increased growth. This value-added growth makes it difficult to
determine when to harvest the plants and place them on the
market. When plants are marketed earlier, revenues are generated earlier and the costs associated with plant growth are minimized. On the other hand, delaying the harvesting of the
ornamental plants usually results in higher prices. But are the additional months of growth and costs worth the delay?
In this case, dynamic programming was used to determine
the optimal growth stages for ornamental plants. Each stage was
associated with a possible growth level. The state variables
included acres of production of ornamental plants and carryover

plants from previous growing seasons. The objective of the

dynamic programming problem was to maximize the after-tax
cash flow. The taxes included self-employment, federal income,
earned income credit, and state income taxes.
The solution was to produce one- and three-gallon containers
of ornamental plants. The one-gallon containers are sold in the fall
and carried over for spring sales. Any one-gallon containers not
sold in the spring are combined into three-gallon container products for sale during the next season. Using dynamic programming
helps to determine when to harvest to increase after-tax cash flow.
Source: Jeffrey Stokes, et al. Optimal Marketing of Nursery Crops from
Container-Based Production Systems, American Journal of Agricultural
Economics (February 1997): 235246.

Z00_REND1011_11_SE_MOD2 PP2.QXD

2/22/11

12:19 PM

Page 7

M2.3

DYNAMIC PROGRAMMING TERMINOLOGY

M2-7

1. State variables for stage 2 are the entering nodes, which are
(a) node 2
(b) node 3
(c) node 4
2. Decision variables for stage 2 are the following arcs or routes:
(a) 45
(b) 35
(c) 36
(d) 25
(e) 26
3. The decision criterion is the minimization of the total distances traveled.
4. The optimal policy for any beginning condition is shown in Figure M2.6 and the following
table:

FIGURE M2.6
Stage 2 from the
Shortest-Route Problem

GIVEN THIS ENTERING

CONDITION

THIS ARC WILL MINIMIZE

TOTAL DISTANCE TO NODE 7

26

36

45

State
variables
are the
entering
nodes.

24

10

14
8
5
1

12
3 6
4

Decision
variables
are all
the arcs.

7
10

2
2
The optimal policy is the arc,
for any entering node, that
will minimize total distance
to the destination at this
stage.

12

Stage 2

Z00_REND1011_11_SE_MOD2 PP2.QXD

M2-8

2/22/11

12:19 PM

Page 8

MODULE 2 DYNAMIC PROGRAMMING

Figure M2.6 may also be helpful in understanding some of the terminology used in the discussion of dynamic programming.

M2.4

Dynamic Programming Notation

An input, decision, output, and

return are specified for each
stage.

In addition to dynamic programming terminology, we can also use mathematical notation to describe any dynamic programming problem. This helps us to set up and solve the problem. Consider stage 2 in the George Yates dynamic programming problem first discussed in Section
M2.2. This stage can be represented by the diagram shown in Figure M2.7 (as could any given
stage of a given dynamic programming problem).
As you can see, for every stage, we have an input, decision, output, and return. Look again
at stage 2 for the George Yates problem in Figure M2.6. The input to this stage is s2, which consists of nodes 2, 3, and 4. The decision at stage 2, or choosing which arc will lead to stage 1, is
represented by d2. The possible arcs or decisions are 45, 35, 36, 25, and 26. The output to
stage 2 becomes the input to stage 1. The output from stage 2 is s1. The possible outputs from
stage 2 are the exiting nodes, nodes 5 and 6. Finally, each stage has a return. For stage 2, the return is represented by r2. In our shortest-route problem, the return is the distance along the arcs
in stage 2. These distances are 10 miles for arc 45, 12 miles for arc 35, 6 miles for arc 36, 4
miles for arc 25, and 10 miles for arc 26. The same notation applies for the other stages and
can be used at any stage. In general, we use the following notation for these important concepts:
sn = Input to stage n
dn = Decision at stage n
rn = Return at stage n

The input to one stage is the

output from another stage.

(M2-1)
(M2-2)
(M2-3)

You should also note that the input to one stage is also the output from another stage. For
example, the input to stage 2, s2, is also the output from stage 3 (see Figure M2.7). This leads us
to the following equation:
sn - 1 = Output from stage n

FIGURE M2.7
Input, Decision, Output,
and Return for Stage 2 in
George Yatess Problem

Decision
d2

Stage 2
Input
s2

Output
s1

Return
r2

(M2-4)

Z00_REND1011_11_SE_MOD2 PP2.QXD

2/22/11

12:19 PM

Page 9

M2.5

A transformation function
allows us to go from one stage
to another.

KNAPSACK PROBLEM

M2-9

The final concept is transformation. The transformation function allows us to go from one
stage to another. The transformation function for stage 2, t2, converts the input to stage 2, s2, and
the decision made at stage 2, d2, to the output from stage 2, s1. Because the transformation function depends on the input and decision at any stage, it can be represented as t2 (s2, d2). In general, the transformation function can be represented as follows:
tn = Transformation function at stage n

(M2-5)

The following general formula allows us to go from one stage to another using the transformation function:
sn - 1 = tn1sn, dn2

The total return function

allows us to keep track of
profits and costs.

M2.5

(M2-6)

Although this equation may seem complex, it is really just a mathematical statement of the fact
that the output from a stage is a function of the input to the stage and any decisions made at that
stage. In the George Yates shortest-route problem, the transformation function consists of a
number of tables. These tables show how we could progress from one stage to another in order
to solve the problem. For more complex problems, we need to use dynamic programming notation instead of tables.
Another useful quantity is the total return at any stage. The total return allows us to keep
track of the total profit or costs at each stage as we solve the dynamic programming problem. It
can be given as follows:
fn = Total return at stage n

(M2-7)

Knapsack Problem
The knapsack problem involves the maximization or the minimization of a value, such as profits or costs. As in a linear programming problem, there are restrictions in a knapsack problem.
Imagine a knapsack or pouch that can hold only a certain weight or volume. We can place different types of items in the knapsack. Our objective is to place items in the knapsack to maximize total value without breaking the knapsack because of too much weight or a similar
restriction.

Types of Knapsack Problems

Many kinds of problems can be classified as knapsack problems. Choosing items to place in the
cargo compartment of an airplane and selecting which payloads to put on the next NASA Space
Shuttle are examples. The restriction can be volume, weight, or both. Some scheduling problems are also knapsack problems. For example, we may want to determine which jobs to complete in the next two weeks. The two-week period is the knapsack, and we want to load it with
jobs in such a way as to maximize profits or minimize costs. The restriction is the number of
days or hours during the two-week period.

Rollers Air Transport Service Problem

The objective of this problem is
to maximize profits.

Rob Roller owns and operates Rollers Air Transport Service, which ships cargo by plane to
most large cities in the United States and Canada. The remaining capacity for one of the
flights from Seattle to Vancouver is 10 tons. There are four different items that Rob can ship
between Seattle and Vancouver. Each item has a weight in tons, a net profit in thousands of
dollars, and a total number of that item that is available for shipping. This information is presented in Table M2.2.
PROBLEM SETUP Rollers Air Transport problem is an ideal problem to solve using dynamic

programming. Stage 4 will be item 1, stage 3 will be item 2, stage 2 will be item 3, and stage

Z00_REND1011_11_SE_MOD2 PP2.QXD

M2-10

2/22/11

12:19 PM

Page 10

MODULE 2 DYNAMIC PROGRAMMING

TABLE M2.2
Items to Be Shipped

ITEM

WEIGHT

PROFIT/UNIT

NUMBER AVAILABLE

$3

MODELING IN THE REAL WORLD

Defining
the Problem

Developing
a Model

Acquiring
Input Data

Developing
a Solution

Testing the
Solution

Analyzing
the Results

Implementing
the Results

Reducing Electric Production Costs

Using Dynamic Programming

Defining the Problem

The Southern Company, with service areas in Georgia, Alabama, Mississippi, and Florida, is a major
provider of electric service, with about 240 generating units. In recent years, fuel costs have increased
faster than other costs. Annual fuel costs are about $2.5 billion, representing about one-third of total expenses for the Southern Company. The problem for the Southern Company is to reduce total fuel costs.

Developing a Model
To deal with this fuel cost problem, the company developed a state-of-the-art dynamic programming
model. The dynamic programming model is embedded in the Wescouger optimization program, which is
a computer program used to control electric generating units and reduce fuel costs through better utilization of existing equipment.

Acquiring Input Data

Data were collected on past and projected electric usage. In addition, daily load/generation data were analyzed. Load/generation charts were used to investigate the fuel requirements for coal, nuclear, hydroelectric, and gas/oil.

Developing a Solution
The solution of the dynamic programming model provides both short-term planning guidelines and longterm fuel usage for the various generating units. Optimal maintenance schedules for generating units are
obtained using Wescouger.

Testing the Solution

To test the accuracy of the Wescouger optimization program, Southern used a real-time economic dispatch program. The results were a very close match. In addition, the company put the solution through an
acid test, in which seasoned operators compared the results against their intuitive judgment. Again, the
results were consistent.

Analyzing the Results

The results were analyzed in terms of their impact on the use of various fuels, the usage of various generating units, and maintenance schedules for generating units. Analyzing the results also revealed other
needs. This resulted in a full-color screen editing routine, auxiliary programs to automate data input, and
software to generate postanalysis reports.

Implementing the Results

The Southern Company implemented the dynamic programming solution. Over a seven-year period, the
results saved the company over $140 million.
Source: S. Erwin, et al. Using an Optimization Software to Lower Overall Electric Production Costs for Southern Company,
Interfaces 21, 1 (JanuaryFebruary 1991): 2741.

Z00_REND1011_11_SE_MOD2 PP2.QXD

2/22/11

12:19 PM

Page 11

M2.5

TABLE M2.3
Relationship Between
Items and Stages

ITEM

STAGE

KNAPSACK PROBLEM

M2-11

1 will be item 4. This is shown in Table M2.3. During the solution, we will be using stage
numbers.
Rollers Air Transport problem can be represented graphically (see Figure M2.8). As you
can see, each item is represented by a stage. Look at stage 4, which is item 1. The total weight
that can be used is represented by s4. This amount is 10 because we havent assigned any units
to be shipped at this time. The decision at this stage is d4 (the number of units of item 1 we will
ship). If d4 is 1, for example, we will be shipping 1 unit of item 1. Also note that r4 is the return
or profit at stage 4 (item 1). If we ship 1 unit of item 1, the profit will be $3 (see Table M2.2).
As mentioned previously, the decision variable dn, represents the number of units of each
item (stage) that can be shipped. Looking back at the original problem, we see that the problem
is constrained by the number of units. This is summarized in the following table:
STAGE

MAXIMUM VALUE OF DECISION

THE TRANSFORMATION FUNCTIONS Next, we need to look at the transformation function. The

general transformation function for knapsack problems follows:

sn - 1 = (an * sn) + (bn * dn) + cn
Note that an, bn, and cn are coefficients for the problem and that dn represents the decision at
stage n. This is the number of units to ship at stage sn.
FIGURE M2.8
Rollers Air Transport
Service Problem

Decisions
d4

s4

Stage 4
(Item 1)

r4

d3

s3

Stage 3
(Item 2)

d2

s2

r3

Stage 2
(Item 3)

r2
Returns

d1

s1

Stage 1
(Item 4)

r1

s0

Z00_REND1011_11_SE_MOD2 PP2.QXD

M2-12

2/22/11

12:19 PM

Page 12

MODULE 2 DYNAMIC PROGRAMMING

The following chart shows the transformation coefficients for Rob Rollers transport problem:
COEFFICIENTS OF
TRANSITION FUNCTION
STAGE

an

bn

cn

-1

-4

-3

-2

First, note that s4 is 10, the total weight that can be shipped. Because s4 represents the first item,
all 10 tons can be utilized. The transformation equations for the four stages are as follows:
s3
s2
s1
s0

=
=
=
=

s4
s3
s2
s1

1d4
4d3
3d2
2d1

stage 4
stage 3
stage 2
stage 1

(a)
(b)
(c)
(d)

Consider stage 3. Equation b reveals that the number of tons still available after stage 3, s2, is
equal to the number of tons available before stage 3, s3, minus the number of tons shipped at
stage 3, 4d3. In this equation, 4d3 means that each item at stage 3 weighs 4 tons.
THE RETURN FUNCTIONS Next, we will look at the return function for each stage. This is the
general form for the return function:

rn = 1an * sn2 + 1bn * dn2 + cn

Note that an, bn, and cn are the coefficients for the return function. Using this general form of
the return function, we can put the return function values in the following table:
COEFFICIENTS OF
RETURN FUNCTION

DECISIONS
STAGE

LOWER

UPPER

an

bn

cn

dn

3
2

dn

dn

dn

Z00_REND1011_11_SE_MOD2 PP2.QXD

2/22/11

12:19 PM

Page 13

M2.5

M2-13

KNAPSACK PROBLEM

The lower value for each decision is zero, and the upper value is the total number available.
The bn coefficient is the profit per item shipped. The actual return functions are:
r4
r3
r2
r1

=
=
=
=

3d4
9d3
8d2
5d1

STAGE-BY-STAGE SOLUTION As you would expect, the return at any stage rn, is equal to the

profit per unit at that stage multiplied by the number of units shipped at that stage, dn. With this
information, we can solve Rollers Air Transport problem, starting with stage 1 (item 4). The
following table shows the solution for the first stage. You may wish to refer to Figure M2.8 for
this discussion.
STAGE 1
S1

d1

r1

s0

f0

f1

S1

d1

r1

s0

f0

f1

10

10

10

10

10

10

10

10

10

10

10

10

10

10

3
4

We consider all possibilities.

f1 is the total return at the first

stage. The total return before the
first stage is f0.

STAGE 1

10

Because we dont know how many tons will be available at stage 1, we must consider all
possibilities. Thus, the number of tons available at stage 1, s1, can vary from 1 to 10. This is seen
in the first column of numbers for stage 1. The number of units that we ship at stage 1, d1, can
vary from 0 to 2. We cant go over 2 because the number available is only 2. For any decision
we compute the return at stage 1, r1, by multiplying the number of items shipped by 5, the profit
per item. The profit at this stage will be 0, 5, or 10, depending on whether 0, 1, or 2 items are
shipped. Note that the total return at this stage, f1, is the same as r1 because this is the only stage
we are considering so far. Also note that the total return before stage 1, f0, is 0 because this is
the beginning of the solution and we are shipping nothing at this point.

Z00_REND1011_11_SE_MOD2 PP2.QXD

M2-14

2/22/11

12:19 PM

Page 14

MODULE 2 DYNAMIC PROGRAMMING

The solution for stage 1 shows the best decision, the return for this stage, and the total return, given all possible number of tons available (0 to 10 tons). Using the results of stage 1, we
can now proceed to stage 2. The solution for stage 2 is as follows:

STAGE 2

STAGE 2
S2

d2

r2

s1

f1

f2

S2

d2

r2

s1

f1

f2

10

10

10

18

16

16

10

10

10

18

16

21

10

10

4
5
6

10

10

10

10

10

18

13

16

21

10

10

10

10

10

13

10

10

18

16

16

16

10

26

The solution for stage 2 is found in exactly the same way as for stage 1. At stage 2 we still
have to consider all possible number of tons available (from 0 to 10). See the s2 column (the first
column). At stage 2 (item 3) we still only have 2 units that can be shipped. Thus, d2 (second column) can range from 0 to a maximum of 2. The return for each s2 and d2 combination at stage 2,
r2, is shown in the third column. These numbers are the profit per item at this stage, 8, times the
number of items shipped. Because items shipped at stage 2 can be 0, 1, or 2, the profit at this
stage can be 0, 8, or 16. The return for stage 2 can also be computed from the return function:
r2 = 8d2.
Now look at the fourth column, s1, which lists the number of items available after stage 2.
This is also the number of items available for stage 1. To get this number, we have to subtract
the number of tons we are shipping at stage 2 (which is the tonnage per unit times the number of
units) from the number of tons available before stage 2 (s2). Look at the row in which s2 is 6 and
d2 is 1. We have 6 tons available before stage 2, and we are shipping 1 item, which weighs 3
tons. Thus we will have 3 tons still available after this stage. The s1 values can also be determined using the transformation function, which is s1 = s2 - 3d2.
The last two columns of stage 2 contain the total return. The return after stage 1 and before
stage 2 is f1. These are the same values that appeared under the f1 column for stage 1. The return after stage 2 is f2. It is equal to the return from stage 2 plus the total return before stage 2.
Stage 3 is solved in the same way as stages 1 and 2. The following table presents the
solution for stage 3; look at each row and make sure that you understand the meaning of each
value.

Z00_REND1011_11_SE_MOD2 PP2.QXD

2/22/11

12:19 PM

Page 15

M2.5

M2-15

KNAPSACK PROBLEM

STAGE 3

STAGE 3

s3

d3

r3

s2

f2

f3

s3

d3

r3

s2

f2

f3

18

18

17

21

21

10

19

10

10

21

21

13

22

13

13

10

26

26

16

25

5
6

16

16

14

8
9
10

Now we solve the last stage of the problem, stage 4. The following table shows the solution
procedure:
STAGE 4
s4

d4

r4

s3

f3

f4

10

10

26

26

22

25

21

27

18

27

12

16

28

15

13

28

18

10

28

The first thing to note is that we only have to consider one value for s4, because we know
the number of tons available for stage 4; s4 must be 10 because we have all 10 tons available.
There are six possible decisions at this stage, or six possible values for d4, because the number
of available units is 6. The other columns are computed in the same way. Note that the return after stage 4, f4, is the total return for the problem. We see that the highest profit is 28. We also
see that there are three possible decisions that will give this level of profit, shipping 4, 5, or 6
items. Thus we have alternate optimal solutions. One possible solution is as follows:
FINAL SOLUTION
STAGE

OPTIMAL DECISION

OPTIMAL RETURN

18

10

Total

28

Z00_REND1011_11_SE_MOD2 PP2.QXD

M2-16

2/22/11

12:19 PM

Page 16

MODULE 2 DYNAMIC PROGRAMMING

We start with shipping 6 units at stage 4. Note that s3 is 4 from the stage 4 calculations,
given that d4 is 6. We use this value of 4 and go to the stage 3 calculations. We find the rows in
which s3 is 4 and pick the row with the highest total return, f3. In this row d3 is 0 items with a
total return (f3) of 10. As a result, the number of units available, s2, is still 4. We next go to the
calculations for stage 2 and then stage 1 in the same way. This gives us the optimal solution already described. see if you can find one of the alternate optimal solutions.

Summary
Dynamic programming is a flexible, powerful technique. A
large number of problems can be solved using this technique,
including the shortest-route and knapsack problems. The
shortest-route problem finds the path through a network that
minimizes total distance traveled, and the knapsack problem
maximizes the value or return. An example of a knapsack problem is to determine what to ship in a cargo plane to maximize
total profits given the weight and size constraints of the cargo
plane. The dynamic programming technique requires four
steps: (1) Divide the original problem into stages, (2) solve the
last stage first, (3) work backward solving each subsequent
stage, and (4) obtain the optimal solution after all stages have
been solved.

Dynamic programming requires that we specify stages,

state variables, decision variables, decision criteria, an optimal
policy, and a transformation function for each specific problem. Stages are logical subproblems. State variables are possible input values or beginning conditions. The decision
variables are the alternatives that we have at each stage, such
as which route to take in a shortest-route problem. The decision criterion is the objective of the problem, such as finding
the shortest route or maximizing total return in a knapsack
problem. The optimal policy helps us obtain an optimal solution at any stage, and the transformation function allows us to
go from one stage to the next.

Glossary
Decision Criterion A statement concerning the objective of
a dynamic programming problem.
Decision Variable The alternatives or possible decisions that
exist at each stage of a dynamic programming problem.
Dynamic Programming A quantitative technique that
works backward from the end of the problem to the beginning of the problem in determining the best decision for a
number of interrelated decisions.
Optimal Policy A set of decision rules, developed as a result
of the decision criteria, that gives optimal decisions at any
stage of a dynamic programming problem.

Stage A logical subproblem in a dynamic programming

problem.
State Variable A term used in dynamic programming to
describe the possible beginning situations or conditions
of a stage.
Transformation An algebraic statement that shows the
relationship between stages in a dynamic programming
problem.

Key Equations
(M2-1) sn = Input to stage n
The input to stage n. This is also the output from stage
n + 1.
(M2-2) dn = Decision at stage n
The decision at stage n.
(M2-3) rn = Return at stage n
The return function, usually profit or loss, at stage n.
(M2-4) sn - 1 = Output from stage n
The output from stage n. This is also the input to stage
n - 1.

(M2-5) tn = Transformation function at stage n

A transformation function that allows us to go from
one stage to another.
(M2-6) sn - 1 = tn1sn, dn2
The general relationship that shows how the output
from any stage is a function of the input to the stage
and the decisions made at that stage.
(M2-7) fn = Total return at stage n
This equation gives the total return (profit or costs) at
any stage. It is obtained by summing the return at each
stage, rn.

Z00_REND1011_11_SE_MOD2 PP2.QXD

2/22/11

12:19 PM

Page 17

SOLVED PROBLEM

M2-17

Solved Problem
Solved Problem M2-1
Lindsey Cortizan would like to travel from the university to her hometown over the holiday season. A
road map from the university (node 1) to her home (node 7) is shown in Figure M2.9. What is the best
route that will minimize total distance traveled?
FIGURE M2.9
Road Map for
Lindsey Cortizan

39

20
6
13

28
18

36
10

22

18

Solution
The solution to this problem is identical to the one presented earlier in the chapter. First, the problem is
broken into three stages. See the network in Figure M2.10. Working backward, we start by solving stage
3. The closest and only distance from node 6 to 7 is 13, and the closest and only distance from node 5 to
node 7 is 10. We proceed to stage 2. The minimum distances are 52, 41, and 28 from node 4, node 3,
and node 2 to node 7. Finally, we complete stage 3. The optimal solution is 50. The shortest route is
1257 and is shown in the following network. This problem can also be solved using tables, as shown
following the network solution.
Problem type:
Number of stages:
Transition function type:
Recursion function type:
FIGURE M2.10
Solution for the
Lindsey Cortizan
Problem

Minimization network
3
sn - 1 = sn - dn
fn = rn + fn - 1

52
4

39
13

20
50
1

41

6
13

28

18
3

36

10

22
18

2
10
28
Stage 3

Stage 2

Stage 1

Z00_REND1011_11_SE_MOD2 PP2.QXD

M2-18

2/22/11

12:19 PM

Page 18

MODULE 2 DYNAMIC PROGRAMMING

STAGE

NUMBER OF DECISIONS

STAGE

STARTING NODE

ENDING NODE

RETURN VALUE

22

18

20

18

36

28

39

10

13

STAGE 1
s1

d1

r1

s0

f0

f1

67

13

13

57

10

10

STAGE 2
s2

d2

r2

s1

f1

f2

46

39

13

52

36

28

13

41

26

36

13

49

25

18

10

28

STAGE 3
s3

d3

r3

s2

f2

f3

14

20

52

72

13

18

41

59

12

22

28

50

FINAL SOLUTION
STAGE

OPTIMAL DECISION

OPTIMAL RETURN

12

22

25

18

57

10

Total

50

Z00_REND1011_11_SE_MOD2 PP2.QXD

2/22/11

12:19 PM

Page 19

SELF-TEST

M2-19

Self-Test

Before taking the self-test, refer back to the learning objectives at the beginning of the module, the notes in the margins, and
the glossary at the end of the module.
Use the key at the back of the book to correct your answers.
Restudy pages that correspond to any questions that you answered incorrectly or material you feel uncertain about.

1. Dynamic programming divides problems into

a. nodes.
b. arcs.
c. decision stages.
d. branches.
e. variables.
2. Possible beginning situations or conditions of a dynamic
programming problem are called
a. stages.
b. state variables.
c. decision variables.
d. optimal policy.
e. transformation.
3. The statement concerning the objective of a dynamic programming problem is called
a. stages.
b. state variables.
c. decision variables.
d. optimal policy.
e. decision criterion.
4. The first step of a dynamic programming problem is to
a. define the nodes.
b. define the arcs.
c. divide the original problem into stages.
d. determine the optimal policy.
e. none of the above.
5. In working a problem with dynamic programming, we
usually
a. start at the first part of the problem and work forward
to the next parts.
b. start at the end of the problem and work backward.
c. start at the most expensive part of the problem.
d. start at the least expensive part of the problem.
6. An algebraic statement that reveals the relationship between stages is called
a. the transformation.
b. state variables.
c. decision variables.
d. the optimal policy.
e. the decision criterion.
7. In this chapter, dynamic programming is used to solve
what type of problem?
a. quantity discount
b. just-in-time inventory
c. shortest route
d. minimal spanning tree
e. maximal flow
8. In dynamic programming terminology, a period or logical
subproblem is called
a. the transformation.
b. a state variable.

9.

10.

11.

12.

13.

14.

c. a decision variable.
d. the optimal policy.
e. a stage.
The statement that the distance from the beginning stage
is equal to the distance from the preceding stage to the
last node plus the distance from the given stage to the preceding stage is called
a. the transformation.
b. state variables.
c. decision variables.
d. the optimal policy.
e. stages.
In dynamic programming, sn is
a. the input to the stage n.
b. the decision at stage n.
c. the return at stage n.
d. the output of stage n.
e. none of the above.
The distance from the beginning stage is equal to the distance from the preceding stage to the last node plus the
distance for the given stage to the preceding stage. This
relationship is used to solve which type of problem?
a. knapsack
b. JIT
c. shortest route
d. minimal spanning tree
e. maximal flow
In using dynamic programming to solve a shortest-route
problem, the distance from one point to the next would be
called a
a. state.
b. stage.
c. return.
d. decision.
In using dynamic programming to solve a shortest-route
problem, the decision variables at one stage of the problem would be
a. the distances from one node to the next.
b. the possible arcs or routes that can be selected.
c. the number of possible destination nodes.
d. the entering nodes.
In using dynamic programming to solve a shortest-route
problem, the entering nodes would be called
a. the stages.
b. the state variables.
c. the returns.
d. the decision variables.

Z00_REND1011_11_SE_MOD2 PP2.QXD

M2-20

2/22/11

12:19 PM

Page 20

MODULE 2 DYNAMIC PROGRAMMING

Discussion Questions and Problems

M2-10 Solve the shortest-route problem for the network
shown in Figure M2.11.
M2-11 Solve the shortest-route problem for the network
shown in Figure M2.12.
M2-12 Mail Express, an overnight mail service, delivers
mail to customers throughout the United States,
Canada, and Mexico. Fortunately, Mail Express has
additional capacity on one of its cargo planes. To
maximize profits, Mail Express takes shipments
from local manufacturing plants to warehouses for
other companies. Currently, there is room for another 6 tons. The following table shows the items
that can be shipped, their weights, the expected
profit for each, and the number of available parts.
How many units of each item do you suggest that
Mail Express ship?

Discussion Questions
M2-1 What is a stage in dynamic programming?
M2-2 What is the difference between a state variable and a
decision variable?
M2-3 Describe the meaning and use of a decision criterion.
M2-4 Do all dynamic programming problems require an
optimal policy?
M2-5 Why is transformation important for dynamic programming problems?

Problems
M2-6 Refer to Figure M2.1. What is the shortest route between Rice and Dixieville if the road between Hope
and Georgetown is improved and the distance is reduce to 4 miles?
M2-7 Due to road construction between Georgetown and
Dixieville, a detour must be taken through country
roads (Figure M2.1). Unfortunately, this detour has
increased the distance from Georgetown to Dixieville to 14 miles. What should George do? Should
he take a different route?
M2-8 The Rice brothers have a gold mine between Rice
and Brown. In their zeal to find gold, they have
blown up the road between Rice and Brown. The
road will not be in service for five months. What
should George do? Refer to Figure M2.1.
M2-9 The Rice brothers (Figure M2.1) wish to determine
the potential savings from using the shortest route
from Rice to Dixieville rather than just randomly selecting any route. Use dynamic programming to find
the longest route from Rice to Dixieville. How much
farther is this than the shortest route?
FIGURE M2.11
(for Problem M2-10)

ITEMS TO BE SHIPPED
NUMBER
AVAILABLE

$3

2
4

10
2 4
5

PROFIT/UNIT

M2-13 Leslie Bessler must travel from her hometown to

Denver to see her friend Austin. Given the road map
of Figure M2.13, what route will minimize the distance that Leslie travels?
M2-14 An air cargo company has the shipping requirements
shown in the accompanying table. Two planes are
available with a total capacity of 11 tons. How many
of each item should be shipped to maximize profits?

ITEM

WEIGHT
(TONS)

7 6

12
5 6

11
3

4
8

10
6

Z00_REND1011_11_SE_MOD2 PP2.QXD

2/22/11

12:19 PM

Page 21

DISCUSSION QUESTIONS AND PROBLEMS

FIGURE M2.12
(for Problem M2-11)

10

5
3

M2-21

3
2

11

4
7

M2-18 In Problem M2-16, what is the shortest distance

from node 6 to the ending node? How does this
change if the road between node 6 and node 11 is no
longer in service (see Problem M2-17)?
M2-19 Paige Schwartz is planning to go camping in Big
Bend National Park. She will carry a backpack with
food and other items, and she wants to carry no more
than 18 pounds of food. The four possible food items
she is considering each have certain nutritional ingredients. These items and their weights are as follows:

ITEMS TO BE SHIPPED
ITEM

WEIGHT
(TONS)

PROFIT/UNIT

NUMBER
AVAILABLE

$3

2
ITEM

WEIGHT (POUNDS)

600

1,000

1,500

300

M2-15 Because of a new manufacturing and packaging procedure, the weight of item 2 in Problem M2-14 can
be cut in half. Does this change the number or types
of items that can be shipped by the air transport
company?
M2-16 What is the shortest route through the network in
Figure M2.14?
M2-17 Refer to Problem M2-16. The road between node 6
and node 11 is no longer in service due to construction. What is the shortest route, given this situation?
FIGURE M2.13
(for Problem M2-13)

How many of each item should paige carry if she

wishes to maximize the total nutritional units?

6
2

11

4
1

NUTRITIONAL UNITS

12
3

4
5

3
10

2
7

Z00_REND1011_11_SE_MOD2 PP2.QXD

M2-22

2/22/11

12:19 PM

Page 22

MODULE 2 DYNAMIC PROGRAMMING

FIGURE M2.14
(for Problem M2-16)

3
3

13

16
3

11

15
6

19
2

4
5

20

17

14

3
10

3
3

18
3

7
4

12

4
4

Case Study
United Trucking
Like many other trucking operations, United Trucking got
started with one truck and one ownerJudson Maclay. Judson
is an individualist and has always liked to do things his way.
He was a fast driver, and many people called the 800 number
on the back of his truck when he worked for Hartmann Trucking. After two years with Hartmann and numerous calls about
his bad driving, Judson decided to go out on his own. He
started United Trucking.
In the early days of United Trucking, Judson was the only
driver. On the back of his truck was the message: How do you
like my driving? Call 1-800-AMI-FAST. He was convinced
that some people actually tried to call the number. Soon, a number of truck operators had the same or similar messages on the
back of their trucks. After three years of operation, Judson had
15 other trucks and drivers working for him. He traded his driving skills for office and management skills. Although 1-800AMI-FAST was no longer visible, Judson decided to never
place an 800 number on the back of any of his trucks. If someone really wanted to complain, they could look up United
Trucking in the phone book.
Judson liked to innovate with his trucking company. He
knew that he could make more money by keeping his trucks
full. Thus he decided to institute a discount trucking service. He
gave substantial discounts to customers that would accept delivery to the West Coast within two weeks. Customers got a great
price, and he made more money and kept his trucks full. Over
time, Judson developed steady customers that would usually

have loads to go at the discounted price. On one shipment, he

had an available capacity of 10 tons in several trucks going to
the West Coast. Ten items can be shipped at discount. The
weight, profit, and number of items available are shown in the
following table:

ITEM

WEIGHT
(TONS)

PROFIT/UNIT

AVAILABLE

$10

10

20

25

11

30

50

10

10

Discussion Questions
1. What do you recommend that Judson do?
2. If the total available capacity were 20 tons, how would
this change Judsons decision?

Internet Case Study

See our Internet home page at www.pearsonhighered.com/render for this additional case study:
Briarcliff Electronics.

Z00_REND1011_11_SE_MOD2 PP2.QXD

2/22/11

12:19 PM

Page 23

BIBLIOGRAPHY

M2-23

Bibliography
Bellman, R. E. Dynamic Programming. Princeton, NJ: Princeton University
Press, 1957.

Howard, R. A. Dynamic Programming. Cambridge, MA: The MIT Press,

1960.

Bourland, Karla. Parallel-Machine Scheduling with Fractional Operator Requirements, IEE Transactions (September 1994): 56.

Ibarake, Toshihide, and Yuichi Nakamura. A Dynamic Programming Method

for Single Machine Scheduling, European Journal of Operations
Research (July 1994): 7282.

Elmaghraby, Salah. Resource Allocation via Dynamic Programming,

European Journal of Operations Research (January 1993): 199215.
El-Rayes, Khaled, et al. Optimized Scheduling for Highway Construction,
Transactions of AACE International (February 1997): 311315.

Idem, Fidelis, and A. M. Reisman. An Approach to Planning for Physician

Requirements in Developing Countries Using Dynamic Programming,
Operations Research (July 1990): 607618.

Hillard, Michael R., et al. Scheduling the Operation Desert Storm Airlift: An
Advanced Automated Scheduling Support System, Interfaces 21, 1
(JanuaryFebruary 1992): 131146.

Stokes, Jeffrey, et al. Optimal Marketing of Nursery Crops from ContainerBased Production Systems, American Journal of Agricultural
Economics (February 1997): 235246.

Z00_REND1011_11_SE_MOD2 PP2.QXD

2/22/11

12:19 PM

Page 24