Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Operations Research II: Algorithms

Branch & Bound and Heuristic Algorithms

Ling-Chieh Kung

Department of Information Management

National Taiwan University

OR II: Branch & Bound and Heuristics 1 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Introduction

I In some cases, variables must only take integer values.

I The subject of formulating and solving models with integer variables is
Integer Programming (IP).
I An IP is typically a linear IP (LIP).
I If the objective function or any functional constraint is nonlinear, it is a
nonlinear IP (NLIP).
I We will focus on linear IP in this course.

OR II: Branch & Bound and Heuristics 2 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Introduction

I The branch-and-bound algorithm finds an optimal solution for

any IP.
I It “decomposes” an IP to multiple LPs, solve all the LPs, and compares
those outcomes to reach a conclusion.
I Each LP is solved separately (with the simplex method or other ways).
I In general, the process may take a very long time.
I As finding an optimal solution for an IP may be too time-consuming,
we often look for a near-optimal feasible solution instead in practice.
I An algorithm that generates a feasible solution in a short time is
called a heuristic algorithm.
I Hopefully it is near-optimal.
I A good heuristic algorithm does not always work, but it works for most
of the time.

OR II: Branch & Bound and Heuristics 3 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Road map

I Linear relaxation.
I The branch-and-bound algorithm.
I Solving the knapsack problem.
I Heuristic algorithms.

OR II: Branch & Bound and Heuristics 4 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Solving an IP
max 3x1 + x2
I Suppose we are given an IP, how may we s.t. 4x1 + 2x2 ≤ 11
solve it? xi ∈ Z+ ∀i = 1, 2.
I The simplex method does not work!
I The feasible region is not “a region”.
I It is discrete.
I There is no way to “move along edges”.
I But all we know is how to solve LPs. How
about solving a linear relaxation first?

Definition 1 (Linear relaxation)

For a given IP, its linear relaxation is
the resulting LP after removing all the
integer constraints.

OR II: Branch & Bound and Heuristics 5 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Linear relaxation
I What is the linear relaxation of

max x1 + x2
s.t. x1 + 3x2 ≤ 10
2x1 − x2 ≥ 5
xi ∈ Z+ ∀i = 1, 2?
I Z is the set of all integers. Z+ is the set of all nonnegative integers.
I The linear relaxation is

max x1 + x2
s.t. x1 + 3x2 ≤ 10
2x1 − x2 ≥ 5
xi ≥ 0 ∀i = 1, 2.

OR II: Branch & Bound and Heuristics 6 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Linear relaxation

I For the knapsack problem

max 16x1 + 22x2 + 12x3 + 8x4

s.t. 5x1 + 7x2 + 4x3 + 3x4 ≤ 10
xi ∈ {0, 1} ∀i = 1, ..., 4,

the linear relaxation is

max 16x1 + 22x2 + 12x3 + 8x4
s.t. 5x1 + 7x2 + 4x3 + 3x4 ≤ 10
xi ∈ [0, 1] ∀i = 1, ..., 4,
I xi ∈ [0, 1] is equivalent to xi ≥ 0 and xi ≤ 1.

OR II: Branch & Bound and Heuristics 7 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Linear relaxation provides a bound

I For a minimization IP, its linear relaxation provides a lower bound.

Proposition 1
Let z ∗ and z 0 be the objective values associated to optimal solutions of
a minimization IP and its linear relaxation, respectively, then z 0 ≤ z ∗ .

Proof. They have the same objective function. However, the linear
relaxation’s feasible region is (weakly) larger than that of the IP.
I For a maximization IP, linear relaxation provides an upper bound.

OR II: Branch & Bound and Heuristics 8 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Linear relaxation may solve the IP

I If we are lucky, the linear relaxation may be infeasible or unbounded.
I The IP is then infeasible or unbounded.
I If we are lucky, an optimal solution to the linear relaxation may be
feasible to the original IP. When this happens, the IP is solved:

Proposition 2
Let x0 be an optimal solutions to the linear relaxation of an IP. If x0
is feasible to the IP, it is optimal to the IP.

Proof. Suppose x0 is not optimal to the IP, there must be another

feasible solution x00 that is better. However, as x00 is feasible to the IP,
it is also feasible to the linear relaxation, which implies that x0 cannot
be optimal to the linear relaxation.
I What if we are unlucky?

OR II: Branch & Bound and Heuristics 9 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Rounding a fractional solution

I Suppose we solve a linear relaxation with an LR-optimal solution x0 .

I “LR-optimal” means x0 is optimal to the linear relaxation.
I x0 , however, has at least one variable violating the integer constraint in
the original IP.
I We may choose to round the variable.
I Round up or down?
I Is the resulting solution always feasible?
I Will the resulting solution be close to an IP-optimal solution x∗ ?

OR II: Branch & Bound and Heuristics 10 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Rounding a fractional solution

I Consider the following IP

max 8x1 + 5x2

s.t. x1 + x2 ≤ 6
9x1 + 5x2 ≤ 45
xi ∈ Z+ ∀i = 1, 2.

I x∗ = (5, 0) is IP-optimal.
I But x1 = ( 15 9
4 , 4 ) is LR-optimal!
I Rounding up any variable results
in infeasible solutions.
I None of the four grid points
around x1 is optimal.
I We need a way that guarantees to
find an optimal solution.

OR II: Branch & Bound and Heuristics 11 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Road map

I Linear relaxation.
I The branch-and-bound algorithm.
I Solving the knapsack problem.
I Heuristic algorithms.

OR II: Branch & Bound and Heuristics 12 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Rounding a fractional solution

I x1 = ( 15 9
4 , 4 ) is LR-optimal.
I Rounding up or down x1
(i.e., adding x1 = 4 or x1 = 3
into the program) both fail
to find the optimal solution.
I Because we eliminate too
many feasible points!
I Instead of adding equalities,
we should add inequalities.
I What will happen if we add
x1 ≥ 4 or x1 ≤ 3 into the
program?
I We will branch this problem
into two problems, one with
an additional constraint.
OR II: Branch & Bound and Heuristics 13 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Rounding a fractional solution

If we add x1 ≤ 3: If we add x1 ≥ 4:

I The optimal solution to the IP must be contained in one of the above

two feasible regions. Why?

OR II: Branch & Bound and Heuristics 14 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Rounding a fractional solution

I So when we solve the linear relaxation and find any variable violating
an integer constraint, we will branch this problem into two problems,
one with an additional constraint.
I The two new programs are still linear programs.
I Once we solved them:
I If their LR-optimal solutions are both IP-feasible, compare them and
choose the better one.
I If any of them results in a variable violating the integer constraint,
branch on that variable recursively.
I Eventually compare all the IP-feasible solutions we obtain.

OR II: Branch & Bound and Heuristics 15 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Example

I Let’s illustrate the

branch-and-bound algorithm
with the following example:

max 3x1 + x2
(P0 ) s.t. 4x 1 + 2x 2 ≤ 11
xi ∈ Z+ ∀i = 1, 2.

OR II: Branch & Bound and Heuristics 16 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Subproblem 1

I First we solve the linear

relaxation:
max 3x1 + x2
(P1 ) s.t. 4x1 + 2x2 ≤ 11
xi ≥ 0 ∀i = 1, 2.
I The optimal solution is
x1 = ( 11
4 , 0).
I So we need to branch on x1 .

OR II: Branch & Bound and Heuristics 17 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Branching tree

I The branch and bound algorithm produces a branching tree.

I Each node represents a subproblem (which is an LP).
I Each time we branch on a variable, we create two child nodes.

OR II: Branch & Bound and Heuristics 18 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Subproblem 2

I When we add x1 ≤ 2:
max 3x1 + x2
s.t. 4x1 + 2x2 ≤ 11
(P2 ) x1 ≤ 2
xi ≥ 0 ∀i = 1, 2.

I An (P2 )-optimal solution is

x2 = (2, 32 ).
I So later we need to branch
on x2 .
I Before that, let’s solve (P3 ).

OR II: Branch & Bound and Heuristics 19 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Subproblem 3

I When we add x1 ≥ 3:
max 3x1 + x2
s.t. 4x1 + 2x2 ≤ 11
(P3 ) x1 ≥ 3
xi ≥ 0 ∀i = 1, 2.
I The problem is infeasible!
I This node is “dead” and does
not produce any candidate
solution.

OR II: Branch & Bound and Heuristics 20 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Branching tree
I The current progress can be summarized in the branching tree.

I Note that z2 = 7.5 < 8.25 = z1 .

I In general, when we branch to the next level, the objective value
associated with a subproblem-optimal solution will always be weakly
lower (for a maximization problem). Why?

OR II: Branch & Bound and Heuristics 21 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Branching tree
3
I As x2 = 2 in x2 , we will branch subproblem 2 on x2 .

OR II: Branch & Bound and Heuristics 22 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Subproblem 4

I When we add x2 ≤ 1:

3x1 + x2
max
4x1 + 2x2 ≤ 11
s.t.
(P4 ) x1 ≤ 2
x2 ≤ 1
xi ≥ 0 ∀i = 1, 2.

I Note that we add x2 ≤ 1 into

subproblem 2, so x1 ≤ 2 is
still there.

OR II: Branch & Bound and Heuristics 23 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Subproblem 5

I When we add x2 ≥ 2:

max 3x1 + x2
s.t. 4x1 + 2x2 ≤ 11
(P5 ) x1 ≤ 2
x2 ≥ 2
xi ≥ 0 ∀i = 1, 2.

OR II: Branch & Bound and Heuristics 24 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Branching tree
I x4 satisfies all the integer constraints.
I It is IP-feasible and thus a candidate solution to the original IP.
I But branching subproblem 5 may result in a better solution.

OR II: Branch & Bound and Heuristics 25 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Branching tree
I Let’s branch subproblem 5 on x1 .

OR II: Branch & Bound and Heuristics 26 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Subproblem 6

I When we add x1 ≤ 1:

max 3x1 + x2
s.t. 4x1 + 2x2 ≤ 11
x1 ≤ 2
(P6 ) x2 ≥ 2
x1 ≤ 1
xi ≥ 0 ∀i = 1, 2.

I x6 = (1, 27 ). We may need to

branch on x2 again. However,
let’s solve subproblem 7 first.

OR II: Branch & Bound and Heuristics 27 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Subproblem 7

I When we add x1 ≥ 2:

max 3x1 + x2
s.t. 4x1 + 2x2 ≤ 11
x1 ≤ 2
(P7 ) x2 ≥ 2
x1 ≥ 2
xi ≥ 0 ∀i = 1, 2.

I The problem is infeasible.

I The node is “dead”.

OR II: Branch & Bound and Heuristics 28 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Branching tree

I The only “alive”

node is subproblem
6, with x2
fractional.
I Before we branch
subproblem 6,
consider the
following:

OR II: Branch & Bound and Heuristics 29 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Bounding
I z6 = 132 . If we branch
(P6 ), all the candidate
solutions (if any) under it
will be (weakly) worse
than 132 .
I However, 13 2 < 7 = z4 ,
and x4 is already a
candidate!
I So there is no need to
branch (P6 ). This is the
“bounding” situation in
the branch-and-bound
algorithm.
I This allows us to solve
fewer subproblems.

OR II: Branch & Bound and Heuristics 30 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Summary

I In running the branch-and-bound algorithm, we maintain a tree.

I If a subproblem-optimal solution is IP-feasible, set it to the candidate
solution if it is currently the best among all IP-feasible solutions. Stop
branching this node.
I If a subproblem is infeasible, stop branching this node.
I If a subproblem-optimal solution is not IP-feasible:
I If it is better than the current candidate solution, branch.
I Otherwise, stop branching.

OR II: Branch & Bound and Heuristics 31 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Another example

I Now let’s go back to our motivating example:

max 8x1 + 5x2

s.t. x1 + x2 ≤ 6
(Q0 ) 9x1 + 5x2 ≤ 45
xi ∈ Z+ ∀i = 1, 2.

I Let’s solve it with the branch-and-bound algorithm.

OR II: Branch & Bound and Heuristics 32 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Subproblem 1

I x1 = ( 15 9
4 , 4 ).
I We may branch on either
variable. Let’s branch on x1 .

OR II: Branch & Bound and Heuristics 33 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Subproblems 2 and 3

I Subproblem 2 generates a
candidate solution.
I x3 = (4, 95 ). As
z3 = 41 > z2 = 39, we should
branch subproblem 3.

OR II: Branch & Bound and Heuristics 34 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Subproblems 4 and 5

I x4 = ( 40
9 , 1). As
z4 = 40.56 > z2 = 39, we should
branch subproblem 4.
I Subproblem 5 is infeasible.

OR II: Branch & Bound and Heuristics 35 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Subproblems 6 and 7
I x6 = (4, 1) but z6 = 37 < 39 = z2 .
I x7 = (5, 0) and z7 = 40 > 39 = z2 .
As it is also the last node, x7 is
an optimal solution.

OR II: Branch & Bound and Heuristics 36 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Remarks
I To select a node to branch:
I Among all alive nodes, there are many different ways of selecting a node
to branch.
I One common approach is to branch the node with the highest objective
value (for a maximization problem). Why?
I Another popular approach is “once a node is branched, all its
descendants are branched before any nondescendant. Why?
I Choosing a variable to branch on is also a challenging task.
I The branch-and-bound algorithm guarantees to find an optimal
solution, if one exists.
I However, it is an exponential-time algorithm.
I Roughly speaking, with n integer variables, the number of subproblems
solved is approximately proportional to 2n .

OR II: Branch & Bound and Heuristics 37 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Road map

I Linear relaxation.
I The branch-and-bound algorithm.
I Solving the knapsack problem.
I Heuristic algorithms.

OR II: Branch & Bound and Heuristics 38 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

The knapsack problem

I We start our illustration with the classic knapsack problem.

I There are five items to select:

Item 1 2 3 4 5
Value ($) 2 3 4 1 3
Weight (kg) 4 5 3 1 4

I The knapsack capacity is 11 kg.

I We maximize the total value without exceeding the knapsack capacity.

OR II: Branch & Bound and Heuristics 39 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

The knapsack problem

I The complete formulation:

max 2x1 + 3x2 + 4x3 + x4 + 3x5

s.t. 4x1 + 5x2 + 3x3 + x4 + 4x5 ≤ 11
xi ∈ {0, 1} ∀i = 1, ..., 5.

I Let’s solve the knapsack problem with the branch-and-bound

algorithm.

OR II: Branch & Bound and Heuristics 40 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Solving the linear relaxation

I The first step is always to solve the linear relaxation:

max 2x1 + 3x2 + 4x3 + x4 + 3x5

s.t. 4x1 + 5x2 + 3x3 + x4 + 4x5 ≤ 11
xi ∈ [0, 1] ∀i = 1, ..., 5.

I A very intuitive way works:

I Sort all items in the descending order of vi , where vi and wi are the
wi
value and weight of item i, respectively.
I Select items one by one according to the order until the knapsack is full.
Note that the last item may be partially selected.
I The five ratios are 24 , 35 , 43 , 11 , and 34 . The order is item 3, item 4, item
5, item 2, and lastly item 1.
I An optimal solution to the linear relaxation is (0, 3 , 1, 1, 1).
5

OR II: Branch & Bound and Heuristics 41 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Subproblems 1, 2, and 3
I We solve the linear relaxation and
get x1 = (0, 0.6, 1, 1, 1) with
objective value 9.8.
I We branch on x2 .
I With x2 ≤ 0: x2 = (0.75, 0, 1, 1, 1)
and z2 = 9.5.
I We set x2 = 0 and then apply
the sorting idea on the
remaining.
I With x2 ≥ 1: x3 = (0, 1, 1, 1, 0.5)
and z3 = 9.5.
I We set x2 = 1 and then apply
the sorting idea on the remaining
four variables and residual
capacity 6.

OR II: Branch & Bound and Heuristics 42 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Subproblems 4 and 5

I We branch the
node for x2 .
I With x2 ≤ 0 and
x1 ≤ 0:
x4 = (0, 0, 1, 1, 1)
and z4 = 8. x4 is
the first candidate
solution.
I With x2 ≤ 0 and
x1 ≥ 1:
x5 = (1, 0, 1, 1, 34 )
with z5 = 9.25.

OR II: Branch & Bound and Heuristics 43 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Subproblems 6 and 7

I We branch the
node for x3 .
I With x2 ≥ 1 and
x5 ≤ 0:
x6 = (0.5, 1, 1, 1, 0)
and z6 = 9.
I With x2 ≥ 1 and
x5 ≥ 1:
x7 = (0, 1, 23 , 0, 1)
and z7 = 263 .

OR II: Branch & Bound and Heuristics 44 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Subproblems 8 and 9
I With x2 ≤ 0,
x1 ≥ 1, x5 ≤ 0:
x8 = (1, 0, 1, 1, 0)
and z8 = 7.
I With x2 ≤ 0,
x1 ≥ 1, x5 ≥ 1:
x9 = (1, 0, 1, 0, 1)
and z9 = 9. x9
becomes the
currently best
candidate.
I No need to branch
the nodes for x6
and x7 (why?).
I x9 is optimal.
OR II: Branch & Bound and Heuristics 45 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Remarks

I The major process is still branch and bound.

I The only unique thing is that we have a special way to solve the
linear relaxation of the knapsack problem.
I In other words, we may customize the branch-and-bound algorithm
for each specific type of IP.

OR II: Branch & Bound and Heuristics 46 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Road map

I Linear relaxation.
I The branch-and-bound algorithm.
I Solving the knapsack problem.
I Heuristic algorithms.

OR II: Branch & Bound and Heuristics 47 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

The knapsack problem

I Recall our knapsack example with five items to select and a knapsack
with capacity 11 kg:

Item 1 2 3 4 5
Value ($) 2 3 4 1 3
Weight (kg) 4 5 3 1 4

I The formulation is

max 2x1 + 3x2 + 4x3 + x4 + 3x5

s.t. 4x1 + 5x2 + 3x3 + x4 + 4x5 ≤ 11
xi ∈ {0, 1} ∀i = 1, ..., 5.

I An optimal solution x∗ = (1, 0, 1, 0, 1) may be found by branch and

bound. The associated objective value is z ∗ = 9.

OR II: Branch & Bound and Heuristics 48 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Exact and heuristic algorithms

I Unfortunately, the knapsack problem is NP-hard.

I Roughly speaking, most researchers in the world believe that there is no
efficient (i.e., polynomial-time) exact algorithm that finds an optimal
solution.
I Branch and bound may need too much time to solve an instance of a
knapsack problem with thousands of variables.
I In this case, people look for good heuristic algorithms.
I The algorithm reports a solution in a short (i.e., polynomial) time.
I The reported solution is near-optimal.

OR II: Branch & Bound and Heuristics 49 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

A greedy algorithm
I Our intuitive greedy algorithm may serve as a good one.
I First, we sort all items by their value-to-weight ratios from large to small.
I Along the order, we then try to put each item into the knapsack. If
possible, do it; otherwise, throw it away.
I We stop when all items are tried once.
I Recall our example:

Item 1 2 3 4 5
Value ($) 2 3 4 1 3
Weight (kg) 4 5 3 1 4

I The order: items 3, 4, 5, 2, and 1.

I Items 3, 4, and 5 are okay. Items 2 and 1 are not.
I The reported solution is xALG = (0, 0, 1, 1, 1). The associated objective
value is z ALG = 8.

OR II: Branch & Bound and Heuristics 50 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Optimality gap

I The reported solution is xALG = (0, 0, 1, 1, 1) with z ALG = 8.

I An optimal solution is x∗ = (1, 0, 1, 0, 1) with z ∗ = 9.
I A natural measurement of the quality/performance of a reported
solution is the optimality gap:
I The absolute error is
z ∗ − z ALG = 1.
I The percentage error is

z ∗ − z ALG
= 11.1%.
z∗

OR II: Branch & Bound and Heuristics 51 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Performance evaluation
I To test the average-case performance of a heuristic algorithm:
I Small-scale random instances are generated.
I Each instance is solved by the algorithm and an exact one.
I All the percentage errors are averaged.
I The smaller the average percentage error, the better the algorithm.
I Moreover:
I Large-scale random instances are generated.
I Each instance is solved by the algorithm.
I An upper bound of the objective value of an optimal solution (for a
maximization problem) is obtained (e.g., by linear relaxation).
I The smaller the average percentage error (which is an upper bound for
that calculated with an optimal solution), the better the algorithm.
I In the above example:
I Linear relaxation gives xLR = (0, 0.6, 1, 1, 1) with z LR = 9.8.
LR ALG
I The errors are z LR − z ALG = 1.8 and z −z = 18.4%.
z LR

OR II: Branch & Bound and Heuristics 52 / 53 Ling-Chieh Kung (NTU IM)
Linear relaxation Branch and bound Solving the knapsack problem Heuristic algorithms

Remarks
I Some people refer to the gap between a reported solution and an upper
bound (rather than an optimal solution) as the optimality gap.
I Some people analyze the problem and heuristic algorithm to prove a
worst-case performance guarantee.
I Fascinating but difficult.
I Designing (or proposing) a heuristic algorithm is simple. Evaluation
and analysis are hard.
I Thanks to integer programming, branch and bound, and linear
relaxation:
I We may generate an optimal solution.
I We may use an optimal solution or its upper/lower bound as a
benchmark to evaluate a feasible solution.
I We may thus evaluate a heuristic algorithm.

OR II: Branch & Bound and Heuristics 53 / 53 Ling-Chieh Kung (NTU IM)