Branch-and-Bound

Algorithm

Lecture 3
Branch-and-Bound Algorithm Complete Enumeration

Branch-and-Bound
MATH3220 Operations Research and Logistics Algorithm

Jan. 13, 2015

Pan Li
The Chinese University of Hong Kong
3.1
 Branch-and-Bound
Agenda Algorithm

Complete Enumeration

Branch-and-Bound
Algorithm

1 Complete Enumeration

2 Branch-and-Bound Algorithm

3.2
 Branch-and-Bound
Is IP easy to solve? Algorithm

At the first glance, IP may be easy to solve since the

number of feasible solution is smaller and is limited. But...

If the feasible region is bounded, the number of feasible Complete Enumeration

Branch-and-Bound
solution is finite. But, the number is simply too large Algorithm

(exponential growth). With n variables, there are 2n

solutions to be considered.

The corner point is (generally) no longer feasible.

3.3
 Branch-and-Bound
Overview Algorithm

Enumerating all solution is too slow for most problems.

Branch and bound starts the same as enumerating, but it

cuts out a lot of the enumeration whenever possible. Complete Enumeration

Branch-and-Bound
Algorithm
Branch and bound is the starting point for all solution
techniques for integer programming.

3.4
 Branch-and-Bound
Capital Budgeting Example Algorithm

Complete Enumeration
investment budget = $14,000 Branch-and-Bound
Algorithm
Investment 1 2 3 4 5
Cash Required $5,000 $4,000 $7,000 $3,000 $6,000
Present Value $12,000 $11,000 $13,000 $8,000 $15,000

max 12x1 + 11x2 + 13x3 + 8x4 + 15x5

s.t. 5x1 + 4x2 + 7x3 + 3x4 + 6x5 ≤ 14
xj ∈ {0, 1} for each j = 1 to 5

3.5
 Branch-and-Bound
Complete enumeration Algorithm

Systematically considers all possible values of the

decision variables. - If there are n binary variables, there
are 2n different ways. Complete Enumeration

Branch-and-Bound
Usual idea: iteratively break the problem into two. Algorithm

At the first iteration, we consider separately the case that

x1 = 0 and x1 = 1.

Each node of the tree represents the original problem plus

additional constraints.

3.6
 Branch-and-Bound
An Enumeration Tree Algorithm

Complete Enumeration

Branch-and-Bound
Algorithm

3.7
 Branch-and-Bound
An Enumeration Tree Algorithm

Complete Enumeration

Branch-and-Bound
Algorithm

We refer to node 2 and 3 as the children of node 1 in the

enumeration tree.
We refer to node 1 as the parent of node 2 and 3.

3.8
 Branch-and-Bound
An Enumeration Tree Algorithm

Complete Enumeration

Branch-and-Bound
Algorithm

Which of the following is false?

1 IP(1) is the original integer program.
2 IP(3) is obtained from IP(1) by adding the constraint
x1 = 1.
3 It is possible that there is some solution that is feasible for
both IP(2) and IP(3).

3.9
 Branch-and-Bound
An Enumeration Tree Algorithm

Complete Enumeration

Branch-and-Bound
Algorithm

3.10
 Branch-and-Bound
An Enumeration Tree Algorithm

Complete Enumeration

Branch-and-Bound
Algorithm

Number of leaves of the tree : 32. ⇒ If there are n variables,

the number of leaves is 2n .
3.11
 Branch-and-Bound
On complete enumeration Algorithm

Complete Enumeration
Suppose that we could evaluate 1 billion solutions per Branch-and-Bound
second. Algorithm

Let n = number of binary variables.

Solution times
- n = 30, 1 second
- n = 40, 17 minutes
- n = 50, 11.6 days
- n = 60, 31 years
- n = 70, 31,000 years

3.12
 Branch-and-Bound
How to solve large size integer program faster? Algorithm

Complete Enumeration

Branch-and-Bound
Algorithm

Only a tiny fraction of the feasible solutions actually need to be

examined.

3.13
 Branch-and-Bound
Subtrees of an enumeration tree Algorithm

Complete Enumeration

Branch-and-Bound
Algorithm

If we can eliminate an entire subtree in one step, we can

eliminate a fraction of all complete solutions at a single step.
3.14
 Branch-and-Bound
Branch-and-Bound Algorithm Algorithm

Branch-and-bound algorithm are the most popular

methods for solving integer programming problems.

Basic Idea: Enumeration procedure can always find the Complete Enumeration
optimal solution for any bounded IP problem. But it takes Branch-and-Bound
Algorithm
too much time. So, we consider the partial enumeration.
That is, divide and conquer.

They enumerate the entire solution space but only

implicity; hence they are called
implicit enumeration algorithms.

Bounding, branching, and fathoming are the three

components of Branch-and-bound technique.

Running time grows exponentially with the problem size,

but small to moderate size problems can be solved in
reasonable time.

3.15
 Branch-and-Bound
A simpler problem to work with Algorithm

Complete Enumeration

Branch-and-Bound
Algorithm

Max 24x1 + 2x2 + 20x3 + 4x4

s.t. 8x1 + x2 + 5x3 + 4x4 ≤ 9 IP(1)
xi ∈ {0, 1} for i = 1 to 4

3.16
 Branch-and-Bound
LP Relaxation Algorithm

LP Relaxation: The LP obtained by omitting all integer or 0-1

constraints on variables is called the LP relaxation of IP. Complete Enumeration

Branch-and-Bound
IP: Algorithm

max (or min) cx

subject to Ax ≤ b
x ≥ 0 and integers

LP Relaxation:

max (or min) cx

subject to Ax ≤ b
x ≥0

3.17
 Branch-and-Bound
IP and LP Relaxation Algorithm

Since LP relaxation is less constrained than IP, the following

are immediate:

If IP is a minimization problem, the optimal objective value Complete Enumeration

for LP relaxation is less than or equal to the optimal Branch-and-Bound
Algorithm
objective for IP.

If IP is a maximization, the optimal objective value for LP

relaxation is greater than or equal to that of IP.

If LP relaxation is infeasible, then so is IP.

If LP relaxation is optimized by integer variables, then that

solution is feasible and optimal for IP.

So, solving LP relaxation does give some information: it gives a

bound on the optimal value, and if we are lucky, may give the
optimal solution to IP.

3.18
 Branch-and-Bound
Why can’t we solve the LP relaxation and round it up/down to Algorithm

get the required integer solution?

Rounding does not guarantee the feasibility.

Complete Enumeration

Branch-and-Bound
Algorithm

Max z = x2
s.t. −x1 + x2 ≤ 0.5
x1 + x2 ≤ 3.5
x1 , x2 ≥ 0 and
are integers

3.19
 Branch-and-Bound
Why can’t we solve the LP relaxation and round it up/down to Algorithm

get the required integer solution?

Rounding does not guarantee the optimality.

Complete Enumeration

Branch-and-Bound
Algorithm

Max z = x1 + 5x2
s.t. x1 + 10x2 ≤ 20
x1 ≤ 2
x1 , x2 ≥ 0 and
are integers

3.20
 Branch-and-Bound
Branch-and-Bound Algorithm

The essential idea: search the enumeration tree, but at each

node:
1 Solve the LP relaxation at the node
Complete Enumeration

Branch-and-Bound
2 Eliminate the subtree (fathom it) if Algorithm

1 The solution is integer (there is no need to go further) or

2 There is no feasible solution or
3 The best solution in the subtree cannot be as good as the
best available solution (the incumbent).

3.21
 Branch-and-Bound
Bounding - For each problem or subproblem (will define Algorithm

later), we need to obtain a bound on how good its best

feasible solution can be.

Usually, the bound is obtained by solving the LP relaxation.

Complete Enumeration
LP relaxation of IP(1): Branch-and-Bound
Algorithm

Max 8x1 + 11x2 + 6x3 + 4x4

s.t. 5x1 + 7x2 + 4x3 + 3x4 ≤ 14 LP(1)
0 ≤ xi ≤ 1 for i = 1 to 4

3.22
 Branch-and-Bound
Bounding - For each problem or subproblem (will define Algorithm

later), we need to obtain a bound on how good its best

feasible solution can be.

Usually, the bound is obtained by solving the LP relaxation.

Complete Enumeration
LP relaxation of IP(1): Branch-and-Bound
Algorithm

Max 8x1 + 11x2 + 6x3 + 4x4

s.t. 5x1 + 7x2 + 4x3 + 3x4 ≤ 14 LP(1)
0 ≤ xi ≤ 1 for i = 1 to 4

The LP relaxation of the knapsack problem can be solved

using a "greedy algorithm".
Think of the objective in terms of dollars, and consider the
constraint as bound on the weight.

3.23
 Branch-and-Bound
Solving the LP relaxation (LP(1)) Algorithm

Max 8x1 + 11x2 + 6x3 + 4x4

s.t. 5x1 + 7x2 + 4x3 + 3x4 ≤ 14 LP(1) Complete Enumeration

Branch-and-Bound
0 ≤ xi ≤ 1 for i = 1 to 4 Algorithm

item 1 2 3 4
unit value $1.6 $1.571 $1.5 $1.333

Consider the unit value of the four items. Put items into the
knapsack in decreasing order of unit value. What do you get?

3.24
 Branch-and-Bound
Solving the LP relaxation (LP(1)) Algorithm

Max 8x1 + 11x2 + 6x3 + 4x4

s.t. 5x1 + 7x2 + 4x3 + 3x4 ≤ 14 LP(1) Complete Enumeration

Branch-and-Bound
0 ≤ xi ≤ 1 for i = 1 to 4 Algorithm

item 1 2 3 4
unit value $1.6 $1.571 $1.5 $1.333

Consider the unit value of the four items. Put items into the
knapsack in decreasing order of unit value. What do you get?
⇒ (x1 , x2 , x3 , x4 ) = (1, 1, 0.5, 0), with z = 22
⇒ No integer solution will have value lager than 22.

3.25
 Branch-and-Bound
Branching - partitioning the entire set of feasible solutions Algorithm

into smaller and smaller subsets.

Set " No incumbent with objective value z ∗ = −∞".
x3 - branching variable
Complete Enumeration

Branch-and-Bound
Algorithm

Note that any optimal solution to the overall problem must be

feasible to one of the subproblems. 3.26
 Branch-and-Bound
More on the incumbent Algorithm

The incumbent is the feasible solution for the IP. It is the

best solution so far in the B&B search.

As Branch-and-bound proceeds, new solutions will be Complete Enumeration

evaluated. If a new solution is better than the current Branch-and-Bound
Algorithm
incumbent, it replaces the current incumbent.

So, the incumbent is always the best solution seen so far.

3.27
 Branch-and-Bound
Solving the LP relaxation (LP(2)) Algorithm

IP(2): x3 = 0

Max 8x1 + 11x2 + 6 × 0 + 4x4 Complete Enumeration

s.t. 5x1 + 7x2 + 4 × 0 + 3x4 ≤ 14 LP(2) Branch-and-Bound

Algorithm

0 ≤ xi ≤ 1 for i = 1, 2, 4

item 1 2 3 4
unit value $1.6 $1.571 $0 $1.333

⇒ (x1 , x2 , x3 , x4 ) = (1, 1, 0, 32 ), with z = 21 23

⇒ No integer solution for this subproblem will have value lager
than 21, but we don’t have any feasible integer solution.

3.28
 Branch-and-Bound
Solving the LP relaxation (LP(3)) Algorithm

IP(3): x3 = 1

Max 8x1 + 11x2 + 6 × 1 + 4x4 Complete Enumeration

s.t. 5x1 + 7x2 + 4 × 1 + 3x4 ≤ 14 LP(3) Branch-and-Bound

Algorithm

0 ≤ xi ≤ 1 for i = 1, 2, 4

item 1 2 4
unit value $1.6 $1.571 $1.333

⇒ (x1 , x2 , x3 , x4 ) = (1, 75 , 1, 0), with z = 21 67

⇒ No integer solution for this subproblem will have value lager
than 21, but we don’t have any feasible integer solution.

3.29
 Branch-and-Bound
Enumeration Tree Algorithm

Complete Enumeration

Branch-and-Bound
Algorithm

We do not have any feasible integer solution. So, we will take a

subproblem and branch on one of its variables.
In general, we will choose the subproblem as follows:
choose an active subproblem, which so far only means we
have not chosen before, and
choose the subproblem with the highes solution value (for
maximization) (lowest for minimization).
⇒ Choose IP(3) and branch on x2 .

3.30
 Branch-and-Bound
Enumeration Tree Algorithm

Complete Enumeration

Branch-and-Bound
Algorithm

IP(4): Feasible integer solution with value 18. ⇒ No further

branching on subproblem 4 is needed.
⇒ Fathoming case 1: The optimal solution for its LP relaxation
is integer.
⇒ If this solution is better than the incumbent, it becomes the
new incumbent. 3.31
 Branch-and-Bound
Enumeration Tree Algorithm

Complete Enumeration

Branch-and-Bound
Algorithm

IP(6): Feasible integer solution with value 21 (> 18).

⇒ No further branching on subproblem 6 is needed.
(Fathoming case 1).
⇒ Update the incumbent and its value.
3.32
 Branch-and-Bound
Enumeration Tree Algorithm

Complete Enumeration

Branch-and-Bound
Algorithm

IP(7): Infeasible solution

⇒ No further branching on subproblem 7 is needed.
⇒ Fathoming case 2: The LP relaxation has no feasible
solutions.
3.33
 Branch-and-Bound
Enumeration Tree Algorithm

Complete Enumeration

Branch-and-Bound
Algorithm

Only one active subproblem left is subproblem 2 with its bound

= 21 23 .
Since the optimal objective value of the original problem is
integer, if the best value solution for a node is at most 21 23 ,
then we know the best bound is at most 21. (Other bounds can
also be rounded down.) 3.34
 Branch-and-Bound
Enumeration Tree Algorithm

Complete Enumeration

Branch-and-Bound
Algorithm

Only one active subproblem left is subproblem 2 with its bound

= 21.
⇒ we fathom this subproblem.
⇒ Fathoming case 3: Subproblem’s bound ≤ z.
3.35
 Branch-and-Bound
Enumeration Tree Algorithm

Complete Enumeration

Branch-and-Bound
Algorithm

There are no longer any active subproblems, so the optimal

solution value is 21.

3.36
 Branch-and-Bound
Branch-and-Bound Algorithm Algorithm

Branch-and-bound strategy:
Solve the linear relaxation of the problem.
If the solution is integer, then we are done.
Otherwise, create two new subproblems by branching on a Complete Enumeration

fractional variable. Branch-and-Bound

Algorithm

A node (subproblem) is not active when any of the following

occurs:
1 The node is being branched on;
2 The solution is integral;
3 The subproblem is infeasible;
4 You can fathom the subproblem by a bounding argument.

Choose an active node and branch on a fractional variable.

Repeat until there are no active subproblems.

3.37
 Branch-and-Bound
A branch-and-bound algorithm for mixed integer Algorithm

programming

n
X
Max z= cj xj Complete Enumeration

j=1 Branch-and-Bound
Algorithm
n
X
s.t. aij xj ≤ bi , for i = 1, . . . , m
j=1

xj ≥ 0, for j = 1, . . . , n
xj is integer, for j = 1, . . . , I (I ≤ n)

Modifications:
branching variables: only variables considered are the
integer-restricted variables that have a noninteger value in
the optimal solution for the LP relaxation of the current
subproblem.
values assigned to the branching variable for creating the
new smaller subproblems:
xj ≤ bxj∗ c and xj ≥ dxj∗ e
3.38
 Branch-and-Bound
Example Algorithm

max z = −7x1 − 3x2 − 4x3


 x1 + 2x2 + 3x3 −x4 = 8, Complete Enumeration
s.t. 3x1 + x2 + x3 −x5 = 5, Branch-and-Bound
x1 , x2 , ··· , x5 ≥ 0 and integer.
Algorithm


3.39
 Branch-and-Bound
Algorithm

Complete Enumeration

Branch-and-Bound
Algorithm

3.40
 Branch-and-Bound
Algorithm

Complete Enumeration

Branch-and-Bound
Algorithm

3.41
 Branch-and-Bound
Algorithm

Complete Enumeration

Branch-and-Bound
Algorithm

3.42
 Branch-and-Bound
Lessons learned Algorithm

Branch-and-bound can speed up the search. - Only 7

nodes (LPs) out of 16 were evaluated.

Branch-and-bound relies on eliminating subtrees, either Complete Enumeration

because the IP at the node was solved, or else because Branch-and-Bound

Algorithm
the IP solution cannot possibly be optimum.

Complete enumerations not possible (because the running

time) if there are more than 100 variables. (Even 50
variables would take too long.)

In practice, there are a lot of ways to make

Branch-and-bound even faster.

3.43
 Branch-and-Bound
How to Branch? Algorithm

We want to divide the current problem into two or more

subproblems that are easier than the original. A commonly
used branching method:
Complete Enumeration

xi ≤ bxi∗ c, xi ≥ dxi∗ e Branch-and-Bound

Algorithm

where xi∗ is a fractional variable.

Which variable to branch?
A commonly used branching rule: Branch the most
fractional variable.
We would like to choose the branching that minimizes the
sum of the solution times of all the created subproblems.
How do we know how long it will take to solve each
subproblem?
Answer: We don’t.
Idea: Try to predict the difficulty of a subproblem.
A good branching rule: The value of the linear
programming relaxation changes a lot! 3.44
 Branch-and-Bound
Which Node to Select? Algorithm

An important choice in branch and bound is the strategy

for selecting the next subproblem to be processed.

Goals: Complete Enumeration

Minimizing overall solution time. Branch-and-Bound

Algorithm
Finding a good feasible solution quickly.

Some commonly used search strategies:

Best First
Depth First
Hybrid Strategies
Best Estimate

3.45
 Branch-and-Bound
The Best First Approach Algorithm

One way to minimize overall solution time is to try to

minimize the size of the search tree. We can achieve this
by choosing the subproblem with the best bound (lowest
lower bound if we are minimizing). Complete Enumeration

Branch-and-Bound
Algorithm
Drawbacks of Best First
Doesn’t necessarily find feasible solutions quickly since
feasible solution are "more likely" to be found deep in the
tree.
Node setup costs are high. The linear program being solved
may change quite a bit from one node evaluation to the
next.
Memory usage is high. It can require a lot of memory to
store the candidate list, since the tree can grow "broad".

3.46
 Branch-and-Bound
The Depth First Approach Algorithm

The depth first approach is to always choose the deepest

node to process next. Just dive until you prune, then back
up and go the other way.
Complete Enumeration
This avoids most of the problems with best first: The Branch-and-Bound
Algorithm
number of candidate nodes is minimized (saving memory).
The node set-up costs are minimized.

LPs change very little from one iteration to the next.

Feasible solutions are usually found quickly.

Drawback: If the initial lower bound is not very good, then

we may end up processing lots of non-critical nodes.

Hybrid Strategies: Go depth-first until you find a feasible

solution, then do best first search.

3.47
 Branch-and-Bound
Example Algorithm

Consider the IP(1) in the previous example, an optimal LP

tableau is obtained as in the following table:
x1 x2 x3 x4 x5 solution
Complete Enumeration
x1 1 0 − 51 1
5 − 25 2
5 Branch-and-Bound
8 Algorithm
x2 0 1 5 − 35 1
5
19
5
3 2 11
z 0 0 5 5 5 − 71
5

If the constraint x2 ≤ 3 is added, how can we obtain the new

optimal solution from this optimal tableau?

3.48
 Branch-and-Bound
Example Algorithm

Consider the IP(1) in the previous example, an optimal LP

tableau is obtained as in the following table:
x1 x2 x3 x4 x5 solution
Complete Enumeration
x1 1 0 − 51 1
5 − 25 2
5 Branch-and-Bound
8 Algorithm
x2 0 1 5 − 35 1
5
19
5
3 2 11
z 0 0 5 5 5 − 71
5

If the constraint x2 ≤ 3 is added, how can we obtain the new

optimal solution from this optimal tableau?

Rewrite x2 ≤ 3 as x2 + s = 3, where s is a
slack variable.

Hence s = 3 − x2 = 3 − ( 19 8 3 1


5 − 5 x3 − − 5 x4 − 5 x5 ), or
s − 85 x3 + 35 x4 − 15 x5 = − 54 . We add it to the tableau:

3.49
 Branch-and-Bound
Example Algorithm

s − 58 x3 + 35 x4 − 15 x5 = − 54

x1 x2 s x3 x4 x5 solution Complete Enumeration

x1 1 0 0 − 14 1
5 − 25 2
5
Branch-and-Bound
Algorithm
8
x2 0 1 0 5 − 35 1
5
19
5
s 0 0 1 − 85 3
5 − 15 − 45
3 2 11
z 0 0 0 5 5 5 − 71
5

The tableau is optimal but not feasible as s = − 45 < 0.

We can use the dual simplex method to get the optimal
solution.

3.50
 Branch-and-Bound
Example Algorithm

x1 x2 s x3 x4 x5 solution
x1 1 0 0 − 14 1
5 − 25 2
5 Complete Enumeration
8
x2 0 1 0 5 − 35 1
5
19
5
Branch-and-Bound
Algorithm

s 0 0 1 − 85 3
5 − 15 − 45
3 2 11
z 0 0 0 5 5 5 − 71
5

Leaving variable: s
Entering variable: xj is selected such that |zj /yjs | is the
minimum amongst all yjs < 0. Since
   
z3 z5 3/5 11/5 3
min , = min , =
y3s y5s −8/5 −1/5 8

⇒ The entering variable is x3 . Thus, we do pivot operation on

− 58 .

3.51
 Branch-and-Bound
Example Algorithm

After the pivot operation on − 85 , we get

x1 x2 s x3 x4 x5 solution
x1 1 0 − 18 0 1
8 − 38 1
2
Complete Enumeration

Branch-and-Bound
Algorithm
x2 0 1 1 0 0 0 3
x3 0 0 − 58 1 − 38 1
8
1
2
3 5 17
z 0 0 8 0 8 8 − 29
2

which is both optimal and primal feasible. Hence we obtain the

optimal solution x1 = 21 , x2 = 3, x3 = 12 , x4 = x5 = 0 at node 1.

3.52
 Branch-and-Bound
Rounding down to improve bounds Algorithm

If all cost coefficients of a maximization problem are integer

valued, then the optimal objective value (for the IP) is integer.
And zIP (j) ≤ bzLP (j)c.
Complete Enumeration

Branch-and-Bound
Algorithm

3.53
 Branch-and-Bound
A bad example Algorithm

Max 2x1 + 2x2 + 2x3 + . . . + 2x100

s.t. 2x1 + 2x2 + 2x3 + . . . + 2x100 ≤ 101 Complete Enumeration

Branch-and-Bound
xi ∈ {0, 1} for i = 1, 2, . . . , 100. Algorithm

What would happen if we use branch-and-bound as described

before?

3.54
 Branch-and-Bound
Adding constraints to improve bounds Algorithm

A constraint is called a valid constraint if it is satisfied by all

integer solutions of an IP (but possibly not the linear
solution of its LP relaxation).
Complete Enumeration
Adding a valid inequality might improve the bound. Branch-and-Bound
Algorithm

3.55
 Branch-and-Bound
Algorithm

Opt LP(A)
Max 4x1 + 3x2 + 3x3 + 3x4
x ∗ = (1, 0.5, 0, 0)
s.t. 2x1 + 2x2 + 2x3 + 2x4 ≤ 3 A zLP = 5.5
xi ∈ {0, 1} for i = 1, . . . , 4 Complete Enumeration

Branch-and-Bound
Algorithm

Max 4x1 + 3x2 + 3x3 + 3x4

s.t. x1 + x2 + x3 + x4 ≤ 1.5 B
xi ∈ {0, 1} for i = 1, . . . , 4

Opt LP(C)
Max 4x1 + 3x2 + 3x3 + 3x4
x ∗ = (1, 0, 0, 0)
s.t. x1 + x2 + x3 + x4 ≤ 1 C zLP = 4
xi ∈ {0, 1} for i = 1, . . . , 4

The solution for LP(C) is optimal for IP(A)!

3.56
 Branch-and-Bound
Summary Algorithm

Making Branch-and-bound work well in practice requires

lots of good ideas.

There was not time in class to cover all of these ideas in Complete Enumeration
any detail. Branch-and-Bound
Algorithm

The best idea for speeding up Branch-and-bound is to add

valid inequalities, or improve the inequalities.

3.57