thesimplexmethoderodri/webpage/cps/theory/lp/simplex/slides.pdfglobalidea 2/41 the fundamental...

The Simplex Method

Combinatorial Problem Solving (CPS)

Javier Larrosa Albert Oliveras Enric Rodrıguez-Carbonell

March 29, 2019

Global Idea

2 / 41

■ The Fundamental Theorem of Linear Programming ensuresit is sufficient to explore basic feasible solutionsto find the optimum of a feasible and bounded LP

■ The simplex method moves from one basic feasible solutionto another that does not worsen the objective function while

◆ optimality or

◆ unboundedness

are not detected

Bases and Tableaux

3 / 41

■ Given a basis B, its tableau is the system of equations

xB = B−1b−B−1RxR

which expresses values of basic variables in terms of non-basic variables

min−x− 2yx+ y + s1 = 3x+ s2 = 2y + s3 = 2x, y, s1, s2, s3 ≥ 0

B = {x, y, s2}

x = 1 + s3 − s1y = 2− s3s2 = 1− s3 + s1

Basic Solution in a Tableau

4 / 41

■ The basic solution can be easily obtained from the tableau by looking atindependent terms

x = 1 + s3 − s1y = 2− s3s2 = 1− s3 + s1

Note that by definition of basic solution,the values for non-basic variables are null

Detecting Optimality (1)

5 / 41

■ Tableaux can be extended withthe expression of the cost function in terms of the non-basic variables

min−x− 2y =⇒ min−5 + s1 + s3x = 1 + s3 − s1y = 2− s3s2 = 1− s3 + s1

■ Value of objective function at basic solutioncan be easily found by looking at independent term


5 / 41




■ Coefficients of non-basic variables in objective function after substitutionwith tableau are called reduced costs

■ By convention, reduced costs of basic variables are 0


5 / 41




■ Coefficients of non-basic variables in objective function after substitutionwith tableau are called reduced costs

■ By convention, reduced costs of basic variables are 0

■ Sufficient condition for optimality: all reduced costs are ≥ 0

The cost of any other feasible solution can’t improve on the basic solution

So the basic solution is optimal!


6 / 41

■ If reduced costs ≥ 0:sufficient condition for optimality but not necessary

■ In the example, both bases are optimalbut in one we cannot detect optimality!

minx+ 2yx+ y = 0x, y ≥ 0

B = {x}

{

min y

x = −y

B = {y}

{

min −x

y = −x

Improving the Basic Solution

7 / 41

■ What to do when the tableau does not satisfy the optimality condition?

min−x− 2yx+ y + s1 = 3x+ s2 = 2y + s3 = 2x, y, s1, s2, s3 ≥ 0

B = (s1, s2, s3)

min −x− 2ys1 = 3− x− y

s2 = 2− x

s3 = 2− y

■ E.g. variable y has a negative reduced cost

■ If we can get a new solution where y > 0 and the rest of non-basicvariables does not worsen the objective value, we will get a better solution

■ In general, to improve the objective value:increase the value of a non-basic variable with negative reduced costwhile the rest of non-basic variables are frozen to 0

E.g. increase y while keeping x = 0


8 / 41

■ Let us increase value of variable y

while satisfying non-negativity constraints on basic variables

s1 = 3− x− y Limits new value to ≤ 3s2 = 2− x Does not limit new values3 = 2− y Limits new value to ≤ 2

■ Best possible new value for y is min(3, 2) = 2

■ The bound due to s3 is tight, i.e.,the constraint s3 ≥ 0 limits the new value for y


9 / 41

■ The new solution does not seem to be basic... but in fact it is.We need to change the basis.

■ When increasing the value of the improving non-basic variable,all basic variables for which the bound is tight become 0

y = 2 → s3 = 0

■ Choose a tight basic variable, here s3,to be exchanged with the improving non-basic variable, here y

■ We can get the tableau of the new basis bysolving the non-basic variable in terms of the basic one and substituting:

s3 = 2− y ⇒ y = 2− s3

min −x− 2ys1 = 3− x− y

s2 = 2− x

s3 = 2− y

=⇒

min −4− x+ 2s3s1 = 1 + s3 − x

s2 = 2− x

y = 2− s3


10 / 41

x

y

x ≤ 2

x + y ≤ 3

y ≤ 2

y ≥ 0

x ≥ 0

(0, 2)

(0, 0)

min−x− 2y


11 / 41

■ Let us now increase value of variable x

min −4− x+ 2s3s1 = 1 + s3 − x Limits new value to ≤ 1s2 = 2− x Limits new value to ≤ 2y = 2− s3 Does not limit new value

■ Best possible new value for x is min(2, 1) = 1

■ Variable s1 leaves the basis and variable x enters

min −4− x+ 2s3s1 = 1 + s3 − x

s2 = 2− x

y = 2− s3

=⇒

min−5 + s1 + s3x = 1 + s3 − s1s2 = 1− s3 + s1y = 2− s3


12 / 41

x

y

x ≤ 2

x + y ≤ 3

y ≥ 0

x ≥ 0

(1, 2)

y ≤ 2 min−x− 2y

(0, 2)

(0, 0)

Unboundedness

13 / 41

■ Unboundedness is detected whenthe new value for the chosen non-basic variable is not bounded.

x

y

2x + y ≥ 3

y ≥ 0

x ≥ 0

max x+ y

2x+ y ≥ 3x, y ≥ 0

⇓

{

min−x− y

−2x− y + s = −3

⇓{

min−3 + x− s

y = 3− 2x+ s

Outline of the Simplex Algorithm

14 / 41

1. Initialization: Pick a feasible basis.

2. Pricing: If all reduced costs are ≥ 0,then return OPTIMAL.Else pick a non-basic variable with reduced cost < 0.

3. Ratio test: Compute best value for improving non-basic variablerespecting non-negativity constraints of basic variables.If best value is not bounded,then return UNBOUNDED.Else select basic variable for exchange with improving non-basic variable.

4. Update: Update the tableau and go to 2.

Finding an Initial Basis

15 / 41

■ Note that to optimize

min cTxAx = b

x ≥ 0

initially we need a feasible basis at step 1.

Steps 2-4 of previous procedure are called phase II of simplex algorithm

■ Phase I looks for a feasible basis

■ We can get a feasible basis with the same procedure by solving anotherLP for which phase I is trivial

■ Let us assume wlog. that b ≥ 0

■ Introduce new artificial variables z and solve

min 1T zAx+ z = b

x, z ≥ 0


16 / 41

[LP ]min cTxAx = b

x ≥ 0=⇒ [LP ′]

min 1T zAx+ z = b where b ≥ 0x, z ≥ 0

■ LP ′ is feasible, and a trivial feasible basis is B = (z)

■ LP ′ cannot be unbounded: z ≥ 0 implies 1T z ≥ 0

So LP ′ has optimal solution with objective value ≥ 0

■ If x∗ is feasible solution to LP then (x, z) = (x∗, 0) is optimal solution toLP ′ with objective value 0

■ If (x, z) = (x∗, z∗) is optimal solution to LP ′ with objective value 0 thenz∗ = 0 and so x∗ is feasible solution to LP


17 / 41

[LP ]min cTxAx = b

x ≥ 0=⇒ [LP ′]

min 1T zAx+ z = b where b ≥ 0x, z ≥ 0

■ LP is feasible iff optimum of LP ′ is 0

■ Still: how can we get a feasible basis for LP?

■ Assume that optimum of LP ′ is 0. Then:

1. If all artificial variables are non-basic,then an optimal basis for LP ′ is a feasible basis for LP

2. Any basic artificial variable can be made non-basicby Gaussian elimination (since A has full rank)


18 / 41

■ Improvement: use slack variables instead of artificial variablesin the initial basis whenever possible

■ Alternative phase I approaches do not introduce new variables andwork by minimizing the sum of infeasibilities:

min{

∑

βi<0

βi | B basis with basic solution β}


19 / 41

min −x− 2y1 ≤ x+ y ≤ 30 ≤ x ≤ 20 ≤ y ≤ 2

⇒

min−x− 2yx+ y + s1 = 3x+ y − s2 = 1x+ s3 = 2y + s4 = 2

⇒

min z1x+ y + s1 = 3x+ y − s2 + z1 = 1x+ s3 = 2y + s4 = 2


20 / 41

min 1− x− y + s2s1 = 3− x− y

z1 = 1− x− y + s2s3 = 2− x

s4 = 2− y

⇒

min z1s1 = 2 + z1 − s2x = 1− z1 − y + s2s3 = 1 + z1 + y − s2s4 = 2− y

Feasible tableau

s1 = 2− s2x = 1− y + s2s3 = 1 + y − s2s4 = 2− y


21 / 41

x ≤ 2

y ≤ 2

y ≥ 0

(1, 0)

y

x + y ≤ 3

x + y ≥ 1

x ≥ 0

x

(0, 0)

Big M Method

22 / 41

■ Alternative to phase I + phase II approach

■ LP is changed as follows, where M is a “big number”

min cTxAx = b

x ≥ 0=⇒

min cTx+M · 1T zAx+ z = b where b ≥ 0x, z ≥ 0

■ Again by taking the artificial variables we get an initial feasible basis

■ The search of a feasible basis for the original problem is not blind wrt. cost

■ Problems:

◆ If M is a fixed big number,then the algorithm becomes numerically unstable

◆ If M is kept symbolically,then handling costs becomes more expensive

Big M Method

23 / 41

min −x− 2y1 ≤ x+ y ≤ 30 ≤ x ≤ 20 ≤ y ≤ 2

⇒

min−x− 2yx+ y + s1 = 3x+ y − s2 = 1x+ s3 = 2y + s4 = 2

⇒

min −x− 2y +Mz

x+ y + s1 = 3x+ y − s2 + z = 1x+ s3 = 2y + s4 = 2

Big M Method

24 / 41

minM + (−1−M)x+ (−2−M)y +Ms2s1 = 3− x− y

z = 1− x− y + s2s3 = 2− x

s4 = 2− y

=⇒

minx− 2− 2s2 + (M + 2)zs1 = 2 + z − s2y = 1− x− z + s2s3 = 2− x

s4 = 1 + z + x− s2

Then we could drop the artificial variable z and continue the optimization.

Termination and Complexity

25 / 41

■ At each step of the simplex algorithm:

cost improvement = reduced cost · increment (of chosen non-basic var)

■ A step of the simplex algorithm is degenerateif the increment of the chosen non-basic variable is 0

■ If the step is degenerate then there is no cost improvement

■ Degenerate steps can only happen with degenerate bases

■ Assume no degenerate bases occur.

Then there is a strict improvement from a base to the next one

So simplex terminates, as bases cannot be repeated

No. steps is at most exponential: there are ≤(

nm

)

bases

Tight bound for pathological cases (Klee-Minty cube)

In practice the cost is polynomial

Termination and Complexity

26 / 41

■ When there is degeneracy simplex may loop forever

■ Termination guaranteed with anticycling rules, e.g. Bland’s rule:

Assume there is a fixed ordering of variables.

Pricing: among non-basic vars with reduced cost < 0, take the least one

Ratio test: among tight basic vars, take the least one

Termination with Bland’s Rule

27 / 41

PROOF:

States of simplex algorithm determined by bases.

To prove termination, enough to prove we can’t repeat bases

Let us prove termination by contradiction.

Assume there is a cycle: Bk, ...,Bt,Bt+1 such that Bk = Bt+1

Var xj is fickle if it is in some, but not all, bases of the cycle

For all ratio tests in cycle, entering variable takes value 0

Hence pivoting steps do not change basic solution:basic solution is the same for all bases of the cycle

So fickle variables have value 0 in basic solution


28 / 41

Let xr be the largest fickle variable

Let l ∈ {k, ..., t} be such that xr ∈ Bl and xr ∈ Rl+1

Let xr =∑

xj∈Rlλjxj be the respective row in Bl’s tableau

Let xs ∈ Rl be the non-basic variable that is swapped with xr in Bl

Let dl(xj) be the reduced cost of a variable xj in Bl

Since xs is entering the basis, dl(xs) < 0 and λs < 0

Moreover, xs is fickle too, and hence xs ≺ xr


29 / 41

Let Bp be the first basis after Bl+1 where xr gets basic again:xr ∈ Rp and xr ∈ Bp+1

Let dp(xj) be the reduced cost of a variable xj in Bp

Since xr is entering the basis, dp(xr) < 0

Moreover dp(xs) ≥ 0:

■ If xs ∈ Rp: by Bland’s rule and xs ≺ xr

■ If xs ∈ Bp: reduced costs of basic vars are null


30 / 41

Let γl be the value of the objective function at the basic solution of Bl

Then for any x such that Ax = b: cTx = γl +∑

dl(xj)xj

Let γp be the value of the objective function at the basic solution of Bp

Then for any x such that Ax = b: cTx = γp +∑

dp(xj)xj

As basic solution is the same all the time: γl = γpHence for any x such that Ax = b:

∑

dl(xj)xj =∑

dp(xj)xj

If xs = t and xj = 0 for all xj ∈ Rl, j 6= s then xBl= B−1

l b−B−1

l ast. So:

∑

xj∈Bl

dl(xj)xj +∑

xj∈Rl

dl(xj)xj =∑

xj∈Bl

dp(xj)xj +∑

xj∈Rl

dp(xj)xj

0 + dl(xs)t =∑

xki∈Bl

dp(xki)(βi − αist) + dp(xs)t

where β = B−1

l b and αs = B−1

l as


31 / 41

Hence dl(xs) = −∑

xki∈Bl

dp(xki)αis + dp(xs)

As dl(xs) < 0 and dp(xs) ≥ 0, it must be∑

xki∈Bl

dp(xki)αis > 0

There must exist xki ∈ Bl such that dp(xki)αis > 0

So dp(xki) 6= 0 and xki 6∈ Bp. As xki ∈ Bl, xki is fickle. Now:

■ xki = xr: dp(xr) < 0 and αis > 0 implies dp(xki)α

is < 0 !!!

■ xki ≺ xr: as we didn’t chose xki to enter Bp, dp(xki) ≥ 0

Since dp(xki)αis > 0, we have dp(xki) > 0 and αi

s > 0

But xki is fickle, so its basic value at Bl is 0

By the ratio rule, xki has ratio 0, so it could leave Bl

Contradiction! xki ≺ xr and xr was chosen to leave Bl

Pricing Strategies

32 / 41

1. Full pricing

Choose the variable with the most negative reduced cost

2. Partial pricing

Make a list with the indices of the P variableswith the most negative reduced costs.

In following iterations choose variables from the listuntil reduced costs are all ≥ 0

Pricing Strategies

33 / 41

3. Best-improvement pricing

Let θk be the increment for a non-basic variable xk with reduced costdk < 0. Choose the variable j such that

|dj| · θj = max{|dk| · θk such that dk < 0, k ∈ R}

4. Normalized pricing.

Let nk = ||αk|| (in practice nk =√

1 + ||αk||2)

where αk is the column in the tableau of variable xk.

Take criteria 1. or 2. but using dknk

instead of dk

5. Other more sophisticate normalized pricing strategies:

steepest edge, devex

Bounded Variables

34 / 41

■ LP solvers implement a variant of the simplex algorithmthat handles bounds more efficiently for LP’s of the form

min cTxAx = b

ℓ ≤ x ≤ u

■ ℓi may be −∞ and/or ui may be +∞

■ Bounds are incorporated into pricing and ratio test

■ Now non-basic variables will take values at the lower or the upper bound

Bounded Variables

35 / 41

min−x− 2y

x+ y ≤ 30 ≤ x ≤ 20 ≤ y ≤ 2

=⇒

min−x− 2y

x+ y + s = 30 ≤ x ≤ 20 ≤ y ≤ 2s ≥ 0

=⇒

min−x− 2y

s = 3− x− y

0 ≤ x ≤ 20 ≤ y ≤ 2s ≥ 0

■ Initially non-basic variables x, y are at lower bound

■ We choose variable x in pricing

Bounded Variables

36 / 41

min−x− 2ys = 3− x− y Limits new value to ≤ 3 as s ≥ 00 ≤ x ≤ 2 Limits new value to ≤ 2 as x ≤ 20 ≤ y ≤ 2s ≥ 0

■ Best possible new value for x is min(3, 2) = 2

■ Bound flip: x is still non-basic, but is now at upper bound

min−x− 2ys = 3− x− y

0 ≤ x ≤ 20 ≤ y ≤ 2s ≥ 0

Bounded Variables

37 / 41

■ Pricing considers the bound status of non-basic variables

■ A non-basic variable xj with reduced cost dj can improve the cost function

◆ if xj is at lower bound and dj < 0; or

◆ if xj is at upper bound and dj > 0

■ Choose y in pricing:

min−x− 2ys = 3− x− y Limits new value to ≤ 1 as s ≥ 00 ≤ x ≤ 20 ≤ y ≤ 2 Limits new value to ≤ 2 as y ≤ 2s ≥ 0

■ Best possible new value for y is min(1, 2) = 1

Bounded Variables

38 / 41

■ Usual pivoting step now:

s = 3− x− y ⇒ y = 3− x− s

min−x− 2ys = 3− x− y

0 ≤ x ≤ 20 ≤ y ≤ 2s ≥ 0

=⇒

min−6 + x+ 2sy = 3− x− s

0 ≤ x ≤ 20 ≤ y ≤ 2s ≥ 0

Bounded Variables

39 / 41

■ Choose x in pricing. To respect bounds for y:

0 ≤ y(x) ≤ 2

0 ≤ 3− x ≤ 2

(since x decreases its value, 0 ≤ y(x) is OK)

3− x ≤ 2

1 ≤ x

min−6 + x+ 2sy = 3− x− s Limits new value to ≥ 10 ≤ x ≤ 2 Limits new value to ≥ 00 ≤ y ≤ 2s ≥ 0

■ Best possible new value for x is max(1, 0) = 1

Bounded Variables

40 / 41


y = 3− x− s ⇒ x = 3− y − s

Bounded Variables

40 / 41


y = 3− x− s ⇒ x = 3− y − s

min−6 + x+ 2sy = 3− x− s

0 ≤ x ≤ 20 ≤ y ≤ 2s ≥ 0

=⇒

min−3 + s− y

x = 3− y − s

0 ≤ x ≤ 20 ≤ y ≤ 2s ≥ 0

■ Since upper bound of y was tight, now y is set to its upper bound

■ Optimal solution: (x, y, s) = (1, 2, 0) with cost −5

■ Now reading the basic solution and its cost is more involved!

Bounded Variables

41 / 41

min−x− 2y

x

y ≤ 2

(0, 0)

y ≥ 0

x ≤ 2

x ≥ 0

(2, 0)

(2, 1)

x + y ≤ 3

y

(1, 2)

thesimplexmethoderodri/webpage/cps/theory/lp/simplex/slides.pdfglobalidea 2/41 the fundamental...

Documents