linear programming

Introduction Geometry Simplex Duality

VU Algorithmics

Part V: Linear Programming

Markus Leitner, Gunther Raidl,Mario Ruthmair

Algorithms and Data Structures GroupInstitute of Computer Graphics and Algorithms

Vienna University of Technology

WS 2011/12

VU Algorithmics M. Leitner 1


Topics of this part

Introduction to Linear Programming

Geometry of Linear Programming

The Simplex Method

Duality Theory



Introduction



Literature for this part

1 D. Bertsimas and J. N. Tsitsiklis, Introduction to LinearOptimization, Athena Scientific, 1997.

2 6.251J/15.081J Introduction to Mathematical Programming - Fall2009, MIT OpenCourseWare (http://ocw.mit.edu).

3 G. Nemhauser and L. A. Wolsey, Integer and CombinatorialOptimization, Wiley, 1999.

4 A. Schrijver, Theory of Linear and Integer Programming, Wiley,1986.



What is Linear Optimization / Linear Programming?

Definition 1 (Linear Programming)

Linear programming is the problem of minimizing a linear cost functionsubject to linear equality and inequality constraints.

Example 2 (Exemplary Linear Program)

minimize 2x1 − x2 + 4x3

subject to x1 + x2 + x4 ≤ 2

3x2 − x3 = 5

x3 + x4 ≥ 3

x1 ≥ 0

x3 ≤ 0



General Linear Programming Problem

In the general case we are given a

cost vector c = (c1, . . . , cn)

and seek to minimize a

linear cost function c′x =∑n

i=1 cixi

over all n-dimensional vectors x = (x1, . . . , xn), subject to a set of

linear equality and inequality constraints.

Let

M1, M2, and M3 be finite index sets for each of which we are givenan n-dimensional vector ai and a scalar bi , used to form the i-thconstraint.

N1 and N2 be subsets of {1, . . . , n} indicating which variables xj areconstrained to be nonnegative or nonpositive, respectively.



General Linear Programming Problem

min c′x

s.t. ai′x ≥ bi ∀i ∈ M1

ai′x ≤ bi ∀i ∈ M2

ai′x = bi ∀i ∈ M3

xj ≥ 0 ∀j ∈ N1

xj ≤ 0 ∀j ∈ N2

decision variables x1, . . . , xn

objective (cost) function c′x

free (unrestricted) variables xj , j /∈ N1 ∪ N2



Further Notations and Definitions

vector x satisfying all constraints is called a feasible solution orfeasible vector

set of feasible solutions is called feasible set or feasible region

a feasible solution x∗ that minimizes the objective function (x∗ ≤ xfor all feasible solutions x) is called an optimal feasible solution oroptimal solution;value c′x is then called optimal cost



Transformation Rules

1 max c′x⇔ min c′x

2 ai′x = bi ⇔ ai′x ≤ bi ∧ ai′x ≥ bi

3 ai′x ≤ bi ⇔ −ai′x ≥ −bi

4 xj ≥ 0 and xj ≤ 0 are special cases of ai′x ≥ bi

(j-th unit vector ai, bi = 0)

⇒ feasible set of a linear programming problem can be expressedexclusively in terms of inequality constraints of the form ai′x ≥ bi .

Suppose there are

m constraints (indexed by i = 1, . . . ,m)

A is a m × n matrix with row vectors ai′, i = 1, . . . ,m

b = (b1, . . . , bm)



Definition 3 (Linear Program in General Form (Compact))

min c′x

s.t. Ax ≥ b

x ∈ Rn

Definition 4 (Feasible Set)

P = {x ∈ Rn : Ax ≥ b}



Where do Linear Optimization Programs arise?

Real world problems

may have a (straightforward) description as LP or

may be approximated as LPs

modeling is an important task!

Integer Linear Programs ⇒ last part of “Algorithmics”,mathematical programming (summer term)

Transportations and Logistics

Telecommunications

Scheduling

Manufacturing

Engineering

Portfolio optimization

. . .



Exemplary Transportation Problem (Data)

Given

m plants

n warehouses

si supply of ith plant, i = 1, . . . ,m

dj demand of jth warehouse, j = 1, . . . , n

cij transportation costs per unit from plant i to warehouse j∑mi=1 si =

∑nj=1 dj (for simplicity)

Objective

find a minimum cost transportation plan such that

all demands are satisfied andand all supplies are respected



Exemplary Transportation Problem (Formulation)

Decision variables

xij . . . number of units to send from supply i to plant j

Formulation (Linear Program)

minm∑i=1

n∑j=1

cijxij

s.t.m∑i=1

xij = dj j = 1, . . . , n

n∑j=1

xij = si i = 1, . . . ,m

xij ≥ 0



Exemplary (Nurse) Scheduling Problem (Data)

Given

planning horizon of seven days

nurses work five days in a row

dj : number of nurses needed on day j , j = 1, . . . , 7

Objective

identify the minimum number of nurses to be hired



Exemplary Scheduling Problem (Formulation)Decision variables

xj . . . number of nurses starting to work on day j

Formulation (Linear Program)

min x1 +x2 +x3 +x4 +x5 +x6 +x7x1 +x4 +x5 + x6 +x7 ≥ d1

x1 +x2 +x5 + x6 +x7 ≥ d2

x1 +x2 +x3 + x6 +x7 ≥ d3

x1 +x2 +x3 +x4 +x7 ≥ d4

x1 +x2 +x3 +x4 +x5 ≥ d5

+x2 +x3 +x4 +x5 +x6 ≥ d6

+x3 +x4 +x5 +x6 +x7 ≥ d7

xj ≥ 0 j = 1, . . . , 7



Guidelines

How to formulate

1 define your decision variables

2 write constraints and objective function

Good LP formulations often have

a small number of variables and constraints

sparse constraint matrix

Reminder: Models must be linear, e.g. you are not allowed to multiplytwo variables!



Standard form problems

Definition 5 (Standard Form)

A linear programming problem of the form

min c′x

s.t. Ax = b

x ≥ 0

is said to be in standard form.

Interpretation

x ∈ Rn, Ai is ith column of A

then Ax = b can be written as∑n

i=1Aixi = b

Ai are resource vectors that should synthesize b by using anonnegative amount xi of each resource

ci is the unit cost of each resource



Reduction to standard form

obviously, standard form is special case of general form

we will see that general form can be transformed into standard form

Note

usually, the general form Ax ≥ b is used to develop the theory oflinear programming

when talking about algorithms, e.g. simplex method, standard formAx = b, x ≥ 0, will be used (computationally more convenient)



Reduction rules

Elimination of free variables

Assume xj is an an unrestricted variable in a general form problem

xj ⇔ x+j − x−j with x+

j , x−j ≥ 0

Elimination of inequality constraints

Given an inequality∑n

j=1 aijxj ≤ bi

introduce a slack variable si ≥ 0∑nj=1 aijxj + si = bi

Given an inequality∑n

j=1 aijxj ≥ bi

introduce a surplus variable si ≥ 0∑nj=1 aijxj − si = bi



Reduction to standard form (Example)

The problem

min 2x1 + 4x2

s.t. x1 + x2 ≥ 3

3x1 + 2x2 = 14

x1 ≥ 0

is equivalent to the standard form problem

min 2x1 + 4x+2 − 4x−2

s.t. x1 + x+2 − x−2 − x3 = 3

3x1 + 2x+2 − 2x−2 = 14

x1, x+2 , x

−2 , x3 ≥ 0



Graphical Representation andSolution of LPs



Graphical Representation of LPs

Motivation

provide useful geometric insights into the nature of linearprogramming problems

LPs with two variables can (easily) be solved geometrically

HowTo (Minimization Problem)

1 determine feasible region

2 move line corresponding to objective function in direction ofdecreasing objective value

3 stop if no further moving is possible without leaving the feasibleregion

4 optimal solutions are then given by all feasible points of “objectivefunction line”



Graphical Representation of LPs (Example 1)

min − x1 − x2

s.t. x1 + 2x2 ≤ 3

2x1 + x2 ≤ 3

x1, x2 ≥ 0

line z = c′x = −x1 − x2is perpendicular to c

z = −2 is optimal value

Figure from Bertsimas, Tsitsiklis, 1997



Graphical Representation of LPs (Example 2)

−x1 + x2 ≤ 1 and x1, x2 ≥ 0




Optimal Solutions of a Linear Program

For a Linear Program

min c′x

s.t. Ax ≤ b

with feasible set P = {x | Ax ≤ b}

the following possibilities exist

P = ∅ ⇒ no feasible solution exists

optimal cost is −∞, i.e. inf{c′x | x ∈ P} does not exist, and henceno feasible solution is optimal (LP is unbounded)

a unique optimal solution exists

multiple optimal solutions exist; set of optimal solutions may bebounded or unbounded



Oil Refinery (Modeling)

create different end products from raw oil

heavy oil (H)medium oil (M)light oil (L)

different crack processes produce different quantities

crack process 1: 2H, 2M, 1L costs: 3 EURcrack process 2: 1H, 2M, 4L costs: 5 EUR

required quantities

3H, 5M, 4L

Objective: minimize costs



Oil Refinery (Variables)

Variables

x1: level of production of crack process 1

x2: level of production of crack process 2

Interpretation of x1 = 2.5

process 1 is applied to 2.5 units raw oil

costs 2.5 ∗ 3 EUR

result 2.5 ∗ 2 H, 2.5 ∗ 2 M and 2.5 ∗ 1 L

Each non-negative vector (x1, x2) ∈ R2 describes a certain level ofproduction of both crack processes.



Oil Refinery (Constraints)

at least 3 units H2x1 + x2 ≥ 3

at least 5 units M2x1 + 2x2 ≥ 5

at least 4 units Lx1 + 4x2 ≥ 4

total costsz = 3x1 + 5x2



Oil Refinery (Linear Program)

min 3x1 + 5x2

s.t. 2x1 + x2 ≥ 3

2x1 + 2x2 ≥ 5

x1 + 4x2 ≥ 4

x1 ≥ 0

x2 ≥ 0



Oil Refinery (Graphical Solution)

0

C1 C2 C3

C4

C5

x2

x1��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

5

1 2 3 4 5 6

1

2

3

4

feasible region




��

��

0

C1 C3 C3

C5

x2

x1

C4

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

z=19 e.g.: x=(3,2)

5

1 2 3 4 5 6

1

2

3

4




��

��

0

C1 C2 C3

C4

C5

x2

x1��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

5

1 2 3 4 5 6

1

2

3

4




��

��

0

C1 C2 C3

C4

C5

x2

x1��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

5

1 2 3 4 5 6

1

2

3

4

z=8.5, x*=(2, 0.5)



The Geometry of Linear Programming



Linear Combination

Definition 6

Let x1, x2, . . . , xk ∈ R and

λ =

λ1...λk

∈ Rk .

Then

y =k∑

i=1

λixi

is called linear combination of x1, . . . , xk .



Specific Linear Combinations

Definition 7

If

y =k∑

i=1

λixi

andλ ≥ 0∑k

i=1 λi = 1∑ki=1 λi = 1, λ ≥ 0

then y is called

conicaffineconvex

combination of x1, x2, . . . , xk .



Sets of Linear Combinations

Definition 8

For ∅ 6= S ⊆ Rn

lin(S)cone(S)aff(S)conv(S)

is the

linearconicaffineconvex

hull of S .




Definition 9

IfL = lin(S)L = cone(S)L = aff(S)L = conv(S)

, then L is a

linear spaceconeaffine spaceconvex set



Subspaces

Definition 10

L ⊆ Rn is a linear subspace of Rn ⇔there exists A ∈ Rm×n with L = {x ∈ Rn | Ax = 0}.

Definition 11

L ⊆ Rn is a affine subspace of Rn ⇔there exist A ∈ Rm×n and b ∈ Rm with L = {x ∈ Rn | Ax = b}.



Hyperplanes, Halfspaces, and Polyhedra

Definition 12 (Polyhedron)

A polyhedron P is a set that can be described in the formP = {x ∈ Rn | Ax ≥ b} where A ∈ Rm×n and b ∈ Rm.(cmp. feasible set of LPs)

Definition 13

A set S ⊆ Rn is bounded if there exists a constant K such that theabsolute value of every component of every element of S is less thanequal to K .

Definition 14 (Polytope)

A bounded polyhedron is called polytope, i.e.

P ⊆ {x ∈ Rn | ||x|| ≤ B} for some B ≥ 0.




Definition 15

Let a be a nonzero vector in Rn and let b be a scalar

(a) the set {x ∈ Rn | a′x ≥ b} is called a halfspace

(b) the set {x ∈ Rn | a′x = b} is called a hyperplane

a hyperplane is the boundary of a corresponding halfspace

a hyperplane is an affine subspace (m = 1)

halfspaces are polyhedrons

each polyhedron is equal to the intersection of a finite number ofhalfspaces

a polyhedron P is a convex set, i.e. if x, y ∈ P, thenλx + (1− λ)y ∈ P⇒ the feasible set of each LP is a convex set



Representation

Theorem (Minkowski and Weyl)

A polyhedron P ∈ Rn can be represented in the form

P = conv(V ) + cone(E )

with V and E being finite subsets of Rn.

Remark

polytopes can be represented as P = conv(V )



Example

Polytope P represented as:

P =

{(x

y

)∈ R2 | x ≤ 1, y ≤ 1, x + y ≥ 1

}



Faces

Definition 16 (Valid Inequality)

The inequality a′x ≤ a0 is called a valid inequality for a polyhedron P ifit is satisfied by all points in P.

Definition 17

Given a polyhedron P = {x ∈ Rn : Ax ≤ b},F ⊆ P is a face of P, if there exists a valid inequality a′x ≤ a0 forP, such that F = {x ∈ P : a′x = a0}.a face F is a proper face if F 6= ∅ and F 6= P.

A facet of P is a maximal (relative to inclusion) proper face of P.

A “corner point” of P is a face of P consisting of one element.



Faces

x_1 + x_2 <=2

x_1 <=1facet:

corner point:

P

1

1

0



Extreme Points

Definition 18 (Extreme Point)

Let P be a polyhedron. A vector x ∈ P is an extreme point of P if wecannot find two vectors y, z ∈ P, both different from x, and a scalarλ ∈ [0, 1], such that x = λy + (1− λ)z.


x is an extreme point of P

u, v, w, y, z are no extremepoints of P



Vertices

Definition 19 (Vertex)

Let P be a polyhedron. A vector x ∈ P is a vertex of P if there existssome c such that c′x < c′y for all y satisfying y ∈ P and y 6= x.


x is a vertex of P

w is not a vertex of P



Active ConstraintsConsider a polyhedron P ⊆ Rn defined by linear equality and inequalityconstraints

ai′x ≥ bi , i ∈ M1

ai′x ≤ bi , i ∈ M2

ai′x = bi , i ∈ M3

where M1, M2, and M3 are finite index sets, ai ∈ Rn, and bi ∈ R.

Definition 20

If a vector x∗ satisfies ai′x∗ = bi for some i ∈ M1, M2, or M3, we saythat the corresponding constraint is active or binding at x∗.

If n constraints are active at x∗, x∗ satisfies a system of n linearequations in n unknowns. This system has a unique solution if and onlyif these n equations are linearly independent.



Theorem 21

Let x∗ ∈ Rn and I = {i | a′ix∗ = bi} be the set of indices of constraintsthat are active at x∗. Then the following are equivalent

(a) there exist n vectors in the set {ai | i ∈ I}, which are linearlyindependent

(b) the span of the vectors ai , i ∈ I , is Rn, i.e. every element of Rn canbe expressed as linear combinations of the vectors ai , i ∈ I

(c) the system of equations a′ix∗ = bi , i ∈ I , has a unique solution



Basic Solutions

Definition 22

Consider a polyhedron P defined by linear equality and inequalityconstraints, and let x∗ be an element of Rn

(a) vector x∗ is a basic solution if

(i) all equality constraints are active(ii) out of the constraints that are active at x∗, there are n of them that

are linearly independent

(b) if x∗ is a basic solution that satisfies all of the constraints, we saythat it is a basic feasible solution (BFS)



Active Constraints and Basic Feasible Solutions

P = {(x1, x2, x3) | x1 + x2 + x3 = 1, x1, x2, x3 ≥ 0}


3 constraints active at A,B, C , and D

2 constraints active at E ,i.e. x1 + x2 + x3 = 1 andx2 = 0

A, B, and C are basicfeasible solutions



Extreme Point, Vertex, Basic Feasible Solution

Theorem 23

Let P be a nonempty polyhedron and let x∗ ∈ P. Then, the following areequivalent:

(a) x∗ is a vertex

(b) x∗ is an extreme point

(c) x∗ is a basic feasible solution

Corollary 24

Given a finite number of linear inequality constraints, there can only be afinite number of basic feasible solutions.

Definition 25

Two distinct basic solutions are adjacent if there are n − 1 linearlyindependent constraints that are active at both of them.



Polyhedra in Standard Form

P = {x ∈ Rn | Ax = b, x ≥ 0}

Essential for the development of the simplex algorithm!

Theorem 26

Consider the constraints Ax = b and x ≥ 0 and assume that the m × nmatrix A has linearly independent rows. A vector x ∈ Rn is a basicfeasible solution if and only if we have Ax = b, and there exists indicesB(1), . . . ,B(m) such that:

(a) The columns AB(1), . . . ,AB(m) are linearly independent and

(b) if i 6= B(1), . . . ,B(m), then xi = 0.



Basic Solutions for Polyhedra in Standard Form

Constructing basic solutions

1 Choose m linearly independent columns AB(1), . . . ,AB(m)

2 Let xi = 0 for all i 6= B(1), . . . ,B(m)

3 Solve the system of m equations Ax = b for the unknownsxB(1), . . . , xB(m)

if a basic solution constr. is nonneg. it is a basic feasible solutionif x is a basic solution, then xB(1), . . . , xB(m) are called basicvariables; the remaining variables are called nonbasicthe columns AB(1), . . . ,AB(m) are called the basic columns and sincethey are linearly independent, they form a basis of Rm

B(1), . . . ,B(m) are the basic indicesarranging the m basic columns next to each other, we obtain anm ×m matrix B called basis matrixBxB = b yields xB = B−1b




standard form problem with n = 4and m = 2

A1, A2 form a basis; correspondingbasic solution is infeasible (negativevalue of x2 is needed)

A1, A3 form another basis; basicsolution is feasible

A1, A4 do not form a basis(linearly dependent)



The Full Row Assumption on A

Theorem 27

Let P = {x | Ax = b, x ≥ 0} be a nonempty polyhedron, where A is amatrix of dimensions m × n with rows a′1, . . . , a

′m. Suppose that

rank(A) = k < m and that the rows a′i1 , . . . a′ik

are linearly independent.Consider the polyhedron

Q = {x | a′i1x = bi1 , . . . , a′ikx = bik , x ≥ 0}.

Then Q = P

Which shows that the full row rank assumption on the matrix A inTheorem 26 results in no loss of generality.



Degeneracy

at each basic solution we have at least n active constraints

may be more than n (of course not linearly independent) →degenerate basic solution

huge impact on algorithms for solving LPs

Definition 28

A basic solution x ∈ Rn is said to be degenerate if more than n of theconstraints are active at x.



Degeneracy


The points A and C are degenerate basic feasible solutions. The pointsB and E are nondegenerate basic feasible solutions. The point D is adegenerate basic solution.



Degeneracy in Standard Form Problems

in standard form, the m equality constraints are active at a basicsolution

having more than n active constraints is the same as having morethan n −m variables at zero level

Definition 29

Consider the standard form polyhedron P = {x ∈ Rn | Ax = b, x ≥ 0}and let x be a basic solution. Let m be the number of rows in A. Thevector x is a degenerate basic solution if more than n −m of thecomponents of x are zero.

For a degenerate basic solution, there are typically multiple possibilitiesto choose the n −m nonbasic variables, i.e. several bases correspond tothe same basic solution.



Degeneracy and Representation

Degeneracy is, in general, not a geometric (representation independent)property, but rather depends on the particular representation of apolyhedron.

Example

P = {(x1, x2, x3) | x1 − x2 = 0, x1 + x2 + 2x3 = 2, x1, x2, x3 ≥ 0}n = 2, m = 3, and n −m = 1

→ (1, 1, 0) is not degenerated, but (0, 0, 1) is

P = {(x1, x2, x3) | x1 − x2 = 0, x1 + x2 + 2x3 = 2, x1 ≥ 0, x3 ≥ 0}→ (0, 0, 1) not degenerated in this representation



Existence of Extreme Points

Definition 30

A polyhedron P ⊂ Rn contains a line if there exists a vector x ∈ P and anonzero vector d ∈ Rn such that x + λd ∈ P for all scalars λ.

Theorem 31

Suppose that the polyhedron P = {x ∈ Rn | a′ix ≥ bi , i = 1, . . . ,m} isnonempty. Then, the following are equivalent:

(a) P has at least one extreme point

(b) P does not contain a line

(c) there exist n vectors out of a1, . . . , am which are linearly independent



Existence of Extreme Points

a bounded polyhedron does not contain a line

the positive orthant {x | x ≥ 0} does not contain a line

since a polyhedron in standard form is contained in the positiveorthant, it does not contain a line either

Corollary 32

Every nonempty bounded polyhedron and every nonempty polyhedron instandard form has at least one basic feasible solution.



Optimality of Extreme Points

Theorem 33

Consider the linear programming problem of minimizing c′x over apolyhedron P. Suppose that P has at least one extreme point and thatthere exists an optimal solution. Then there exists an optimal solutionwhich is an extreme point of P.




Optimality of Extreme Points

Theorem 34 (stronger variant)

Consider the linear programming problem of minimizing c′x over apolyhedron P. Suppose that P has at least one extreme point. Theneither the optimal cost is equal to −∞, or there exists an extreme pointwhich is optimal.

Corollary 35

Consider the linear programming problem of minimizing c′x over anonempty polyhedron. Then, either the optimal cost is equal to −∞ orthere exists an optimal solution.



Summary

Conclusions regarding the solutions of LPs

(a) If the feasible set is nonempty and bounded, there exists an optimalsolution. Furthermore, there exists an optimal solution which is anextreme point.

(b) If the feasible set is unbounded, there are three possibilities

(i) There exists an optimal solution which is an extreme point.(ii) There exists an optimal solution, but no optimal solution is an

extreme point. (Only possible if the feasible set has no extremepoints; This never happens in standard form problems)

(iii) The optimal cost is −∞.

(c) If the optimal cost is finite and the feasible set contains at least oneextreme point, then there are only finitely many extreme points.

(d) The number of extreme points may increase exponentially with thenumber of variables and constraints.



The Simplex Method



Historical Notes

simplex method was pioneered by Dantzig in 1947

comprehensive text: G. B. Dantzig, Linear Programming andExtensions, Princeton University Press, 1963

Basic Strategy

start from a vertex (BFS) of the polyhedra

may be difficult to find!

move from one BFS to another along the edges of the feasible set ina cost reducing direction

which directions / edges are possible?which one to choose?

repeat while further improvement is possible

global optimum?



Optimality Conditions

basic strategy of many optimization algorithms: given a feasiblesolution, search its neighborhood to find a nearby feasible solutionwith lower costs (as long as it exists)

→ such algorithms terminate in a locally optimal solution which neednot to be globally optimal

LP: optimization of a convex function over a convex set!

Theorem 36 (Local Minima of Convex Functions)

Let F : Rn → R be a convex function and S ⊂ R be a convex set. Let x∗

be an element of S. Suppose x∗ is a local optimum for the problem ofminimizing f (x) over S; i.e. there exists some ε > 0 such thatf (x∗) ≤ f (x) for all x ∈ S for which ||x− x∗|| ≤ ε. Then x∗ is globallyoptimal, i.e. f (x∗) ≤ f (x) for all x ∈ S.



Assumptions

In this chapter, we consider the standard form problem

min c′x

s.t. Ax = b

x ≥ 0

where P is the corresponding feasible set

A is a m × n matrix with linearly independent rows

Ai is the ith column of A

a′i is the ith row of A



Feasible Directions

given a point x ∈ P we want to move in the direction of d ∈ Rn

we do only want to consider choices for d that do not immediatelytake us outside the feasible set

Definition 37 (Feasible Direction)

Let x be an element of a polyhedron P. A vector d ∈ Rn is said to be afeasible direction at x, if there exists a positive scalar θ for whichx + θd ∈ P



Basic Direction

given BFS x, i.e. Ax = b, basis matrix B, basic variablesxB = (xB(1), . . . , xB(m)) (xB = B−1b)

move away from x to x + θd by selecting a nonbasic variable xj(initially zero) increasing it to θ

x + θd, with dj = 1 (zero for all other nonbasic variables)

A(x + θd) = b should hold (feasibility) and Ax = b holds and θ > 0,we need Ad = 0

→ 0 = Ad =∑n

i=1Aidi =∑m

i=1AB(i)dB(i) + Aj = BdB + Aj

⇒ dB = −B−1Aj

which is called jth basic direction



Basic Direction

equality constraints are respected when moving along the basicdirection d

nonnegativity constraints?

variable xj is increasedother nonbasic variables stay zero

Other basic variables

(a) suppose x is a nondegenerate BFS, i.e. xB > 0, and thusxB + θdB ≥ 0 is θ is sufficiently small (→ feasibility is maintainedand d is a feasible direction)

(b) if x is degenerate, d is not always a feasible direction! (a basicvariable xB(i) may be zero while dB(i) is negative)



Effects on the Cost Function

jth basic direction d: rate c′d of cost change along d is given byc′BdB + cj , where cB = (cB(1), . . . , cB(m)), which is equal to

→ cj − c′BB−1Aj

cj . . . cost per unit increase of xj−c′BB

−1Aj . . . cost of the compensating change in the basic variablesdue to Ax = b

Definition 38 (Reduced Cost)

Let x be a basic solution, let B be an associated basis matrix, and let cBbe the vector of costs of the basic variables. For each j , we define thereduced cost c j of the variable xj according to the formula

c j = cj − c′BB−1Aj

(Note: reduced costs of basic variables are zero)



Optimality Conditions

Theorem 39

Consider a basic feasible solution x associated with a basis matrix B, andlet c be the corresponding vector of reduced costs.

(a) if c ≥ 0, then x is optimal

(b) if x is optimal and nondegenerate, then c ≥ 0

Definition 40

A basis matrix B is said to be optimal if:

(a) B−1b ≥ 0, and

(b) c′ = c′ − c′BB−1A ≥ 0′



Development of the Simplex Method

Main Missing Part

How to move to a better BFS, when a profitable basic direction isdiscovered

assume that every BFS is nondegenerate

reduced costs have been computed and c j < 0 (jth basic direction d)

when moving along d, xj becomes positive; we say that xj (or Aj)enters the basis



How far can we move?

∃j : c j < 0: jth basic direction allows for a cost reduction

since costs are decreasing along d we want to move as far aspossible to x + θ∗d, where

θ∗ = max{θ ≥ 0 | x + θd ∈ P}

if d ≥ 0, then x + θd ≥ 0, for all θ ≥ 0, hence θ∗ =∞ (unbounded)

if di < 0, then xi + θdi ≥ 0⇒ θ ≤ −xi/di

must be satisfied for every i with di < 0→ θ∗ = min{i|di<0}(−xi/di )

since we only need to consider basis variables→ θ∗ = min{i=1,...,m|dB(i)<0}(−xB(i)/dB(i))

θ∗ > 0, because xB(i) > 0 (nondegeneracy)



New Basic Feasible Solution y

y = x + θ∗d

since xj = 0 and dj = 1, we have yj = θ∗ > 0

Let l be a minimizing index, i.e.

−xB(l)

dB(l)= min{i=1,...,m|dB(i)<0}

(−

xB(i)

dB(i)

)= θ∗

in particular dB(l) < 0 and xB(l) + θ∗dB(l) = 0

basic variable xB(l) becomes zero, whereas nonbasic variable xjbecomes positive

→ xj should replace xB(l) in the basis, i.e. B is replaced by new basis

matrix B by replacing AB(l) with Aj

we have y 6= x and c′y < c′x



New Basic Feasible Solution y

Theorem 41

Let y, B, be constructed as detailed before. Then

(a) The columns AB(i), i 6= l , and Aj are linearly independent and,

therefore B is a basis matrix

(b) The vector y = x + θ∗d is a basic feasible solutions associated withthe basis matrix B.



An Iteration of the Simplex Method

1 start with a basis (columns AB(1), . . . ,AB(m)), and associated BFS x

2 compute reduced costs c j = cj − c′BB−1Aj for all nonbasic indices j ;

if all of them are nonnegative, current solution is optimal (algorithmterminates); else choose some variable j for which c j < 0

3 compute u = −dB = B−1Aj ; if no component of u is positive, wehave θ∗ =∞ and the optimal cost is −∞ (algorithm terminates)

4 if some component of u is positive, let

θ∗ = min{i=1,...,m|ui>0}

xB(i)

ui

5 let l be such that θ∗ = xB(l)/ul ; form new basis by replacing AB(l)

with Aj (xj enters basis, xB(l) leaves basis)



Correctness of Simplex (Nondegenerate case)

Theorem 42

Assume that the feasible set is nonempty and that every basic feasiblesolution is nondegenerate. Then, the simplex method terminates after afinite number of iterations. At termination, there are the followingpossibilities:

(a) We have an optimal basis B and an associated basic feasiblesolution which is optimal.

(b) We have found a vector d satisfying Ad = 0, d ≥ 0, and c′d < 0,and the optimal cost is −∞.



The Simplex Method for Degenerate Problems

Suppose, we apply the same algorithm in the presence of degeneracy.Then, two new possibilities may be encountered:

(a) If current BFS x is degenerate, θ∗ can be equal to zero and thusy = x

happens if some xB(l) is equal to zero and dB(l) < 0

still, we can define a new basis B, by replacing AB(l) by Aj

(b) if θ∗ > 0, more than one of the original basis variables may becomezero at x + θ∗d; since only one variable exists the basis, we get anew BFS which is degenerate

basic changes while “staying at the same BFS” do not reduce cost

degeneracy may lead to cycling (i.e. algorithm may loop indefinitely)



Simplex and Degeneracy (Example)




Pivot Selection

simplex algorithm has certain degrees of freedom:selection of nonbasic variable xj with negative reduced cost

Dantzig’s rule: most negative reduced costlargest cost decrease (more computational effort!)Bland’s rule: smallest subscript (smallest index j such that c j < 0)(Devex, steepest edge, candidate lists, partial pricing,. . . )

selection of variable leaving basis (if multiple indices attain theminimum in definition of θ∗)

Bland’s rule: smallest subscript

rules for making such choices are called pivoting rules



Full Tableau Implementation

Structure of the representation

−cTBB−1b cT − cTBB−1A

B−1b B−1A

or, in more detail

−cTB xB c1 . . . cn

xB(1) | |... B−1A1 . . . B−1An

xB(m) | |



Full Tableau Implementation – Iteration

1 start with tableau associated with basis matrix B and BFS x

2 examine reduced costs (0th row of tableau); if they are nonnegative,current solution is optimal; else choose some j with c j < 0

3 consider vector u = B−1Aj (jth column of tableau):

if no component of u is positive, optimal cost is −∞; terminationfor each i with ui > 0 compute xB(i)/ui ; let l be index of a row withsmallest ratio; column AB(l) exists basis, Aj enters basis

4 add to each row a constant multiple of the pivot row (lth row) sothat ul (pivot element) becomes one and all other entries of thepivot column become zero



Full Tableau Implementation (Example from [1])

After introducing slack variables, we obtain the standard form problem




Basic feasible solution: x = (0, 0, 0, 20, 20, 20); accordingly B(1) = 4,B(2) = 5, B(3) = 6; B = I

→ x1 has negative reduced costs (enters basis)→ pivot column u = (1, 2, 2); form xB(i)/ui

→ minimum at i = 2 and i = 3; choose l = 2, i.e. x5 leaves the basis




New (degenerate) BFS: x = (10, 0, 0, 10, 0, 0); B(1) = 4, B(2) = 1,B(3) = 6

→ choose x3 to enter basis; pivot column u = (1, 1,−1)→ u3 < 0, hence calculate xB(i)/ui for i = 1, 2 only→ tie; we chose x4 to exit the basis




New (degenerate) BFS: x = (0, 0, 10, 0, 0, 10)

→ x2 enters basis, x6 exists basis




New BFS: x = (4, 4, 4, 0, 0, 0)

→ optimal solution, since all reduced costs are nonnegative!



Full Tableau Implementation (Example from [1])Feasible Set; Path A-D-B-E




Full Tableau Implementation: Cycling (Example from [1])

Initial tableau (pivot elements are marked using “*”)

pivoting rules

nonbasic variable with most negative reduced cost c j enters the basis

out of eligible basic variables, the one with smallest index (subscript)exists the basis



After iteration 1:

After iteration 2:



After iteration 3:

After iteration 4:



After iteration 5:

After iteration 6: same basis and tableau that we started with (cycling!)



Anticycling

appropriate pivoting rules prevent cyclingLexicographic pivoting rule

choose entering variable arbitrarily (with neg. reduced cost)for each row i with ui > 0, divide ith row of tableau by ui and choselexicographic smallest row to identify variable leaving the basis

Bland’s Rulefind smallest j for which c j < 0; xj enters basisin case of ties for exiting variables choose the one with smallest i

using one of these rules, the simplex algorithm terminates afterfinitely many steps

Definition 43 (Lexicographically Smaller)

A vector u ∈ Rn is said to be lexicographically larger (smaller) thananother vector v ∈ Rn if u 6= v and the first nonzero component of u− vis positive (negative).



Finding an initial BFS

to start the simplex algorithm, an initial BFS is needed

sometimes straightforward: Ax ≤ b,b ≥ 0⇒ Ax + s = b⇒ (x, s) = (0,b)

in general, however, finding a BFS is not easy (→ auxiliary LP)



Finding an initial BFSConsider the problem

min c′x

s.t. Ax = b

x ≥ 0

w.l.o.g. we assume b ≥ 0 (possibly multiply some constraints by −1)

introduce artificial variables y ∈ Rm

Auxiliary problem

minm∑i=1

yi

s.t. Ax + y = b

x, y ≥ 0

which has the BFS x = 0 and y = b.VU Algorithmics M. Leitner 100


The Two-Phase Simplex Method (Phase I)

1 possibly multiply some constraints by −1 so that b ≥ 0

2 add artificial variables y1, . . . , ym, if necessary and apply simplex toauxiliary problem with cost

∑mi=1 yi

3 if optimal cost in auxiliary problem is positive, original problem isinfeasible (termination)

4 if optimal cost in auxiliary problem is zero, feasible solution tooriginal problem has been found; if no artificial variable is in the finalbasis, eliminate artificial variables and columns to obtain feasiblebasis for original problem

5 Repeat: if lth basic variable is artificial, examine lth entry of thecolumns B−1Aj , j = 1, . . . , n; if all entries are zero, lth constraint isredundant and is eliminated; otherwise apply change of basis: lthbasic variable exists and xj enters (lth entry of jth column isnonzero)



Two-Phase Simplex (Example from [1])

Initial LP

Auxiliary LP



Two-Phase Simplex (Phase I, Example from [1])

BFS (x5, x6, x7, x8) = b = (3, 2, 5, 1), B = IcB = (1, 1, 1, 1), c i = −c′BAi

→ x4 enters the basis, x8 exits the basis




B = I, only 0th row of the tableau changes





→ x2 enters the basis, x6 exits the basis(degenerate pivot with θ∗ = 0)




cost is zero, i.e. feasible solution to the original problemx7 is still in the basis and we need to drive x7 out of the basisx7 is 3rd basis variable and 3rd entry of columns B−1Aj ,j = 1, . . . , 4, (original variables) is zeroindicates that A has linearly dependent rows and we can remove 3rdrow (redundant constraint)



Two-Phase Simplex (Phase I, Example from [1])Initial tableau for original problem:

we may now compute reduced costs of the original variables andapply simplex to original problem

artificial variable x8 was unnecessary (since x4 appears in a singleconstraint only)

in this case (typical example: slack variable) we can let that variablein the initial basis



Big-M Method

Auxiliary problem with y ∈ Rm

minn∑

j=1

cjxj + Mm∑i=1

yi

s.t. Ax + y = b

x ≥ 0

y ≥ 0

combines both faces into a single one

for sufficiently large M, artificial variables are driven to zero



Computational Complexity of the Simplex Algorithm

computational effort at each iteration: O(mn)

number of iterations?

for all known variants, there are worst case examples such thatexponentially many extreme points are visited (Klee-Minty cube)in practice usually only O(m) pivots are needed

Important open question

Is it true that for every pivoting rule there are examples where thesimplex method takes an exponential number of iterations?



Computational Complexity - Spanning Path


maxn∑

i=1

xi

ε ≤ x1 ≤ 1

εxi−1 ≤ xi ≤ 1− εxi−1 i = 2, . . . , n

→ unit cube has 2n vertices, simplex needs 2n − 1 pivots (costs arestrictly decreasing along path due to ε)



Alternative Approaches

There are algorithms that solve LPs in polynomial time:

ellipsoid method: L. Khachian (1979): mainly of theoretical value(numerics, long running times)

interior point methods: N. Karmarkar (1984), partly competitive(no “warm start”), very important for theoretical research!

Simplex

warmstart is possible (e.g. after adding rows or columns)

has good practical (empirical) performance

is by far the most commonly used method in practice andincorporated in most commercial codes for LP



Summary

simplex moves from on extreme point of the feasible set to another,each time reducing the cost, until an optimal solution is reached

cornerstones

optimality conditions (nonnegative of reduced costs) allow to testwhether current basis is optimalsystematic method for performing basis changes while not optimal

degeneracy causes substantial difficulties including the possibility ofcycling (since a change of basis may keep us at the same BFS, withno resulting cost improvement)

cycling can be avoided by choosing suitable pivoting rules

two-phase simplex can be used to obtain initial BFS

simplex method is a rather efficient algorithm in practice



Duality Theory



Motivation

Consider the Primal Problem

min c′x

s.t. Ax = b

x ≥ 0

with an optimal solution x∗ (assumed to exist).

Relaxed Problem:

g(p) = min c′x + p′(b− Ax)

x ≥ 0

where p is a price vector (with dimension of b) and p′(b−Ax) a penalty.



Motivation

Since the relaxed problem allows for more options, we should haveg(p) ≤ c′x∗. Indeed,

g(p) = minx≥0

[c′x + p′(b− Ax)

]≤ c′x∗ + p′(b− Ax∗) = c′x∗

since x∗ is feasible and satisfies Ax∗ = b. Hence, each p yields a lowerbound g(p) for the optimal cost c′x∗.

Dual problem

“search for the tightest lower bound of the type” max g(p)

Main result in duality theory: Optimal cost in the dual problem is equalto the optimal cost c′x in the primal (Violating Ax = b has no value ifprices are chosen according to an optimal solution for the dual problem)



Motivation

By definition of g(p) we have

g(p) = minx≥0

[c′x + p′(b− Ax)

]= p′b + min

x≥0(c′ − p′A)x

Note that

minx≥0

(c′ − p′A)x =

{0 if c′ − p′A ≥ 0′

−∞ otherwise

Thus, when maximizing g(p), we only need to consider values of p forwhich g(p) is not −∞.



Motivation

Dual (Linear Programming) Problem

The dual problem is the same as the linear programming problem

max p′b

s.t. p′A ≤ c′

Note:

we started with equality constraints Ax = b and ended with noconstraints on the sign of the price vector p



Motivation

We can replace inequality constraints (primal), i.e. Ax ≥ b, byAx− s = b, x ≥ 0, which can be written as

[A | −I][xs

]= b.

Leading to the dual constraints

p′ [A | −I] ≤[c′ | 0′

],

or equivalentlyp′A ≤ c′, p ≥ 0.



Motivation

Also, if x is free rather than sign-constrained, we use

minx

(c′ − p′A)x =

{0 if c′ − p′A = 0′

−∞ otherwise

to end up with the constraint

p′A = c′

in the dual.



Motivation

Construction of the dual of a minimization problem

we have a vector of parameters (dual variables) p

for every p we have a method for obtaining a lower bound on theoptimal primal cost

dual problem is a maximization problem that looks for the tightestlower bound

for some vectors p we get −∞ (no useful information)

→ we only need to maximize over those p leading to nontrivial lowerbounds



The dual problem

Let A be a matrix with rows ai and columns Aj . Then the followingdefines a pair of primal and dual problems:

min c′x max p′b

s.t. aix ≥ bi i ∈ M1 s.t. pi ≥ 0 i ∈ M1

aix ≤ bi i ∈ M2 pi ≤ 0 i ∈ M2

aix = bi i ∈ M3 pi free i ∈ M3

xj ≥ 0 j ∈ N1 p′A′j ≤ cj j ∈ N1

xj ≤ 0 j ∈ N2 p′A′j ≥ cj j ∈ N2

xj free j ∈ N3 p′A′j = cj j ∈ N3



Example

Primal Dual

min x1 + 2x2 + 3x3 max 5p1 + 6p2 + 4p3

s.t. − x1 + 3x2 = 5 s.t. p1 free

2x1 − x2 + 3x3 ≥ 6 p2 ≥ 0

x3 ≤ 4 p3 ≤ 0

x1 ≥ 0 −p1 + 2p2 ≤ 1

x2 ≤ 0 3p1 − p2 ≥ 2

x3 free 3p2 + p3 = 3



Relations between primal and dual variables and constraints

PRIMAL minimize DUAL maximize

≥ bi ≥ 0constraints ≤ bi variables ≤ 0

= bi free

≥ 0 ≤ cjvariables ≤ 0 constraints ≥ cj

free = cj

Theorem 44

If we transform the dual into an equivalent minimization problem andthen form its dual, we obtain a problem equivalent to the originalproblem.

Or more compact: “The dual of the dual is the primal”.



Duality Theorems

Theorem 45 (Weak duality)

If x is a feasible solution to the primal problem an p is a feasible solutionto the dual problem, then

p′b ≤ c′x.

Hence the cost of any dual solution is a lower bound for the optimal cost.

Corollary 46

(a) If the optimal cost in the primal is −∞ (unbounded), then the dualproblem must be infeasible.

(b) If the optimal cost in the dual is +∞ (unbounded), them the primalproblem must be infeasible.



Duality Theorems

Corollary 47

Let x and p be feasible solutions to the primal and the dual, respectively,and suppose that p′b = c′x. Then x and p are optimal solutions to theprimal and the dual, respectively.

Theorem 48 (Strong duality)

If a linear programming problem has an optimal solution, so does itsdual, and the respective optimal costs are equal.

Theorem 49 (Complementary Slackness)

Let x and p be feasible solutions to the primal and the dual problem,respectively. They are optimal if and only if:

pi (a′ix− bi ) = 0 ∀i

(cj − p′Aj)xj = 0 ∀j



Summary

for each LP, we can define a dual LP

the dual of the dual is the primal

strong duality theorem!

allows to develop the dual simplex algorithm (important for warmstart when solving LPs)

(more advanced) duality theory provides powerful tools for furtherinsights and understanding of LPs and polyhedra


linear programming

Documents

full tableau

exemplary

tightest lower

jth basic

basic feasible

columns b1

linear programming

xj enters