linear programming - ucsb computer science …suri/cs231/lp.pdf · subhash suri uc santa barbara...

Subhash Suri UC Santa Barbara

Linear Programming

• LP is a general method to solveoptimization problem with linearobjective function and linear constraints.

• Diet Problem Example: Feeding an army.

• 4 menu choices: Fish, Pizza, Hamburger,Burrito.

• Each choice has benefits (nutrients) and acost (calories).

A C D CaloriesF 200 90 100 600P 75 80 250 800H 275 80 510 550B 500 95 200 450

RDA 2000 300 475

• Determine least-caloric diet that satisfiesRDA of nutrients.


LP Formulation

• Introduce variables xF , xP , xH, xB for theamount (units) consumed of each food.

• Obj. Function is total calories.

minimize 600xF + 800xP + 550xH + 450xB

• Each of the nutrient constraint can beexpressed as a linear function:

200xF + 75xP + 275xH + 500xB ≥ 2000

90xF + 80xP + 80xH + 95xB ≥ 300

100xF + 250xP + 510xH + 200xB ≥ 475

• Finally, all food quantities consumedshould be non-negative:

xF ≥ 0; xP ≥ 0; xH ≥ 0; xB ≥ 0

• This is an example of Linear Prog.


Manufacturing Example

• A plant has 3 types of machines, A,B,C.

• Choice of 4 products: 1, 2, 3, 4.

• Resource consumption (how muchmachine time a product uses) and profitper unit.

A B C Profit1 2 0.5 1.5 3.52 2 2 1 4.23 0.5 1 3 6.54 1.5 2 1.5 3.8

Avail. 20/wk 30/wk 15/wk

• Machine shop operates 60 hrs/week.

• Determine most profitable product mix.


LP Formulation

• Introduce variables x1, x2, x3, x4 for theunits of different products manufactured.

• Obj. Function is total profit.

maximize 3.5x1 + 4.2x2 + 6.5x3 + 3.8x4

• Each of the machines provides aconstraint, expressed as a linear function:

2x1 + 2x2 + 0.5x3 + 1.5x4 ≤ 1200

0.5x1 + 2x2 + x3 + 2x4 ≤ 1800

1.5x1 + x2 + 3x3 + 1.5x4 ≤ 900

• Finally, all product quantities should benon-negative:

x1, x2, x3, x4 ≥ 0


Transportation Example

• m supply sites, and n demand sites.

• Supply site i produces ai units;demand site j requires bj units.

• cij cost of shipping one unit from i to j.

Demand NodesSupply Nodes

b1

a1

a2

am

bn

b2

..

..

• Determine optimal shipping cost.


LP Formulation

• Let xij be the amount shipped from i to j.Then, the LP is

minm∑

i=1

n∑

j=1

cijxij

• subject ton∑

j=1

xij = ai, i = 1, 2, . . . , m

m∑

i=1

xij = bj, j = 1, 2, . . . , n

m∑

i=1

ai =n∑

j=1

bj

xij ≥ 0


General Form of LP

• n unknowns x1, . . . , xn, and m constraints.

max c1x1 + c2x2 + · · ·+ cnxn

• subject to

a11x1 + a12x2 + · · ·+ a1nxn ≤ b1

a21x1 + a22x2 + · · ·+ a2nxn ≤ b2

...

am1x1 + am2x2 + · · ·+ amnxn ≤ bm

x1, x2, · · · , xn ≥ 0

• A company making n products, eachrequiring some amount each of mresources. Item i has profit ci.

• bj is total resource of type j. aij is theamount of resource j needed by item i.


Matrix Form

• maximize cx

• subject to Ax ≤ b

• x ≥ 0

• The input data are:

– the m× n coefficient matrix A,– the m-vector of right hand sides b, and– the n-vector of objective costs c.


Slack Variables

• An inequality ai1x1 + · · ·+ ainxn ≤ bi can beconverted to an equation by introducing aslack variable si ≥ 0:

ai1x1 + · · ·+ ainxn + si = bi

• Similarly, an inequality like∑n

j=1 aijxj ≥ bi

can be written as

ai1x1 + · · ·+ ainxn − si = bi

• The Standard form Linear Program is

maximize cx

subject to Ax = b

x ≥ 0

• Slack variable coefficients in the objfunction are typically zero.


Geometrical Method

• 2-variable LP can be solved graphically.

• Useful teaching tool: Helps build intuitionabout the problem.

• Not a practical method for practical linearprogramming problems.

• We plot constraints on a rectangularcoordinate system.

• A constraint is drawn as a line. Theinequality constraint restricts the feasibleregion to one side of the line.

• The non-negative constraints x1, x2 ≥ 0restrict the feasible region to the firstquadrant.


Example

• We will consider the following LP:

max z = 2x1 + 3x2

• subject to

x1 − 2x2 ≤ 4

2x1 + x2 ≤ 18

x2 ≤ 10

x1, x2 ≥ 0


Illustration

• Building the feasible region, oneconstraint at a time.

x1

x2

x1

x2

x1

x2

x1

x2

x1 − 2x2 = 4

2x1 + x2 = 18

x2 = 10

FeasibleRegion


Optimal Solution

• Examine Level curves: z = 1, 2 etc.

• E.g. z = 24 defines the line 2x1 + 3x2 = 24.Similarly, for z = 30.

• Parallel lines–they have same slope.

• Line with highest z value that intersectsfeasible region.

.

.

. .

x1

x2

(4,10)(0,10)

(8,2)

(4,0)(0,0)

z = 24

z = 30

(2,3)

Optimal ExtremePoint

z* = 38


Algebra of LP

.

.

. .

x1

x2

(4,10)(0,10)

(8,2)

(4,0)(0,0)

z = 24

z = 30

(2,3)

Optimal ExtremePoint

z* = 38

• Important points:Feasible region is convex polyhedron,optimal solution is a corner.

• Simplex Method is a famous method forsolving LP, invented by George Dantzig.

• Introduce algebra and geometry toexplain how it works.


Linear Algebra

H+ = {x : ax >= b}

H− = {x : ax <= b}

a

H: ax = b

• A hyperplane in Rn is the set of pointsx = {x1, x2, . . . , xn} that satisfy ax = b,where a ∈ Rn and b ∈ R.

• Examples: Line a1x1 + a2x2 = b in R2.

• H+ = {x : ax ≥ b} is the positive halfspace.

• Vector a is gradient of the linear functionf(x) = ax, and so is normal to H.

• A polyhedral set is the intersection ofhalfspaces.


Convexity

y

x

Convex Sets

y

x

Non−Convex Sets

• A set S is convex if for any two pointsx,y ∈ S, the line segment xy also lies in S.

• That is, αx + (1− α)y ∈ S if x,y ∈ S, for allα ∈ [0, 1].

• Examples: A hyperplane is convex.A halfspace is convex.Intersection of convex sets is convex.

• αx + (1− α)y, where α ∈ [0, 1] is calledconvex combination of x,y.

• αx + (1− α)y, for α ∈ (−∞,∞) is calledlinear combination of x,y.


Extreme Points

. .

...

Extreme PointsAll boundary pointsextreme

• A point x is extreme point of S if it is nota strict convex combination of two otherpoints of S.

• Corners of a polyhedron are extreme.All boundary points of a sphere areextreme.

• Let S = {x : Ax = b,x ≥ 0}, where A ism× n matrix of rank m ≤ n.

• A point x is an extreme point of S iff x isthe intersection of n linearly independenthyperplanes.


Basic Feasible Solution

• LP Ax = b. Assume rank(A) = m ≤ n.

• Partition cols as A = (B : N), whereB is m×m basis matrixN is m× (n−m) non-basis matrix.

• The LP system becomes

BxB + NxN = b

• Because B is non-singular, it is invertible:

xB = B−1b−B−1NxN

• Set xN = 0. We get xB = B−1b.

• xB is vector of basic variables,xN vector of non-basic vectors.

• If xB ≥ 0, then the solution x =(B−1b

0

)is

called a basic feasible solution.


Example of BFS

• Consider linear system:

2x1 + x2 + x3 + x4 = 15

x1 + 3x2 + x3 − x5 = 12

• A =(

2 1 1 1 01 3 1 0 −1

), b =

(1512

).

• Consider basis matrix of 1st and 3rd cols:

B =(

2 11 1

).

• Inverse matrix is B−1 =(

1 −1−1 2

).

• BFS is x =(

xB

xN

), where xN = 0, and

xB =(

x1

x3

)= B−1b

=(

1 −1−1 2

)(1512

)=

(39

)


Fundamental Theorem

Theorem: x is an extreme point ofS = {x : Ax = b,x ≥ 0} iff x is a BFS.

x1 = 0

x2=0

x3=0

x4=0

x5=0

Linear System

x1 − 2*x2 + x3 = 42*x1 + x2 + x4 = 18x2 + x5 = 10x1, x2, x3, x4, x5 >= 0

.

.

. .

x1

x2

(4,10)(0,10)

(8,2)

(4,0)(0,0)

• Each hyperplane ↔ variable, which is 0 onthe hyperplane.

• Each corner has 2 variables zero, and 3non-zero.


Simplex Example

• Consider LP max z = 2x1 + 3x2

• subject to

x1 − 2x2 + x3 = 4

2x1 + x2 + x4 = 18

x2 + x5 = 10

x1, x2, x3, x4, x5 ≥ 0

• Initial basic feasible solution:(x3, x4, x5) = (4, 18, 10), and (x1, x2) = (0, 0).

• Objective function value is zero.

• Entering variable. Pick a xi with ci > 0(max) in the objective, and make it basic.(E.g. x2)

• Departing variable. Increase x1 from zero,until some basic variable becomes zero(and non-basic).

• Repeat until all vars in objective functionhave negative coefficients.


Simplex Example

• Consider LP max z = 2x1 + 3x2

• subject to

x1 − 2x2 + x3 = 4

2x1 + x2 + x4 = 18

x2 + x5 = 10

x1, x2, x3, x4, x5 ≥ 0

• Initial BFS form:

x3 = 4− x1 + 2x2

x4 = 18− 2x1 − x2

x5 = 10− x2

• Set x1, x2 = 0, x3 = 4, x4 = 18, x5 = 10. z = 0.

• Entering variable is x2.

• Max value of x2 is min{∞, 18, 10}• Departing variable is x5.


Simplex Example

• Re-arrange the linear system:

x2 = 10− x5

x3 = 24− x1 − 2x5

x4 = 8− 2x1 + x5

• Set x1 = x5 = 0, and x2 = 10, x3 = 24, x4 = 8.

• Obj. function becomesz = 30 + 2x1 − 3x5 = 30.

• Entering variable is x1.

• Max value of x1 is min{∞, 24, 4}.• Departing variable is x4.


Simplex Example

• Re-arrange the linear system:

x1 = 4− 12x4 +

12x5

x2 = 10− x5

x3 = 20 +12x4 − 5

2x2

• Set x4 = x5 = 0, and x1 = 4, x2 = 10, x3 = 20.

• Obj. function becomesz = 38− x4 − 2x5 = 38.

• This is optimal because all coeffs arenegative.


Complexity of LP

• Simplex invented by G. Dantzig in 1947.

• One of the most popular and widely usedoptimization method. Many commercialsoftware packages.

• Extremely fast in practice, solves LP with105 constraints and variables in secs ormins.

• Worst-case complexity exponential!.Klee-Minty (1972)

• Poly-time LP algorithms in 1980’s.Khachian (1979). Much worse in practice!

• Interior point method of Karmakar (1984)first method to be theoretically in P andcomparable in practice.

• Open Problem: Is there a pivoting rulefor which simplex is polynomial?


LP Duality

• Duality turns out to be a deep property oflinear programming.

• Many famous theorems in other areas arean easy consequence of LP Duality.

• In order to motivate its power, weconsider an investment game.

• Investment Game played during Mon-Friperiod.

• Rules: If invest x one morning, and 2xnext morning, then receive 4x on thirdmorning.

• Amount received on day D can beinvested same day for next round.

• Inability to pony up 2x on second dayforfeits original x.

• Determine optimal investment strategy.


Strategy 1

• Starting fortune is $1.

• Invest $1/3 on Mon, and $2/3 on Tues.

• You get $4/3 on Wed, of which invest$4/9.

• Invest remaining $8/9 on Thurs.

• Collect $16/9 on Friday.

• Total gain is $7/9.

• Can we do better?


Strategy 2

Mon Tues Wed Thurs Fri

1/4 1/2

1/4

(1)

1/2

1/2(1)

(2)1

• Total gain is $1.

• Can we do still better?

• This is a maximization LP, and we wantan upper bound on the solution.

• LP duality aims to answer this question.


Illustrating Example

• Consider LP max z = 4x1 + x2 + 5x3 + 3x4

• subject to

x1 − x2 − x3 + 3x4 ≤ 1

5x1 + x2 + 3x3 + 8x4 ≤ 55

−x1 + 2x2 + 3x3 − 5x4 ≤ 3

xi ≥ 0

• Suppose instead of computing optimal z∗,we just want a quick estimate of the maxpossible (upper bound).

• We can get lower bounds by tryingfeasible solutions:

(0, 0, 1, 0) → z∗ = 5

(2, 1, 1, 1/3) → z∗ = 15

(3, 0, 2, 0) → z∗ = 22

• However, this method still gives noestimate of an upper bound.


Illustrating Example

• LP max z = 4x1 + x2 + 5x3 + 3x4

• subject to

x1 − x2 − x3 + 3x4 ≤ 1

5x1 + x2 + 3x3 + 8x4 ≤ 55

−x1 + 2x2 + 3x3 − 5x4 ≤ 3

xi ≥ 0

• Multiply 2nd constraint by 5/3:

253

x1 +53x2 +

153

x3 +403

x4 ≤ 2753

• This implies z∗ ≤ 2753 , because term by

term this constraint dominates obj.function z.

• One more. Add constraints (2) and (3),

4x1 + 3x2 + 6x3 + 3x4 ≤ 58

• This implies z∗ ≤ 58.


General Strategy

• Form linear combinations of constraints.

• Multiply constraint (i) by yi, then addy1(x1 − x2 − x3 + 3x4)+y2(5x1 + x2 + 3x3 + 8x4)+y3(−x1 + 2x2 + 3x3 − 5x4) ≤ y1 + 55y2 + 3y3.

• Rewrite(y1 + 5y2 − y3)x1+ (−y1 + y2 + 2y3)x2+(−y1 + 3y2 + 3y3)x3+ (3y1 + 8y2 − 5y3)x4 ≤y1 + 55y2 + 3y3.

• For this to dominate z term by termy1 + 5y2 − y3 ≥ 4

−y1 + y2 + 2y3 ≥ 1

−y1 + 3y2 + 3y3 ≥ 5

3y1 + 8y2 − 5y3 ≥ 3

yi ≥ 0

• We then have z ≤ y1 + 55y2 + 3y3.


Dual LP

• Our goal is to find smallest z subject toconstraints on yi.

• We have a linear program:

• minimize u = y1 + 55y2 + 3y3

• subject toy1 + 5y2 − y3 ≥ 4

−y1 + y2 + 2y3 ≥ 1

−y1 + 3y2 + 3y3 ≥ 5

3y1 + 8y2 − 5y3 ≥ 3

yi ≥ 0

• This LP is called the Dual of the originalLP.


Duality Theorems

Weak Duality: If z and u are feasiblesolutions of the primal and dual LPs, then

z ≤ u

• Follows by definition of dual LP. Eachfeasible solution of dual is an upper boundon primal.

Strong Duality: If primal LP has finiteoptimal solution z∗, then the dual also has afinite optimal solution u∗ and

z∗ = u∗


Back to Investment Game

• Variables xi to denote investmentamounts.

Mon Tues Wed Thur Frix1 2x1 (4x1)

x2 2x2 (4x2)x3 2x3 (4x3)

• The initial amount is $1.

• The final amount is 1 + x1 + x2 + x3.

• Formulate as LP: max 1 + x1 + x2 + x3

1− x1 ≥ 0

1− 3x1 − x2 ≥ 0

1 + x1 − 3x2 − x3 ≥ 0

1 + x1 + x2 − 3x3 ≥ 0

xi ≥ 0


Back to Investment Game

• Investment game LP: max 1 + x1 + x2 + x3

• subject to

1− x1 ≥ 0

1− 3x1 − x2 ≥ 0

1 + x1 − 3x2 − x3 ≥ 0

1 + x1 + x2 − 3x3 ≥ 0

xi ≥ 0

• Linear combination 2 ∗ (2) + (3) + (4):4x1 + 4x2 + 4x3 ≤ 4, which implies

1 + (x1 + x2 + x3) ≤ 2

• Thus, strategy 2 is optimal.


Integer Programming

• maximize cx

• subject to Ax ≤ b

• x integer (only diff from LP)

• IP feasible space is discrete, and smallerthan LP feasible space, which iscontinuous.

... . . . ...

.

...

...

IP solLP sol


Integer Programming

• One might think that restricting variablesmakes problem easier to solve.

• In fact, IP is NP-Complete.

• Reduction from Boolean 3-SAT.

• Φ = (x1 ∨ x3 ∨ x4) ∧ (x2 ∨ x3 ∨ x4) ∧ · · ·• Boolean Satisfiability of Φ can be encoded

as an IP:

• minimize x1

• subject to

x1 + (1− x3) + x4 ≥ 1

(1− x2) + x3 + (1− x4) ≥ 1...

xi ∈ {0, 1}


Formulating IP Problems

• Knapsack Problem: n items, weights ai,capacity b.

• maximize∑n

i=1 cixi

• subject ton∑

i=1

ajxj ≤ b

xi binary

• Vertex Cover: G = (V, E)

• minimize∑n

i=1 xi

• subject to

xi + xj ≥ 1, for each edge (i, j) ∈ E

xi binary


Network Problems using IP

• Min Cost Network Flows.

• Given is a network G = (V, E), supply bi foreach node i, capacity cij for each edge, andlower and upper bounds on flow for eachedge.

• minimize∑

cijxij

• subject to∑

j:(i,j)∈E

xij −∑

j:(j,i)∈E

xji = bi

lij ≤ xij ≤ uij, for all (i, j) ∈ E

• No need to restrict xij to integers.



• Shortest Path: 1 unit flow from s to t.

• minimize∑

cijxij

• subject to

∑

j:(s,j)∈E

xsj = 1

∑

j:(j,t)∈E

xjt = −1

∑xij −

∑xji = 0 if i 6= s, t

xij ∈ {0, 1} for all (i, j) ∈ E

s



• Assignment Problem.

• minimize∑

cijxij

• subject to∑

j:(i,j)∈E

xij = 1 for all i ∈ X

∑

j:(j,i)∈E

xji = 1 for all i ∈ Y

xij ∈ {0, 1} for all (i, j) ∈ E

X Y


LP Relaxation of IP

• An IP formulation with integralityconstraint removed is called the LPrelaxation.

• If IP is max cx, with optimal solution zI,and zL is the optimal solution of relaxedLP, then

zI ≤ zL

• It follows because IP feasible space is asubset of LP feasible space.

• When can one guarantee that zL = zI?

• Specifically, shortest path and assignmentformulations require integer variables.But they have combinatorial poly-timealgorithms.


Total Unimodularity

• Consider an IP: max cx s.t. Ax ≤ b, andx integer.

• The constraint matrix A is called TotallyUnimodular (TUM) if det(B) is in{0, 1,−1} for all 2× 2 submatrices B of A.

• E.g. if each col of A has exactly one 1, one-1, and all other zero entries, then A isTUM.

• If A is the incidence matrix of a graph,then A is TUM.

• Thus, our network problems (shortestpath, maxflow, assignment) can be solvedthrough LP relaxation.

linear programming - ucsb computer science …suri/cs231/lp.pdf · subhash suri uc santa barbara...

Documents