column generation by soumitra pal under the guidance of prof. a. g. ranade

Column Generation

By

Soumitra Pal

Under the guidance of

Prof. A. G. Ranade

Agenda

• Introduction

• Basics of Simplex algorithm

• Formulations for the CSP

• (Delayed) Column Generation

• Branch-and-price

• Flow formulation of CSP & solution

• Conclusions

Cutting Stock Problem

• Given larger raw paper rolls

• Get final rolls of smaller widths

10

5 3

Cutting Stock Problem (2)

• Raw width (W) = 10• No of finals given below

i Width (wi) Quantity (bi)

1 3 9

2 5 79

3 6 90

4 9 27

• Minimize total no of raws reqd. to be cut

Agenda

• Introduction

• Basics of Simplex algorithm

• Formulations for the CSP

• (Delayed) Column Generation

• Branch-and-price

• Flow formulation of CSP & solution

• Conclusions

5x1 - 8x2 ≥ -80 -5x1 - 4x2 ≥ -100

-5x1 - 2x2 ≥ -80-5x1 + 2x2 ≥ -50

Min -x1 -x2

Objective/

Cost fn

Constraints

(10,0)

(13,5)

(12,10)

(8,15)

(0,10)

5x1 - 8x2 ≥ -80 -5x1 - 4x2 ≥ -100-5x1 - 2x2 ≥ -80-5x1 + 2x2 ≥ -50

-x1-x2 = -10 -x1-x2 = -20 -x1-x2 = -30

Cost decreases in this direction

(10,0)

(13,5)

(12,10)

(8,15)

(0,10)

Simplex method

contraints of no.

variablesof no.

0

:s.t.

Min

1211

22222121

11212111

2211

m

l

x

bxaxaxa

bxaxaxa

bxaxaxa

xcxcxc

i

mlmlmm

ln

ln

ln

0

:s.t.

000Min

1211

222222121

111212111

212211

i

mnlmlmm

lln

lln

nllln

x

bxxaxaxa

bxxaxaxa

bxxaxaxa

xxxxcxcxc

0

:s.t.

Min

x

bAx

cx

(10,0,130,50,30,0)

(13,5,110,10,0,0)

(12,10,60,0,0,10)

(8,15,0,0,10,40)

(0,10,0, 60,60,70)

(0,0,80, 100,80,50)

Basic & non-basic 5x1 - 8x2 ≥ -80 -5x1 - 4x2 ≥ -100-5x1 - 2x2 ≥ -80-5x1 + 2x2 ≥ -50

'1'

'11'

1'1'

'

''

11

1

1

equation, previousin it Putting

variablebasic-non one increase corner,next out find To0

0 ,0

0min

NBB

NB

NB

N

B

B

BBB

B

B

BB

BNB

NxBxxNxBbBxbBNxBx

x

xx

bBx

NBI

bBcxccxbBxbBx

xbx

NBAx

xcccx

NBccc

xNBcccxcx

xNBccxcxcxccxxcNxBxcxcxccx

BN

NBN

NBNBBNNBB

NNNBBNNBB

1

'1'

'1'''

''1'''

cost reduced called isbracket in termThe

)(costin Change

)()(

cost New

• If all elements of it are non-negative, it is already optimal• Otherwise choose the non-basic variable corresponding to

most negative value to enter the basic

• How to find out outgoing basic variable?

ofcomponent kth

)( ofcomponent kth of Min value

0. becomes ofcomponent )k(say one until increased becan It .

variablebasic new with mutilpies that ofcolumn thebe Let

1

'th

1

1'1'

d

xbB

xx

NBd

bBNxBx

B

B

i

NB

Few steps of simplex algorithm

• Compute reduced cost• If all >= 0, optimal stop.• Otherwise choose the most negative

component• Corresponding variable enters basis• Compute the ratios for finding out leaving basis• Update basic variables and B for next step

NBcc BN1

'1'NBB NxBxx

Smart way of doing

variableleavingabout info gives

gives, ofcolumn theis wherecost reduced and gives

gives

d

xdNaaBd

yNcycyBxbBx

B

NB

BB

yd

Start

yd

Reduced cost vector

yd

Selection of non-basic variable

yd

y vector

yd

Selection of leaving basic variable

Preparation for next step

yd

yd

Entering variable in 2nd iteration

yd

Leaving variable in 2nd iteration

yd

Preparation for 3rd iteration

yd

Final iteration

Done so far

• Basics of (revised) Simplex algorithm– Formulation– Corners, basic variables, non-basic variables– How to move from corner to corners– Optimal value

Agenda

• Introduction

• Basics of Simplex algorithm

• Formulations for the CSP

• (Delayed) Column Generation

• Branch-and-price

• Flow formulation of CSP & solution

• Conclusions

raw k fromcut widthi of finals of no.

otherwise 0used, is raw k if 1

widthafor index iraw afor index k

widthsdifferent of no.mraws available of no.K

,,1,,1integer and 0,,11or 0

,,1

,,1 :s.t.

Min

th

th

th

1

1

1

ik

k

ik

k

k

m

iiki

i

K

kik

K

kk

x

y

miKkxKky

KkWyxw

mibx

y

First step to solution - Formulation

Kantorovich formulation

Kantorovich formulation example

1 2 3 4

1 ,4

3 ,1 ,10 ,1 ,4

21

132111

4231

k

kk

k xx

xxxyyyyK

LP relaxation• Integer programming is hard• Convert it to a linear program

raw k fromcut widthi of finals of no.

otherwise 0used, is raw k if 1

widthafor index iraw afor index k

widthsdifferent of no.mraws available of no.K

,,1,,1integer and 0,,11or 0

,,1

,,1 :s.t.

Min

th

th

th

1

1

1

ik

k

ik

k

k

m

iiki

i

K

kik

K

kk

x

y

miKkxKky

KkWyxw

mibx

y

0 ≤ yk ≤ 1

LP relaxation (2)

• LP relaxation is poor• For our example, optimal LP objective is 120.5• The optimal integer objective solution value is

157• Gap is 36.5• It can be as bad as ½ of the integer solution

• Can we get a better LP relaxation?

jpattern fromcut widthi of finals of no.

jpattern in cut raws of no.

patterns possible all ofset J

pattern cutting aj

widthsdifferent of no.m

widthafor index i

integer and 0

,,1 :s.t.

Min

th

ij

j

j

iJj

jij

Jjj

a

x

Jjx

mibxa

x

Another Formulation

Gilmore-Gomory

Formulation

27

90

79

9

000000001

000000110

000121000

321100100

:9

:6

:5

:3

9

8

7

6

5

4

3

2

1

4

3

2

1

x

x

x

x

x

x

x

x

x

w

w

w

w

Example formulation

3*1 + 5*0 + 6*1 + 9*0 = 9 ≤ 10

LP relaxation of Gilmore-Gomory

• LP relaxation is better

• For our example, 156.7

• Integer objective solution, 157

• Gap is 0.3

• Conjecture: Gap is less than 2 for practical cutting stock problems

Issues

• The number of columns are exponential

• Optimal value is still not integer

• Gilmore-Gomory proposed an ingenuous way of working with less columns

• Branch-and-price to solve the other problem

Done so far

• Basics of (revised) Simplex algorithm– Formulation– Corners, basic variables, non-basic variables– How to move from corner to corners– Optimal value

• Formulation of the CSP– LP relaxation– Bounds

Agenda

• Introduction

• Basics of Simplex algorithm

• Formulations for the CSP

• (Delayed) Column Generation

• Branch-and-price

• Flow formulation of CSP & solution

• Conclusions

Column Generation

• In pricing step of simplex algorithm one basic variable leaves and one non-basic variable enters the basis

• This decision is made from the reduced cost vector

• The component with most negative value determines the entering variable

yNcNBcc NBN or 1

Column Generation (2)

• In terms of columns, we need to find

• That is equivalent to finding a such that

• For cutting stock problem, cj=1, hence it is equivalent to

• With implicit constraint

• Pricing sub-problem

} ofcolumn a is |{min argj

Nayac jjj

} ofcolumn a is |)(min{ Aayaac

}1|max{ yaya

Wwa

Column Generation (3)

AnyNew Columns?

STOP(LP Optimal)

SolveRestricted Master Problem

(RMP)

SolvePricing Problem

Update RMP withNew Columns

No

Yes

27

90

79

9

000000001

000000110

000121000

321100100

:9

:6

:5

:3

9

8

7

6

5

4

3

2

1

4

3

2

1

x

x

x

x

x

x

x

x

x

w

w

w

w

Example formulation

T

BB

a

aaaa

aaaa

yyB

xBC

0101findcan weg,programmin dynamicor n enumeratio Using

109653constraintknapsack Given

1.1.12

1

3

1 maximize toneed we

112/13/11111

27

90

5.39

3

1/27

1/90

2/79

3/9

,

1000

0100

0020

0003

,

27

90

79

9

4321

4321

27

81

5.39

9

27

1.990

5.39

9

27

.90

5.39 ,

1000

0101

0020

0001

replaced is Bin column first ,9

*90*9*1/90*3

1/3 /

0103

1

3dt

t

xB

t

dx

daBd

B

TT

B

T

5.15627815.399 Solution

optimalalready is Hence1 subproblem ofsolution get not can We

109653costraintknapsack Given

1.1.12

1.0 maximize toneed we

112/101111

27

81

5.39

9

,

1000

0101

0020

0001

4321

4321

B

B

x

aaaa

aaaa

yyB

xB

Done so far

• Basics of (revised) Simplex algorithm– Formulation– Corners, basic variables, non-basic variables– How to move from corner to corners– Optimal value

• Formulation of the CSP– LP relaxation– Bounds

• Delayed column generation– Restricted master problem– Sub-problem to generate column– Repeated till optimal

Agenda

• Introduction

• Basics of Simplex algorithm

• Formulations for the CSP

• (Delayed) Column Generation

• Branch-and-price

• Flow formulation of CSP & solution

• Conclusions

Integer solution

• A formulation of the problem which gives ‘good’ LP relaxation

• Solved the LP relaxation using column generation

• But, the LP relaxation is not the ultimate goal, we need integer solution

• In the example x2 is 39.5• One way is to rounding

• What to do if there are multiple fractional variables?

• The method commonly followed is branch -and-bound technique of solving integer programs

• Basic fundae– create multiple sub-problems with extra

constraints

– solve the sub-problems using column generation

– take the best of solutions to the sub problems

• Key is the rules of dividing into sub problems (this is called branching strategy)

• Things to look into– Branching rule does not destroy column

generation sub problem property– Efficiency of the process

• Ingenuity of the formulation and branching strategy

• Use of running bounds to prune (fathomed)

• This technique is called branch-and-price

• We see another formulation next

Done so far• Basics of (revised) Simplex algorithm

– Formulation– Corners, basic variables, non-basic variables– How to move from corner to corners– Optimal value

• Formulation of the CSP– LP relaxation– Bounds

• Delayed column generation– Restricted master problem– Sub-problem to generate column– Repeated till optimal

• Branch-and-price– Basic fundae

Agenda

• Introduction

• Basics of Simplex algorithm

• Formulations for the CSP

• (Delayed) Column Generation

• Branch-and-price

• Flow formulation of CSP & solution

• Conclusions

Flow formulation

• Bin packing problem

• Similar to the cutting stock problem

bin items

w2 w2 w2 w2

w1w1 w1

loss loss loss loss loss

01 2 3 4 5

w2 w2

01 2 3 4 5

loss

Example• Bin capacity W = 5

• Item sizes – w1=3 and w2=2

Formulation equations

• Problem is equivalent to finding minimum flow between nodes 0 & W

0integer

0integer

,,2,1

,

1,,2,1,0

0,

:s.t.

Min

)()(

z

x

mdbxWjz

Wi

jz

xx

z

ij

Awkkdwkk

Ajkjk

Aijij

d

d

• Issue of symmetry in the graph

• 3 rules to reduce symmetry1. Arcs are specified according to decreasing

width

2. A bin can never start with a loss

3. Number of consecutive arcs corresponding to a single item size must not exceed number of orders

w2 w2 w2 w2

w1w1 w1

loss loss loss loss loss

01 2 3 4

5

Invalid as per rule 2

Invalid as per rule 1

w2 w2 w2

w1

loss loss loss

0 1 2 3 4 5

Branch-and-price

• The sub-problem finds an longest unit flow path where cost of each arc is given by y vector

• Branching heuristics– Xij >= ceil(xij)– Xij <= floor(xij)

Few other points

• Loss constraints from the LP relaxation value is equated with the sum of loss variables

• With the above constraint it can be shown that the solution will be always integral

• The algorithm uses this fact by incrementally adding the lower bound

• This is done finite no. of times• No of new constraints are added is finite• Algorithm runs in pseudo-polynomial time

Conclusions & Future work

• Column generation is a success story in large scale integer programming

• The flexibility to work with a few columns make real life problems tractable

• Ingenuous formulations and branching heuristics are used to improve efficiency

• Need to explore more applications with different heuristics

Acknowledgements

• Valerio de Carvalho for borrowing some of his explanations exactly

• http://www-fp.mcs.anl.gov/otc/Guide/CaseStudies/simplex/applet/SimplexTool.html for the java applet

• AMPL & CPLEX tools

• My guide

Thank you!