Karmarkar AlgorithmAnup Das, Aissan Dalvandi, Narmada Sambaturu, Seung Min, and Le Tuan Anh

1

Contents• Overview

• Projective transformation

• Orthogonal projection

• Complexity analysis

• Transformation to Karmarkar’s canonical form

2

LP Solutions• Simplex• Dantzig 1947

• Ellipsoid• Kachian 1979

• Interior Point• Affine Method 1967• Log Barrier Method 1977• Projective Method 1984

• Narendra Karmarkar form AT&T Bell Labs3

Linear Programming

5

MSubject to

MSubject to

Karmarkar’s Canonical FormGeneral LP

• Minimum value of is 0

Karmarkar’s Algorithm• Step 1: Take an initial point • Step 2: While • 2.1 Transformation: such that is center of

• This gives us the LP problem in transformed space

• 2.2 Orthogonal projection and movement: Project to feasible region and move in the steepest descent direction. • This gives us the point

• 2.3 Inverse transform: • This gives us

• 2.4 Use as the start point of the next iteration • Set

6

Step 2.1• Transformation: such that is center of - Why?

7

𝑥 ′(𝑘)

𝑟

𝑥(𝑘)

Step 2.2• Projection and movement: Project to feasible region and

move in the steepest descent direction - Why?

8

𝑐 ′

−𝑐 ′

Projection of to feasible region

Big Picture

9

𝑥(𝑘)𝑥 ′(𝑘)

𝑥 ′(𝑘+1)

𝑥(𝑘+1 )

𝑥(𝑘+2 )

Karmarkar’s Algorithm• Step 1: Take an initial point • Step 2: While • 2.1 Transformation: such that is center of

• This gives us the LP problem in transformed space

• 2.2 Orthogonal projection and movement: Project to feasible region and move in the steepest descent direction. • This gives us the point

• 2.3 Inverse transform: • This gives us

• 2.4 Use as the start point of the next iteration • Set

11

Projective Transformation• Transformation: such that is center of

12

Transform

𝑥(𝑘) 𝑥 ′(𝑘)=𝑒𝑛

𝑥(𝑘)=[𝑥1(𝑘)

𝑥2(𝑘)

⋮𝑥𝑛

(𝑘) ] 𝑥 ′(𝑘)=[1𝑛1𝑛⋮1𝑛

]=𝑒𝑛

𝑒= [111…1 ]𝑇

Projective transformation• We define the transform such that

• We transform every other point with respect to point

Mathematically:• Defining D = diag • projective transformation:•

• inverse transformation•

13

Problem transformation:• projective transformation:•

• inverse transformation•

The new objective function is not a linear function.

Minimize

s.t.

MinimizeS.t.

Transform

14

Problem transformation:

MinimizeS.t.

Convert

15

MinimizeS.t.

Karmarkar’s Algorithm• Step 1: Take an initial point • Step 2: While • 2.1 Transformation: such that is center of

• This gives us the LP problem in transformed space

• 2.2 Orthogonal projection and movement: Project to feasible region and move in the steepest descent direction. • This gives us the point

• 2.3 Inverse transform: • This gives us

• 2.4 Use as the start point of the next iteration • Set

16

Orthogonal Projection

• Projecting onto • projection on • normalization

17

MinimizeS.t.

Orthogonal Projection• Given a plane and a point outside the plane, which direction

should we move from the point to guarantee the fastest descent towards the plane?• Answer: perpendicular direction

• Consider a plane and a vector

18

𝑐 ′ 𝑐𝑃

𝑠 𝐴’

ℵ (𝐴′)

Orthogonal Projection• Let and be the basis of the plane ()

• The plane is spanned by the column space of ]

• , so

• is perpendicular to and

• 19

𝑐 ′ 𝑐𝑃

𝑠 𝐴’

ℵ (𝐴′)

Orthogonal Projection

• = projection matrix with respect to ’

• We need to consider the vector • Remember

• is the projection matrix with respect to 20

𝑐 ′ 𝑐𝑃

𝑠 𝐴’

ℵ (𝐴′)

Orthogonal Projection• What is the direction of?

• Steepest descent =

• We got the direction of the movement or projected ’ onto

• Projection on

21

𝑐 ′ 𝑐𝑃

𝑠 𝐴’

Calculate step size

• r = radius of the incircle

22

MinimizeS.t.

Movement and inverse transformation

23

Transform

𝑥(𝑘) 𝑥 ′(𝑘)=𝑒𝑛

𝑥 ′(𝑘+ 1)

𝑥(𝑘+1 ) Inverse Transform

Big Picture

24

𝑥(𝑘)𝑥 ′(𝑘)

𝑥 ′(𝑘+1)

𝑥(𝑘+1 )

𝑥(𝑘+2 )

Matlab Demo

25

Contents• Overview

• Projective transformation

• Orthogonal projection

• Complexity analysis

• Transformation to Karmarkar’s canonical form

26

Running Time• Total complexity of iterative algorithm =

(# of iterations) x (operations in each iteration)

• We will prove that the # of iterations = O(nL)

• Operations in each iteration = O(n2.5)

• Therefore running time of Karmarkar’s algorithm = O(n3.5L)

# of iterations• Let us calculate the change in objective function value at the

end of each iteration

• Objective function changes upon transformation• Therefore use Karmarkar’s potential function

• The potential function is invariant on transformations (upto some additive constants)

# of iterations• Improvement in potential function is

• Rearranging,

# of iterations• Since potential function is invariant on transformations, we

can use

whereandare points in the transformed space.

• At each step in the transformed space,

# of iterations• Simplifying, we get•

Where

• For small

• So suppose both and are small -

# of iterations• It can be shown that • The worst case for

would be if

• In this case

# of iterations• Thus we have proved that

• Thus

where k is the number of iterations to converge

• Plugging in the definition of potential function,

# of iterations• Rearranging, we get

# of iterations• Therefore there exists some constant such that after

iterations,

where , being the number of bits

is sufficiently small to cause the algorithm to converge.

• Thus the number of iterations is in

Rank-one modification• The computational effort is dominated by:

• Substituting A’ in, we have

• The only quantity changes from step to step is D • Intuition:• Let and be diagonal matrices• If and are close the calculation given should be cheap!

Method• Define a diagonal matrix , a “working approximation” to in

step k such that• for • We update D’ by the following strategy• (We will explain this scaling factor in later slides)• For to do

if Then let and update

Rank-one modification (cont)• We have

• Given , computation of can be done in or steps.• If and differ only in entry then

• => rank one modification.• If and differ in entries then complexity is .

Performance Analysis• In each step: • Substituting , and , • Let call • Then Let • Then

Performance analysis - 2• First we scale by a factor

• Then for any entry I such that

• We reset • Define discrepancy between and as• Then just after an updating operation for index

Performance Analysis - 3• Just before an updating operation for index i

Between two successive updates, say at steps k1 and k2

Let call Then

Assume that no updating operation was performed for index I-

Performance Analysis - 4

• Let be the number of updating operations corresponding to index in steps and be the total number of updating operations in steps.

• We have

Because of then

Performance Analysis - 5• Hence

• So

• Finally

Contents• Overview

• Transformation to Karmarkar’s canonical form

• Projective transformation

• Orthogonal projection

• Complexity analysis

44

Transform to canonical form

45

MSubject to

• x

MSubject to

• x

Karmarkar’s Canonical FormGeneral LP

Step 1:Convert LP to a feasibility problem• Combine primal and dual problems

• LP becomes a feasibility problem46

MSubject to

• x

MSubject to

•

Primal Dual

Combined

Step 2: Convert inequality to equality• Introduce slack and surplus variable

47

Step 3: Convert feasibility problem to LP• Introduce an artificial variable to create an interior starting

point.• Let be strictly interior points in the positive orthant.

48

• Minimize • Subject to• • + •

Step 3: Convert feasibility problem to LP

• Change of notation

49

MSubject to

• x

• Minimize • Subject to• • + •

Step4: Introduce constraint • Let ,…, ] be a feasible point in the original LP

• Define a transformation• for

• Define the inverse transformation• for

50

Step4: Introduce constraint • Let denote the column of • If • Then

• We define the columns of a matrix ’ by these equations• for • =

• Then

51

Step 5: Get the minimum value of Canonical form • Substituting

• We get

• Define by

• Then implies

52

Step 5: Get the minimum value of Canonical form • The transformed problem is

53

M

Subject to

References• Narendra Karmarkar (1984). "

A New Polynomial Time Algorithm for Linear Programming”• Strang, Gilbert (1 June 1987). "Karmarkar’s algorithm and its

place in applied mathematics". The Mathematical Intelligencer (New York: Springer) 9 (2): 4–10. doi:10.1007/BF03025891. ISSN 0343-6993. MR '''883185'‘

• Robert J. Vanderbei; Meketon, Marc, Freedman, Barry (1986). "A Modification of Karmarkar's Linear Programming Algorithm". Algorithmica 1: 395–407. doi:10.1007/BF01840454. ^ Kolata, Gina (1989-03-12). "IDEAS & TRENDS; Mathematicians Are Troubled by Claims on Their Recipes". The New York Times. 54

References• Gill, Philip E.; Murray, Walter, Saunders, Michael A., Tomlin, J.

A. and Wright, Margaret H. (1986). "On projected Newton barrier methods for linear programming and an equivalence to Karmarkar’s projective method". Mathematical Programming 36 (2): 183–209. doi:10.1007/BF02592025.

• oi:10.1007/BF02592025. ^ Anthony V. Fiacco; Garth P. McCormick (1968). Nonlinear Programming: Sequential Unconstrained Minimization Techniques. New York: Wiley. ISBN 978-0-471-25810-0. Reprinted by SIAM in 1990 as ISBN 978-0-89871-254-4.

• Andrew Chin (2009). "On Abstraction and Equivalence in Software Patent Doctrine: A Response to Bessen, Meurer and Klemens". Journal Of Intellectual Property Law 16: 214–223.

55

Q&A

56