the ellipsoid algorithmeaton.math.rpi.edu/faculty/mitchell/courses/matp... · introduction the...

The Ellipsoid Algorithm

John E. Mitchell

Department of Mathematical SciencesRPI, Troy, NY 12180 USA

February 2018

Mitchell The Ellipsoid Algorithm 1 / 28

9

Introduction

Outline

1 Introduction

2 Assumptions

3 Each iteration

4 Complexity analysis

5 Separation and optimization

6 Relevance to integer optimization

7 NP-Hard problems


Introduction

Introduction

The simplex algorithm requires exponentially many iterations in theworst case for known pivot rules.

In contrast, the ellipsoid algorithm requires a polynomial number ofiterations.

The ellipsoid algorithm was developed in the 1970s in the SovietUnion, building on earlier Soviet work on nonlinear programming.


Introduction

New York Times, November 7, 1979


Introduction


Published: November 7, 1979Copyright © The New York Times


Introduction


Published: November 7, 1979Copyright © The New York TimesMitchell The Ellipsoid Algorithm 6 / 28

Introduction

Find a feasible solution

The algorithm is a high-dimensional version of binary search.

It can be stated as an algorithm to determine whether a feasiblesolution exists to a system of inequalities:

Does there exist x 2 IRn satisfying aT

ix ci for i = 1, . . . ,m?

We letP := {x 2 IR

n : aT

i x ci for i = 1, . . . ,m}.


Assumptions

Outline

1 Introduction

2 Assumptions

3 Each iteration




7 NP-Hard problems


Assumptions

Assumptions

We need to make two assumptions:Assumption 1: If P is nonempty then it includes points

contained in a ball of radius R centered at the origin.

Assumption 1 can be justified through the use of Cramer’s Rule, whichlets us bound the norm of any extreme point of P.


1 % 4

Assumptions

Assumption 2

Assumption 2: If P is nonempty then it contains a ball of

radius r .

Assumption 2 is like a nondegeneracy assumption.

Cramer’s Rule also lets us place a lower bound on the distancebetween two extreme points.

Technically, we can always make Assumption 2, because the size ofthe numbers in the parameters can be used to define anotherparameter � and then each ci can be increased by � in such a way that

the new system has a solution if and only if the old system has asolution, andthe new system contains a ball of radius r if it is nonempty.


radius r X p

Assumptions

Assumption 2

Assumption 2: If P is nonempty then it contains a ball of

radius r .

Assumption 2 is like a nondegeneracy assumption.

Cramer’s Rule also lets us place a lower bound on the distancebetween two extreme points.

Technically, we can always make Assumption 2, because the size ofthe numbers in the parameters can be used to define anotherparameter � and then each ci can be increased by � in such a way that

the new system has a solution if and only if the old system has asolution, andthe new system contains a ball of radius r if it is nonempty.


Each iteration

Outline

1 Introduction

2 Assumptions

3 Each iteration




7 NP-Hard problems


Each iteration

Each iterationAt each iteration we have an ellipsoid which contains (part of) P, if P isnonempty.

We check the center x̄ of the ellipsoid: does it satisfy

aT

i x̄ ci for i = 1, . . . ,m?

If YES: we are done.

Otherwise, we have a violated constraint aT

ix cj .

We define a new ellipsoid that contains the intersection of the oldellipsoid with the halfspace {x 2 IRn : aT

ix aT

ix̄}.

If n = 1 then this would divide our search space in half.

In IRn, we don’t get such a good improvement. Nonetheless, the size ofthe ellipsoid shrinks enough that we get convergence in a number ofiterations polynomial in n.


Each iteration

Illustration: trying to find a point in P

Ek

xk

P


Each iteration


Ek

xk

aT

ix = aT

ixk

P


Each iteration


Ek

xk

aT

ix = aT

ixk

P

Ek+1

xk+1


E "={ xER" :( x-x "TY!(x-xMeI }A

positivedefinite,

symmetricVolume:proportional

t odeft(Mu)

Each iteration


P

Ek+1

xk+1


Each iteration

Complexity analysis

Initial ellipsoid: Ball of radius R centered at the origin.

How many “Noes” are enough?

If the ellipsoid is too small to contain a ball of radius r then P = ;.The algorithm updates from an ellipsoid E to an ellipsoid E 0.It can be shown that the volumes of these ellipsoids are related as:

Vol(E 0) e�1/(2(n+1))Vol(E).


Complexity analysis

Outline

1 Introduction

2 Assumptions

3 Each iteration




7 NP-Hard problems


Complexity analysis

How many iterations?

How many “Noes” to go from a ball of radius R to a ball of radius r?Let v and V denote the volumes of balls of radius r and R,respectively. Let E0 be the initial ellipsoid, let Ek be the ellipsoid afterk iterations. We want the smallest value of k so that

Vol(Ek ) v .

We also have

Vol(Ek ) he�1/(2(n+1))

ik

V = e�k/(2(n+1))

V .

Thus it suffices to choose k sufficiently large that

e�k/(2(n+1))

V v .


Complexity analysis

How many iterations, part 2

It suffices to choose k sufficiently large that

e�k/(2(n+1))

V v .

Equivalently, after dividing by V and taking logs of both sides, werequire

�k/(2(n + 1)) ln⇣

v

V

⌘,

which can be restated as requiring

k � 2(n + 1) ln✓

V

v

◆.

The ratio V/v can be chosen so that ln(V/v) is O(n3), so we have analgorithm that requires O(n4) iterations.


Complexity analysis

Work per iteration

In exact arithmetic, would need to work with square roots.

Can show that it is OK to approximate the square roots and still havethe volume of the ellipsoid decrease at almost the same rate.


Complexity analysis

Finding optimal solutions to linear programs

If feasible, cut on the objective function.

So restrict attention to the halfspace of points at least as good as thecurrent center.


Separation and optimization

Outline

1 Introduction

2 Assumptions

3 Each iteration




7 NP-Hard problems



Convex feasibility problems

DefinitionGiven a convex set C ✓ IRn and a point x̄ 2 IRn, the separation problemis to determine whether x̄ 2 C. If the answer is NO, a separatinghyperplane aT x = c is determined, where a 2 IRn, c 2 IR, aT x c for

all x 2 C, and aT x̄ > c.


§• I

a t >a

↳ = Ca Tx< c


Equivalence of separation and feasibilty

TheoremAssume the convex set C contains a ball of radius r . If the separation

problem for C can be solved in polynomial time then the feasibility

problem for C can also be solved in polynomial time.

Proof.We use the ellipsoid algorithm. At each iteration, we test whether thecenter xk of the current ellipsoid is in C. If it is not then the separationproblem solution returns a valid constraint for C which we can use toupdate the ellipsoid.

The overall complexity is the product of two polynomial functions,which is polynomial.


⇐"⇐÷€€÷i


Equivalence of separation and optimizationConsider the optimization problem

minx2IRn cT x

subject to x 2 C ✓ IRn (CP)

where C is a convex set and c 2 IRn.

TheoremThe problem (CP) can be solved in polynomial time if and only if the

separation problem for C can be solved in polynomial time.

This result follows from using the ellipsoid algorithm.We can optimize over C using the ellipsoid algorithm, generatingseparating hyperplanes either:

by solving the separation problem to separate the current centerof the ellipsoid from C, orby cutting on the objective function if x 2 C.


÷±÷€€÷÷.

Ex<<7×4

Relevance to integer optimization

Outline

1 Introduction

2 Assumptions

3 Each iteration




7 NP-Hard problems



Integer optimization as a linear programConsider the integer optimization problem

minx cT x

subject to x 2 S (IP)x 2 Zn ✓ IRn

for some subset S, and where Z denotes the set of integers. Let C

denote the convex hull of the set of integer points in S, so

C = conv(S \ Zn)

(The convex hull of a set T is the smallest convex set containing T . Aformal definition will be given next week.)

All the extreme points of C are feasible integer points, so we could inprinciple solve (IP) by solving the linear program

minx cT x

subject to x 2 C ✓ IRn (LP)


.

÷÷%t¥I÷÷÷.


“Solving” the linear programThe difficulty with this approach is that it is very hard to get an explicitformulation for C.

If we can solve the separation problem for C in polynomial time thenwe can solve (LP) in polynomial time, which means we can solve (IP)in polynomial time.

Thus, for NP-complete problems, we cannot expect to solve theseparation problem in polynomial time.

Conversely, if we can solve the optimization problem in polynomial timethen we can solve the separation problem in polynomial time.

For example, we can solve the problem of finding a minimum weightperfect matching on a graph G = (V ,E) in polynomial time, and indeedthe separation problem of determining whether a point x is in theconvex hull of the set of perfect matchings can be solved in O(|V |4)time.


NP-Hard problems

Outline

1 Introduction

2 Assumptions

3 Each iteration




7 NP-Hard problems


NP-Hard problems

NP-hard problems

A problem is called NP-hard if it at least as hard as an NP-completeproblem, in the sense that the NP-complete problem can bepolynomially reduced to the NP-hard problem.

An NP-hard problem need not be a feasibility problem.

In particular, optimization versions of NP-complete integerprogramming feasibility problems are NP-hard.


the ellipsoid algorithmeaton.math.rpi.edu/faculty/mitchell/courses/matp... · introduction the...

Documents