8/12/2019 MIT15_093J_F09_sdp

1/51

Introduction to Semidefinite Programming (SDP)

RobertM.Freund

1 Introduction

Semidefinite programming (SDP) isthemost exciting development inmath-ematicalprogramminginthe1990s.SDPhasapplicationsinsuchdiversefields astraditionalconvex constrained optimization, controltheory, andcombinatorial optimization. BecauseSDP is solvable via interior pointmethods,mostoftheseapplicationscanusuallybesolvedveryefficientlyinpracticeaswellasintheory.

2 Review of Linear Programming

Considerthelinearprogrammingprobleminstandardform:

LP : minimize c xs.t. ai x=bi, i= 1, . . . ,m

n+.x

Herexisavectorofnvariables,andwewritec xfortheinnerproduct

jn=1cjxj,etc.

Also,n+ n x0},andwecalln+thenonnegativeorthant.n

:={x |In fact, isaclosed convex cone, whereKiscalledaclosedaconvexcone+ifKsatisfiesthefollowingtwoconditions:

1

8/12/2019 MIT15_093J_F09_sdp

2/51

3

Ifx,wK,thenx +wKforallnonnegativescalarsand.Kisaclosedset.

Inwords,LP isthefollowingproblem:

Minimize the linear function c x, subject to the condition that xmust solvemgiven equations ai x=bi, i= 1, . . . , m, and that xmust lie in the closedconvex cone K= n .+

Wewillwritethestandardlinearprogrammingdualproblemas:

m

LD: maximize yibii=1

m

s.t. yiai +s=ci=1

n+.s

GivenafeasiblesolutionxofLP anda feasiblesolution (y, s)ofLD,the

duality gap issimplyc xmi=1yibi = (cmi=1yiai) x=s x0,because x0ands0.WeknowfromLPdualitytheorythatsolongasthepri-malproblemLP isfeasibleandhasboundedoptimalobjectivevalue,thenthe primalandthe dual bothattain theiroptimawithno duality gap.Thatis,thereexistsx and (y, s) feasiblefortheprimalanddual,respectively,

m suchthatc x i=1yi bi =s x = 0.

Facts about Matrices and the Semidefinite Cone

IfXisannnmatrix,thenXisa positivesemidefinite (psd)matrix ifnvTXv0 forany v .

2

8/12/2019 MIT15_093J_F09_sdp

3/51

IfXisannnmatrix,thenXisa positive definite (pd)matrix ifnvTXv>0 forany v , v= 0.

LetSn nmatrices, and letSn +thesetof positivesemidefinite (psd)nnsymmetricmatrices. SimilarlyletSn denote thesetof positive definite (pd) n

nsymmetricmatrices.++

LetXandY beanysymmetricmatrices.WewriteX0todenotethatX issymmetricandpositivesemidefinite,andwewriteXYtodenotethatXY 0.WewriteX0todenotethatX issymmetricandpositivedefinite,etc.

2Remark1 Sn+

denote the set of symmetricn denote

{X Sn | X 0} is a closed convex cone in n(n+ 1)/2. of=dimension n

nTo see whythis remark is true, suppose thatX,W S

+

,0.Foranyv n,wehave:vT(X+W)v=vTXv+vTWv0,

. Pickanyscalars

wherebyX n nW S This shows thatS+ +++forwardtoshowthatSn

Recallthefollowingpropertiesofsymmetricmatrices:

IfX Sn,thenX =QDQT forsomeorthonormalmatrixQandsomediagonalmatrixD. (RecallthatQisorthonormalmeansthatQ1=QT,andthatDisdiagonalmeansthattheoffdiagonalentriesofDareallzeros.)

3

isacone.Itisalsostraight-.isaclosedset.

8/12/2019 MIT15_093J_F09_sdp

4/51

4

If X = QDQT

asabove, then the columns ofQ forma set of n orthogonaleigenvectorsofX,whoseeigenvaluesarethecorrespondingdiagonalentriesofD.

X 0ifandonlyifX = QDQT wherethe eigenvalues (i.e., thediagonalentriesofD) areallnonnegative.

X 0ifandonlyifX = QDQT wherethe eigenvalues (i.e., thediagonalentriesofD) areallpositive.

IfX0andifXii = 0,thenXij =Xji = 0forallj = 1, . . . , n.ConsiderthematrixMdefinedasfollows:

P vM= T ,v d

whereP 0,v isavector,andd isascalar. ThenM 0ifandonlyifd vTP1v >0.Semidefinite Programming

LetXSn.WecanthinkofXasamatrix,orequivalently,asanarrayofn2 componentsofthe form (x11, . . . , xnn). Wecanalso justthinkofXasanobject (avector)in thespaceSn. AllthreedifferentequivalentwaysoflookingatXwillbeuseful.

WhatwillalinearfunctionofXlooklike? IfC(X) isalinearfunctionofX,thenC(X) canbewrittenasC X,where

n n

C X:= CijXij .i=1j=1

IfX isasymmetricmatrix,thereisnolossofgeneralityinassumingthat

4

8/12/2019 MIT15_093J_F09_sdp

5/51

thematrixC isalsosymmetric. Withthisnotation,wearenowreadytodefineasemidefinite program. Asemidefinite program (SDP)isanopti-mizationproblemoftheform:

SDP : minimize C Xs.t. Ai X=bi , i= 1, ... , m,

X0,

Notice that inanSDP thatthevariable isthematrixX, butitmightbehelpfultothinkofXasanarrayofn2 numbersorsimplyasavectorinSn.TheobjectivefunctionisthelinearfunctionC XandtherearemlinearequationsthatXmustsatisfy,namelyAi X=bi , i= 1, . . . , m.ThevariableXalsomust lie in the (closedconvex)coneof positivesemidef-initesymmetricmatricesSn NotethatthedataforSDP consistsofthe+.symmetricmatrixC(whichisthedatafortheobjectivefunction) andthemsymmetricmatricesA1, . . . , Am,andthemvectorb,whichformthemlinearequations.

LetusseeanexampleofanSDP forn=3andm=2. Definethefollowingmatrices:

1 0 1 0 2 8 1 2 3

A1=0 3 7, A2=2 6 0, and C=2 9 0,1 7 5 8 0 4 3 0 7

andb1= 11andb2= 19.ThenthevariableXwillbethe3

3symmetric

matrix:

5

8/12/2019 MIT15_093J_F09_sdp

6/51

x11 x12 x13 X= x21 x22 x23 ,x31 x32 x33

andso,forexample,

C X = x11+ 2x12+ 3x13+ 2x21+ 9x22+ 0x23+ 3x31+ 0x32+ 7x33= x11+ 4x12+ 6x13+ 9x22+ 0x23+ 7x33.

since,inparticular,X issymmetric. ThereforetheSDP canbewrittenas:

SDP: minimize x11+ 4x12+ 6x13+ 9x22+ 0x23+ 7x33

s.t.x11+ 0x12+ 2x13+ 3x22+ 14x23+ 5x33 = 11

0x11+ 4x12+ 16x13+ 6x22+ 0x23+ 4x33 = 19 x11 x12 x13

X=x21 x22 x230.x31 x32 x33

NoticethatSDP looksremarkablysimilartoalinearprogram. However,thestandardLPconstraintthatxmust lie in the nonnegativeorthant is re-placedbytheconstraintthatthevariableXmust lie in theconeof positivesemidefinitematrices.Justasx0statesthateachofthencomponentsofxmustbenonnegative,itmaybehelpfultothinkofX

0asstating

thateachoftheneigenvalues ofXmustbenonnegative.

6

8/12/2019 MIT15_093J_F09_sdp

7/51

Itiseasytoseethata linearprogramLP isaspecialinstanceofanSDP.Toseeonewayof doingthis,suppose that (c, a1, . . . , am, b1, . . . , bm)comprisethedataforLP.Thendefine:

ai1 0 ... 0

c1 0 ... 0

0 ai2 ... 0 0 c2 ... 0

Ai = ... ... ... ... , i=1, ...,m, and C= ... ... ... ....

0 0 ... ain 0 0 ... cn

ThenLP canbewrittenas:

SDP : minimize C Xs.t. Ai X=bi , i= 1, ... ,m,

Xij = 0, i= 1, . . . , n, j =i + 1, ... , n,X0,

withtheassociationthat

x1 0 . . . 0 0 x2 . . . 0

X= . . . . .. . . . . . . . 0 0 . . . xn

Ofcourse,inpracticeonewouldneverwant toconvertaninstanceof LPintoaninstanceofSDP. TheaboveconstructionmerelyshowsthatSDPincludeslinearprogrammingasaspecialcase.

7

8/12/2019 MIT15_093J_F09_sdp

8/51

5 Semidefinite Programming DualityThe dual problemofSDP is defined (or derived fromfirst principles)to be:

m

SDD: maximize yibii=1

m

s.t. yiAi +S=Ci=1

S

0.

One convenient wayofthinkingabout this problem is as follows.Given mul-mtipliersy1, . . . , ym,theobjective istomaximizethe linear function i=1yibi.

TheconstraintsofSDDstatethatthematrixSdefinedas

m

S=C yiAii=1

mustbepositivesemidefinite.Thatis,

m

C yiAi 0.i=1

We illustrate this construction with the example presented earlier. Thedualproblemis:

SDD: maximize 11y1+ 19y2

1 0 1 0 2 8 1 2 3s.t. y10 3 7+ y22 6 0+ S=2 9 0

1 7 5 8 0 4 3 0 7

S0,8

8/12/2019 MIT15_093J_F09_sdp

9/51

whichwecanrewriteinthefollowingform:

SDD: maximize 11y1+ 19y2

s.t.

1 1y10y2 2 0y12y2 3 1y18y2 2 0y12y2 9 3y16y2 0 7y10y2 0.3 1y18y2 0 7y10y2 7 5y14y2

It is often easier toseeandto workwitha semidefinite program whenit is presented in the formatofthe dualSDD,sincethevariablesarethemmultipliersy1, . . . , ym.

Asinlinearprogramming,wecanswitchfromoneformatofSDP (pri-

malordual)toanyotherformatwithgreatease,andthereisnolossofgenerality inassuminga particular specific formatfor theprimal orthedual.

Thefollowingpropositionstates thatweakdualitymustholdfor theprimalanddualofSDP:

Proposition5.1 Given a feasible solution Xof SDP and a feasible solu-mtion (y, S)of SDD, the duality gap is C X i=1yibi =S X0. If

m

C X i=1yibi = 0, then Xand (y, S) are each optimal solutions to SDPand SDD, respectively, and furthermore, SX= 0.

Inorder to prove Proposition 5.1, itwill beconvenient toworkwiththetrace ofamatrix,definedbelow:

9

8/12/2019 MIT15_093J_F09_sdp

10/51

n

trace(M) = Mjj .j=1

Simplearithmeticcanbeusedto establishthefollowingtwo elementaryidentities:

trace(MN) =trace(NM)A B=trace(ATB)

Proof of Proposition 5.1. Forthefirstpartoftheproposition,wemustshowthatifS0andX0,thenS X0.LetS=PDPT andX=QEQT whereP,QareorthonormalmatricesandD,Earenonnegativediagonalmatrices.Wehave:

S X=trace(STX) =trace(SX) =trace(PDPTQEQT)

n=trace(DPTQEQTP) = Djj (P

TQEQTP)jj 0,j=1

wherethelastinequalityfollowsfromthefactthatallDjj 0andthefactthat the diagonalofthe symmetric positivesemidefinite matrixPTQEQTPmustbenonnegative.

To provethesecond part of the proposition,suppose thattrace(SX) = 0.Thenfromtheaboveequalities,wehave

n

Djj (PTQEQTP)jj = 0.

j=1

10

8/12/2019 MIT15_093J_F09_sdp

11/51

6

However, thisimplies thatforeachj = 1, . . . , n, eitherDjj =0orthe(PTQEQTP)jj =0. Furthermore,thelattercaseimpliesthatthej

th rowofPTQEQTP is allzeros. ThereforeDPTQEQTP =0, andsoSX =PDPTQEQT = 0.

q.e.d.

Unlikethecaseof linear programming,wecannot assertthateitherSDP

orSDDwillattaintheirrespectiveoptima,and/orthattherewillbenodualitygap,unlesscertainregularityconditionshold. OnesuchregularityconditionwhichensuresthatstrongdualitywillprevailisaversionoftheSlatercondition,summarizedinthefollowingtheoremwhichwewillnotprove:

Theorem5.1 Let zP and z denote the optimal objective function valuesD

of SDPand SDD, respectively. Suppose that there exists a feasible solutionX of SDP such that X 0, and that there exists a feasible solution (y,S)of SDDsuch that S0. Then both SDP and SDDattain their optimal

values, and zP =zD.

Key Properties of Linear Programming that do

not extend to SDPThefollowingsummarizessomeofthemoreimportantpropertiesoflinearprogrammingthatdonotextendtoSDP:

There maybe afinite or infinite duality gap.The primaland/or dualmayormaynotattain theiroptima. Asnoted above inTheorem5.1, bothprogramswillattaintheircommonoptimal value ifbothprogramshave feasiblesolutions in the interiorof thesemidefinitecone.

There isnofinite algorithmforsolvingSDP. Thereisa simplexalgorithm,butitisnotafinitealgorithm. ThereisnodirectanalogofabasicfeasiblesolutionforSDP.

11

8/12/2019 MIT15_093J_F09_sdp

12/51

Givenrational data,the feasibleregionmay havenorationalsolutions.Theoptimalsolutionmaynothaverationalcomponentsorrationaleigenvalues.

Givenrational datawhose binaryencoding issizeL,thenormsofanyfeasibleand/oroptimalsolutionsmayexceed22

L(orworse).

Givenrational datawhose binaryencoding issizeL,thenormsofanyfeasibleand/oroptimalsolutionsmaybelessthan22

L(orworse).

7 SDP in Combinatorial Optimization

SDP haswideapplicability incombinatorialoptimization. AnumberofNPhard combinatorialoptimization problems haveconvexrelaxationsthataresemidefiniteprograms. Inmanyinstances,theSDP relaxationisverytight in practice,and in certain instances in particular, theoptimalsolutiontotheSDP relaxationcanbeconvertedtoafeasiblesolutionfortheorigi-nalproblemwithprovablygoodobjectivevalue. AnexampleoftheuseofSDP incombinatorialoptimizationisgivenbelow.

7.1 An SDP Relaxation of the MAX CUT Problem

LetGbeanundirectedgraphwithnodesN=

{1, . . . , n

},andedgesetE.

Letwij =wji betheweightonedge (i, j),for (i, j)E. Weassumethatwij 0 forall (i, j) E.TheMAXCUTproblemistodetermineasubsetSofthenodesN forwhichthesumoftheweightsoftheedgesthatcrossfromStoitscomplementS ismaximized (where (S:=N\S) .WecanformulateMAXCUT asanintegerprogramasfollows.Letxj = 1

forjSandxj =1forjS.Thenourformulationis:n n

MAXCUT: maximizex1 wij(1 xixj)4

i=1j=1

s.t. xj {1, 1}, j = 1, ... , n.

12

8/12/2019 MIT15_093J_F09_sdp

13/51

Nowlet

Y=xxT ,

whereby

Yij =xixj i= 1, . . . , n, j = 1, ... , n.

AlsoletW bethematrixwhose(i, j)th element iswij for i = 1, . . . , n andj = 1, . . . , n.ThenMAXCUTcanbeequivalentlyformulatedas:

n n

MAXCUT : maximizeY,x1

i=1j=1wij WY4

s.t. xj {1, 1}, j = 1, . . . , n

Y=

xx

T

.

Notice inthisproblem that thefirst set ofconstraintsare equivalent toYjj = 1, j= 1, . . . , n.Wethereforeobtain:

n n

MAXCUT: maximizeY,x1

i=1j=1wij WY4

s.t. Yjj = 1, j = 1, . . . , n

Y=xxT .

13

8/12/2019 MIT15_093J_F09_sdp

14/51

8

Lastofall,noticethatthematrix Y =xxT

isasymmetricrank1posi-tivesemidefinitematrix. Ifwerelax thisconditionbyremovingtherank-1restriction,weobtainthefollowingrelaxtionofMAXCUT,whichisasemidefiniteprogram:

n n

RELAX: maximizeY14 wij W Y

i=1j=1

s.t. Yjj = 1, j = 1, . . . , n

Y0.

ItisthereforeeasytoseethatRELAXprovidesanupperboundonMAX-CUT,i.e.,

MAXCUTRELAX.

Asitturnsout,onecanalsoprovewithouttoomucheffortthat:

0.87856RELAXMAXCUTRELAX.

Thisisanimpressiveresult,inthatitstatesthatthevalueofthesemidefi-niterelaxation is guaranteedto benomore than 12% higher than thevalueofNPhardproblemMAXCUT.

SDP in Convex Optimization

Asstatedabove,SDP hasverywideapplicationsinconvexoptimization.The typesof constraintsthatcan bemodeled in theSDPframework include:linear inequalities, convexquadraticinequalities,lowerboundsonmatrix

14

8/12/2019 MIT15_093J_F09_sdp

15/51

norms, lowerboundsondeterminantsof symmetricpositive semidefinitematrices, lower bounds on the geometricmeanofanonnegativevector, plusmanyothers. Usingtheseandotherconstructions,thefollowingproblems(amongmanyothers)canbecastintheformofasemidefiniteprogram:linearprogramming,optimizingaconvexquadraticformsubjecttoconvexquadraticinequalityconstraints,minimizingthevolumeofanellipsoidthatcoversagiven setofpointsandellipsoids,maximizingthevolumeofanellipsoidthatiscontainedinagivenpolytope,plusavarietyofmaximumeigenvalueandminimumeigenvalueproblems.InthesubsectionsbelowwedemonstratehowsomeimportantproblemsinconvexoptimizationcanbereformulatedasinstancesofSDP.

8.1 SDP for Convex Quadratically Constrained Quadratic

Programming, Part I

Aconvexquadraticallyconstrainedquadraticprogramisaproblemoftheform:

QCQP: minimize xTQ0x+q0Tx+c0

xs.t. xTQix+qi

Tx+ci 0 , i= 1, ... , m,

wheretheQ00andQi 0, i= 1, . . . , m.Thisproblemisthesameas:

QCQP: minimize x,s.t. xTQ0x+q0

Tx+c00xTQix+qi

Tx+ci 0 , i= 1, ... , m.

WecanfactoreachQi into

Qi =MiTMi

15

8/12/2019 MIT15_093J_F09_sdp

16/51

forsomematrixMi.Thennotetheequivalence:

xTIMi

T ciMixqiTx 0 xTQix+qiT x+ci 0.

InthiswaywecanwriteQCQPas:

QCQP: minimize x,s.t.

I M0xxTM0

T c0q0Tx+ 0

I MixxTMT i

Tx0 , i= 1, ... ,m.

i ci q

Noticeintheaboveformulationthatthevariablesare andxandthatallmatrixcoefficientsarelinearfunctionsofandx.

8.2 SDP for Convex Quadratically Constrained Quadratic

Programming, Part II

Asitturnsout,thereisanalternativewaytoformulateQCQPasasemidefiniteprogram.Webeginwiththefollowingelementaryproposition.

Proposition8.1 Given a vector x k and a matrix W kk, then1 xT 0 ifandonlyif WxxT .x W

16

8/12/2019 MIT15_093J_F09_sdp

17/51

Usingthis proposition, it isstraghtforwardtoshowthatQCQPisequiv-alenttothefollowingsemidefiniteprogram:

QCQP: minimize x,W,

s.t.

c01 12q0T 1 xT 0q0 Q0

x W2

ci 21qiT 1 xT1qi Qi x W 0 , i= 1, . . . ,m2

1 xT.0

x W

Noticeinthisformulationthattherearenow(n+1)(2n+2) variables,butthat

theconstraintsareall linear inequalitiesasopposedtosemidefinite inequal-ities.

8.3 SDP for the Smallest Circumscribed Ellipsoid Problem

A givenmatrixR0anda given pointzcanbeusedtodefineanellipsoidinn:

ER,z :={y|(y z)TR(y z) 1}.

OnecanprovethatthevolumeofER,z isproportionaltodet(R1).

17

8/12/2019 MIT15_093J_F09_sdp

18/51

SupposewearegivenaconvexsetX n

describedastheconvexhullofkpoints c1, . . . , ck. We would like tofind an ellipsoidcircumscribingthesekpoints that hasminimumvolume.Our problemcan bewritten in the fol-lowingform:

MCP : minimize vol (ER,z)R,zs.t. ci ER,z, i= 1, ... , k,

whichisequivalentto:

MCP : minimize ln(det(R))R,zs.t. (ci z)TR(ci z) 1, i= 1, . . . , k

R0,

Now factorR =M2 whereM 0(thatis,M isasquarerootof R),andnowMCPbecomes:

MCP: minimize ln(det(M2))M,zs.t. (ci z)TMTM(ci z) 1, i= 1, ... , k,

M0.

Nextnoticetheequivalence:

I Mci Mz(Mci Mz)T 1 0 (ci z)TMTM(ci z) 1

18

8/12/2019 MIT15_093J_F09_sdp

19/51

InthiswaywecanwriteMCPas:

MCP: minimizeM,z

2 ln(det(M))

s.t.I

(Mci Mz)TMci Mz

10, i=1, ..., k,

M0.

Lastofall,makethesubstitutiony=Mztoobtain:

MCP: minimize 2 ln(det(M))M,y

s.t.(Mci

Iy)T Mci1y 0, i= 1, ... , k,

M0.

NoticethatthislastprograminvolvessemidefiniteconstraintswhereallofthematrixcoefficientsarelinearfunctionsofthevariablesMandy.How-ever, theobjective function isnot a linear function.It is possible toconvertthis problem further intoa genuine instanceofSDP, because there is awaytoconvertconstraintsoftheform

ln(det(X))

toasemidefinitesystem. Nevertheless,thisisnotnecessary, either from

a theoreticalor a practicalviewpoint, because it turns out that the functionf(X) =ln(det(X)) is extremelywellbehavedand is veryeasyto optimize(bothintheory andinpractice).

19

8/12/2019 MIT15_093J_F09_sdp

20/51

Finally,notethataftersolvingtheformulationofMCP above,wecanrecoverthematrixRandthecenterzoftheoptimalellipsoidbycomputing

R=M2 and z=M1y.

8.4 SDP for the Largest Inscribed Ellipsoid Problem

RecallthatagivenmatrixR0andagivenpointzcanbeusedtodefineanellipsoidinn:

ER,z :={x|(xz)TR(xz) 1},

andthatthevolumeofER,z isproportionaltodet(R1).

Supposewearegivenaconvexset X n describedastheintersectionofkhalfspaces{x|(ai)Txbi}, i= 1, . . . , k,thatis,

X={x|Axb}

wherethe ith rowof thematrix A consists of the entriesof the vectorai, i= 1, . . . , k. WewouldliketofindanellipsoidinscribedinXofmaxi-mumvolume.Ourproblemcanbewritteninthefollowingform:

MIP : maximize vol(ER,z)R,z

s.t. ER,z X,whichisequivalentto:

20

8/12/2019 MIT15_093J_F09_sdp

21/51

MIP : maximize det(R1)R,zs.t. ER,z {x|(ai)Txbi}, i=1, ..., k

R0,

whichisequivalentto:

MIP : maximize ln(det(R1))R,zs.t. maxx{aiTx|(xz)TR(xz) 1} bi, i= 1, . . . , k

R0.

Foragiven i= 1, . . . , k,thesolutiontotheoptimizationproblemintheithconstraintis

R1aix =z+aTR1aii

withoptimalobjectivefunctionvalue

aiT z+ ai

TR1ai ,

andsoMIP canberewrittenas:

21

8/12/2019 MIT15_093J_F09_sdp

22/51

MIP : maximize ln(det(R1))R,z

s.t. aTi z+ aTi R

1ai bi, i= 1, . . . , k R0.

NowfactorR1=M2whereM0 (thatis,M isasquarerootofR1),andnowMIPbecomes:

MIP : maximize ln(det(M2))M,z

s.t. aTi z+ aTi M

TMai bi, i= 1, . . . , k M0,

whichwecanrewriteas:

MIP : maximize 2 ln(det(M))M,zs.t. aTMTMai (bi aiTz)2, i= 1, . . . , k i

bi aiTz0, i= 1, . . . , k M0.

Nextnoticetheequivalence:

(bi ai z)I Mai aTMTMai (bi aTi z)2 T i (Mai)

Ti z)

0 and (bi aT bi aiTz0.

22

8/12/2019 MIT15_093J_F09_sdp

23/51

InthiswaywecanwriteMIPas:

MIP : minimize 2 ln(det(M))M,z

(bi aiTz)I Mais.t.(Mai)

T (bi aTi z) 0, i= 1, ... , k,M0.

NoticethatthislastprograminvolvessemidefiniteconstraintswhereallofthematrixcoefficientsarelinearfunctionsofthevariablesMandz.How-ever,theobjectivefunctionisnotalinearfunction. Itispossible,butnotnecessary in practice,toconvertthis problem further intoa genuine instanceofSDP,becausethereisawaytoconvertconstraintsoftheform

ln(det(X))

toa semidefinitesystem. Suchaconversionisnotnecessary,eitherfroma theoreticalor a practicalviewpoint, because it turns out that the functionf(X) =ln(det(X)) is extremelywellbehavedand is veryeasyto optimize(bothintheory andinpractice).

Finally,notethataftersolvingtheformulationof MIP above,wecanrecoverthematrixRoftheoptimalellipsoidbycomputing

R=M2.

8.5 SDP for Eigenvalue Optimization

Therearemanytypesofeigenvalueoptimizationproblemsthatcanbefor-mualatedasSDPs.Atypicaleigenvalueoptimizationproblemistocreate

23

8/12/2019 MIT15_093J_F09_sdp

24/51

amatrix

k

S:=B wiAii=1

givensymmetric datamatrices BandAi, i= 1, . . . , k,usingweightsw1, . . . , wk,insuchawaytominimizethedifferencebetweenthelargestandsmallesteigenvalueofS.Thisproblemcanbewrittendownas:

EOP : minimizew,S

max(S) min(S)k

s.t. S=Bi=1

wiAi,

wheremin(S)andmax(S)denotethesmallestandthelargesteigenvalueofS,respectively.WenowshowhowtoconvertthisproblemintoanSDP.

RecallthatScanbefactoredintoS=QDQT whereQisanorthonor-malmatrix (i.e.,Q1=QT)andDisadiagonalmatrixconsistingoftheeigenvaluesofS.Theconditions:

ISI

canberewrittenas:

Q(I)QT QDQT Q(I)QT .

After premultiplyingtheaboveQT and postmultiplyingQ,theseconditionsbecome:

24

8/12/2019 MIT15_093J_F09_sdp

25/51

9

IDI

whichareequivalentto:

min(S) and max(S) .

ThereforeEOPcanbewrittenas:

EOP : minimize w,S,,

k

s.t. S=B wiAii=1

ISI.

Thislastproblemisasemidefiniteprogram.

Usingconstructssuchasthoseshownabove,verymanyothertypesofeigenvalueoptimizationproblemscanbeformulatedasSDPs.

SDP in Control Theory

Avarietyofcontrolandsystem problemscan becastandsolvedas instancesofSDP.However,thistopicisbeyondthescopeofthesenotes.

25

8/12/2019 MIT15_093J_F09_sdp

26/51

10 Interiorpoint Methods for SDPAttheheartofaninteriorpointmethodisabarrierfunctionthatexertsarepellingforcefromtheboundaryofthefeasibleregion. ForSDP,weneedabarrierfunctionwhosevaluesapproach+ aspointsXapproachtheboundaryofthesemidefiniteconeSn+.

LetX S+n . ThenX willhaveneigenvalues, say1(X), . . . , n(X)(possiblycountingmultiplicities). Wecancharacterizetheinteriorofthesemidefiniteconeasfollows:

intSn 1(X) >0, . . . , n(X)+={XSn | >0}.

AnaturalbarrierfunctiontousetorepelX fromtheboundaryofSn+thenis

n n

ln(

i(X)) =

ln(

i(X)) =

ln(det(X)).

j=1 j=1

ConsiderthelogarithmicbarrierproblemBSDP()parameterizedbythepositivebarrierparameter:

BSDP() : minimize C X ln(det(X))s.t. Ai X=bi , i= 1, ... ,m,

X0.

Letf(X)denotetheobjectivefunctionofBSDP(). Thenitisnottoodifficulttoderive:

26

8/12/2019 MIT15_093J_F09_sdp

27/51

f(X) =CX1, (1)

andsotheKarushKuhnTuckerconditionsforBSDP() are:

Ai X=bi , i= 1, ... ,m,

X

0, (2) m CX1= yiAi.

i=1

BecauseX issymmetric,we can factorizeX intoX = LLT . We thencandefine

S=X1=LTL1,

whichimplies

1LTSL=I,

andwecanrewritetheKarushKuhnTuckerconditionsas:

27

8/12/2019 MIT15_093J_F09_sdp

28/51

Ai X=bi , i= 1, ... ,m, X0,X=LLT(3)m yiAi +S=C i=1 1I

LTSL= 0.

From the equationsof (3)it follows that if (X,y,S) is a solutionof (3), thenXisfeasibleforSDP, (y, S) isfeasibleforSDD,andtheresultingdualitygapis

n n n n

S X= SijXij = (SX)jj = =n.i=1j=1 j=1 j=1

Thissuggeststhatwe try solvingBSDP() for avariety ofvaluesof as 0.

However,wecannot usuallysolve (3)exactly, because the fourthequationgroup is not linear in the variables.We will instead define aapproximatesolutionoftheKarushKuhnTuckerconditions(3). Beforedoingso,weintroducethefollowingnormonmatrices,calledtheFrobeniusnorm:

n nM:=M M= M2 . ij

i=1j=1

Forsome important propertiesof the Frobeniusnorm,seethe lastsubsection

ofthissection. AapproximatesolutionofBSDP() isdefinedasanysolution (X,y,S) of

28

8/12/2019 MIT15_093J_F09_sdp

29/51

Ai X=bi , i= 1, ... ,m, X0,X=LLT(4)m yiAi +S=C i=1 1I

LTSL .

Lemma10.1 If (X, y,S) is a approximate solution of BSDP()and

8/12/2019 MIT15_093J_F09_sdp

30/51

10.1 The AlgorithmBasedon the analysis just presented,we are motivated to developthe fol-lowingalgorithm:

Step 0 . Initialization. Datais(X0, y0, S0, 0). k=0. Assumethat(X0, y0, S0) isa approximatesolutionofBSDP(0) forsome knownvalueofthatsatisfies

8/12/2019 MIT15_093J_F09_sdp

31/51

Apictureofthisalgorithmlookssomethinglikethis:

31

8/12/2019 MIT15_093J_F09_sdp

32/51

Someoftheunresolvedissuesregardingthisalgorithminclude:

howtosetthefractionaldecreaseparameter thederivationoftheNewtonstepD andthemultipliersy

whetherornotsuccessive iterativevalues (Xk, yk, Sk) areapproximatesolutionstoBSDP(k),and

howtogetthemethodstartedinthefirstplace.10.2 The Newton Step

SupposethatX isafeasiblesolutiontoBSDP():

BSDP() : minimize C X ln(det(X))s.t. Ai X=bi , i= 1, ... ,m,

X0.

LetusdenotetheobjectivefunctionofBSDP() byf(X),i.e.,

f(X) =C X ln(det(X)).

Thenwecanderive:

f(X) =CX1

andthequadraticapproximationofBSDP()atX=X canbederivedas:

32

8/12/2019 MIT15_093J_F09_sdp

33/51

minimize f(X) + (CX1) (XX) + 1X1(XX) X1(XX) 2 X

s.t. Ai X=bi, i= 1, ... , m.LettingD=XX,thisisequivalentto:

minimize (CX1) D +21X1D X1DD

s.t. Ai D= 0, i= 1, ... , m.

Thesolutionto thisprogramwillbetheNewtondirection. TheKarushKuhnTuckerconditionsforthisprogramarenecessaryandsufficient,andare:

m X1 X1 +X1D = yiAi Ci=1 (8)

Ai D= 0, i= 1, ... ,m.TheseequationsarecalledtheNormal Equations.LetD andy denotethesolution to the Normal Equations.Note in particular from thefirstequationin (8)thatD must besymmetric.Suppose that (D,y) is the (unique)so-lutionof the Normal Equations (8).Weobtain thenewvalueofthe primalvariableXbytakingtheNewtonstep,i.e.,

X =X+D .

Wecan producenewvaluesofthe dualvariables (y, S)bysettingthenewm

valueofytobey andbysettingS =C y Ai. Using (8),then,wei=1

havethat

33

8/12/2019 MIT15_093J_F09_sdp

34/51

S =X1X1D X1. (9)

Wehave thefollowingverypowerfulconvergence theoremwhichdemon-strates the quadraticconvergence ofNewtonsmethod forthisproblem,withanexplicit guaranteeoftherange inwhich quadraticconvergencetakesplace.

Theorem10.1 (ExplicitQuadraticConvergenceofNewtonsMethod). Suppose that (X,y, is a approximate solution of BSDP() and

8/12/2019 MIT15_093J_F09_sdp

35/51

S

=X1

X1D

X1

=LT

(IL1D

LT

)L1

.Wewillfirstshowthat L1DLT . It turnsoutthat thisisthecrucialfact fromwhicheverythingwillfollownicely. Toprovethis,notethat

m m m

yiAi +S =C= yi

Ai +S =

yiAi +LT(IL1D LT)L1.

i=1 i=1 i=1

TakingtheinnerproductwithD yields:

S D

=LTL1 D LTL1DLTL1 D,

whichwecanrewriteas:

LTSL L1D

LT =I L1D

LT L1D LT L1D LT ,whichwefinallyrewriteas:

L1D LT 2= I

1LTSL L1DLT .

InvokingtheCauchySchwartzinequalityweobtain:

L1D

LT

2

I

1LTSL

L1DLT

L1D

LT

,

fromwhichweseethatL1D LT .Itthereforefollowsthat

X

=L(I+L1D

LT)LT 0and

S

=LT(IL1D LT)L10,sinceL1D LT

8/12/2019 MIT15_093J_F09_sdp

36/51

wherewedefineL =LM.Thennotethat

I

1(L

)TSL

=I

1MLTSLM

L1) LT)M=I1MLT(LT(IL1DLT) LM=IM(IL1D

=IMM+M(L1D LT)M=IMM+M(MMI)M= (IMM)(IMM)= (L1D

LT)(L1D

LT).

Fromthiswenexthave:

1

I (L)

T

S

L=(L

1

D

LT

)(L1

D

LT

) (L1

D

LT

)2

2

.

Thisshows that (X

, y

, S

) isa2approximatesolutionofBSDP().q.e.d.

10.3 Complexity Analysis of the Algorithm

Theorem10.2 (Relaxation Theorem). Suppose that (X, y,S)is a -approximate solution of BSDP() and

8/12/2019 MIT15_093J_F09_sdp

37/51

+ 1

n = +

n n

= +

n n= .

q.e.d.

Theorem10.3 (ConvergenceTheorem). Suppose that (X0, y0, S0) is aapproximate solution of BSDP(0) and

8/12/2019 MIT15_093J_F09_sdp

38/51

Abetterpictureofthisalgorithmlookssomethinglikethis:

38

8/12/2019 MIT15_093J_F09_sdp

39/51

Theorem10.4 (ComplexityTheorem). Suppose that (X0, y0, S0)is a= 14approximate solution of BSDP(0). In order to obtain primal anddual feasible solutions (Xk, yk, Sk) with a duality gap of at most , one needsto run the algorithm for at most

k= 6

n ln1.25X0S00.75

iterations.

Proof: Letkbeasdefinedabove.Notethat1 1

= 1

+

n= 1 2+4n = 1 2 + 41n1 6

1

n.12

Therefore

k 1 61

n

k0.

Thisimpliesthat

CXk m biyik =Xk Sk kn(1 + ) 1

61

n

k(1.25n0)

i=1

k1 X0 S0 1 6n (1.25n) 0.75n ,

from (5).Takinglogarithms,weobtain

m ln C Xk

biy

k k ln 1 + ln 1.25X0 S0 i 1 6n 0.75 i=1

1.25X0

6

kn

+ ln0.75

S01.25X0 S0 1.25 ln0.75

+ ln

0.75X0S0 = ln().

m

ThereforeCXk i=1

biyik .

q.e.d.

39

8/12/2019 MIT15_093J_F09_sdp

40/51

10.4 How to Start the Method from a Strictly Feasible PointThealgorithmand its performancerelieson having astarting point (X0, y0, S0)thatisaapproximatesolutionofthe problemBSDP(0).Inthissubsec-tion,weshowhowtoobtainsuchastartingpoint,givenapositivedefinitefeasiblesolutionX0ofSDP.

Wesupposethatwearegivenatargetvalue0 ofthebarrierparam-eter, andwearegivenX = X0 that isfeasible forBSDP(0), that is,Ai X

0=bi, i= 1, . . . , m,andX00. Wewillattempttoapproximately

solveBSDP(0)startingatX=X0,usingtheNewtondirectionateachiteration.Theformalstatementofthealgorithmisasfollows:

Step0. Initialization. Datais (X0, 0).k= 0.AssumethatX0satisfiesAi X

0=bi, i= 1, ... ,m,X00.

Step1. SetCurrentvalues. X =Xk.FactorX =LLT .Step2. ComputeNewtonDirectionandMultipliers. ComputetheNewtonstepD

forBSDP(0) atX=X bysolvingthefollowingsystemofequations in thevariables (D, y):

m X1+0X1DX1 C0 = yiAii=1 (10)

Ai D= 0, i= 1, ... , m.

Denote thesolution to thissystem by (D,y).Set

m

S =C yiAi.i=1

Step 3. Test the Current Point. IfL1D LT 1 ,stop. Inthis4case,X isa14 approximatesolutionofBSDP(

0),along withthedualval-

ues (y,S).

40

8/12/2019 MIT15_093J_F09_sdp

41/51

Step4. UpdatePrimalPoint.X =X+D

where0.2

= .L1D LTAlternatively,canbecomputedbyalinesearchoff0(X+ D).

Step 5. Reset Counter and Continue. Xk+1 X , k k+1. GotoStep1.

ThefollowingpropositionvalidatesStep3ofthealgorithm:

Proposition10.1 Suppose that (D,y) is the solution of the Normal equa-tions (10) for the point X for the given value 0 of the barrier parameter,and that

L1DLT 14.

Then X is a 14approximate solution of BSDP(0).

m

Proof: Wemustexhibitvalues (y, S) thatsatisfy yiAi +S=Candi=1

1LTSL 1.I0 4m

Let (D,y) solvethe Normalequations (10),and letS =C yiAi.Then

i=1

wehavefrom (10)that

1 1I

0LTS

L =I0

LT(0(LTL1LTL1D LTL1))L =L1DLT ,

whereby

I10 L

T

S

L=L1D

LT

41.

q.e.d.

41

8/12/2019 MIT15_093J_F09_sdp

42/51

ThenextpropositionshowsthatwheneverthealgorithmproceedstoStep4,thentheobjectivefunctionf0(X) decreasesbyatleast0.0250:

Proposition10.2 Suppose that X satisfies Ai X =bi, i= 1, . . . , m, andX0. Suppose that (D,y)is the solution of the Normal equations (10)

for the point X for a given value 0of the barrier parameter, and that

L1DLT>1.4

Then for all [0, 1), 2

f0 X +L1

D

LT

D

f0(X) + 0

L1D

LT

+

2(1 ).

In particular,

0.2 f0 X + L1D LT D f0(X) 0.0250. (11)

Inorderto provethis proposition,wewillneedtwo powerful factsaboutthelogarithmfunction:

Fact1. Supposethat|x|

8/12/2019 MIT15_093J_F09_sdp

43/51

q.e.d.Fact2. SupposethatRSn andthatR

8/12/2019 MIT15_093J_F09_sdp

44/51

Then

f0 X +L1D

LTD

= f0 X +D

= C X +C D 0ln(det(L(I+L1DLT)LT))

= C X 0ln(det(X)) + C D 0ln(det(I+L1D LT)) f0(X) + CD 0IL1D LT +0 22(1)= f0(X) + C D

0LTL1 D +0

2(1

2)

= f0(X) + C0X1 D +0 2 2(1) .

Now, (D , y) solvetheNormalequations:

m X1+0X1D X1 C0 =y Ai ii=1 (12)

Ai D = 0, i= 1, ... ,m.

Takingtheinnerproductofbothsidesofthefirstequationabovewith Dandrearrangingyields:

0X1D

X1D

=(C0X1) D .Substitutingthisinourinequalityaboveyields:

44

8/12/2019 MIT15_093J_F09_sdp

45/51

f0 X +L1D

LTD

f0(X) 0X1D X1D +0 2(12)= f0(X) 0L1D LT L1D LT +0 2(12)= f0(X) 0L1D LT2+0 2(12)= f0(X) 0L1D LT+0 22(1)

= f0(X

) +

0

L1

D

LT

+2(1

2) .

Subsituting= 0.2andandL1DLT>14 yieldsthefinalresult.q.e.d.

Lastofall,weproveaboundonthenumberofiterationsthatthealgo-rithmwillneedinordertofinda14 approximatesolutionofBSDP(

0):

Proposition10.3 Suppose that X0satisfies Ai X0=bi, i= 1, . . . , m, andX00. Let 0 be given and let f

0 be the optimal objective function valueof BSDP(0). Then the algorithm initiated at X0will find a 14approximate

solution of BSDP(0) in at most

f0(X0) fk=

00.0250

iterations.

Proof: This follows immediatelyfrom (11).Each iteration that isnota14-approximatesolutiondecreasestheobjectivefunctionf0(X) ofBSDP(0)byatleast0.0250.Therefore,therecannotbemorethan

f0(X0) f00.025

0

iterationsthatarenot14 approximatesolutionsofBSDP(0).

q.e.d.

45

8/12/2019 MIT15_093J_F09_sdp

46/51

10.5 Some Properties of the Frobenius NormProposition10.4 If MSn, then

n

1. M = (j(M))2,where 1(M), 2(M), . . . , n(M) is an enu-j=1

meration of the neigenvalues of M.

2. If is any eigenvalue of M, then || M.3. |trace(M)| nM.4. If M0,andsoM0.q.e.d.

Proposition10.5 If A,BSn, then AB AB.Proof: Wehave 2 n n n

AB= AikBkji=1j=1 k=1

46

8/12/2019 MIT15_093J_F09_sdp

47/51

= =

n n n n

ik kj A2 B2i=1j=1 k=1 k=1

n n n nA2 B2ik kj AB.

i=1k=1 j=1k=1

q.e.d.

11 Issues in Solving SDP using the Ellipsoid Al-gorithm

Toseehowtheellipsoidalgorithmisusedtosolveasemidefiniteprogram,assumeforconveniencethattheformatoftheproblemisthatofthedualproblemSDD.Thenthefeasibleregionoftheproblemcanbewrittenas:

m

F= (y1, . . . , ym) m |Ci=1

yiAi 0 ,

andtheobjectivefunctionis mi=1yibi. NotethatF isjustaconvexre-gioninm.

Recallthatatany iterationoftheellipsoidalgorithm,thesetofsolutionsofSDDis knownto lie inthecurrentellipsoid,andthecenterofthecurrentellipsoidis,say,y= (y1, . . . ,ym). Ify F,thenweperformanoptimality

m mcutoftheform i=1yibi i=1yibi,andusestandardformulastoupdatetheellipsoidanditsnewcenter.If F,thenweperformafeasibilitycuty /bycomputingavectorh m suchthathTy > hTy forallyF.

TherearefourissuesthatmustberesolvedinordertoimplementtheaboveversionoftheellipsoidalgorithmtosolveSDD:

47

8/12/2019 MIT15_093J_F09_sdp

48/51

1.Testingify F. ThisisdonebycomputingthematrixS=Cmi=1yiAi. IfS 0,then y F. TestingifthematrixS 0can

bedonebycomputinganexactCholeskyfactorizationofS,whichtakesO(n3) operations,assumingthatcomputingsquarerootscanbeperformed (exactly)inoneoperation.

2.Computingafeasibilitycut. Asabove,testingifS 0canbecom-puted inO(n3) operations.IfS0,thenagainassumingexactarith-metic,wecanfindannvectorvsuchthatTS 3)oper-v v

8/12/2019 MIT15_093J_F09_sdp

49/51

determinedbyexaminingtheinputlengthofthedata,asisthecaseinlinearprogramming.Oneneedstoknowsomespecialinformationabout the specificproblemathand inorderto determinerbeforesolvingthesemidefiniteprogram.

12 Current Research in SDP

Therearemanyveryactiveresearchareasinsemidefiniteprogramminginnonlinear (convex) programming, in combinatorialoptimization,and in con-troltheory. Intheareaofconvexanalysis,recentresearchtopicsincludethe geometryand the boundarystructureofSDPfeasibleregions (including

notionsofdegeneracy)andresearchrelatedtothecomputationalcomplex-ityofSDP suchasdecidabilityquestions,certificatesofinfeasibility,anddualitytheory. Intheareaofcombinatorialoptimization,therehasbeenmuchresearchonthe practicalandthetheoreticaluseofSDPrelaxationsofhardcombinatorialoptimizationproblems.Asregardsinteriorpointmeth-ods,thereareahostofresearchissues,mostlyinvolvingthedevelopmentofdifferentinteriorpointalgorithmsandtheirproperties,includingratesofconvergence,performanceguarantees,etc.

13 Computational State of the Art of SDP

BecauseSDPhas so manyapplications,and because interior pointmethodsshowsomuchpromise,perhapsthemostexcitingareaofresearchonSDPhas to do withcomputation and implementationof interior pointalgorithmsforsolvingSDP. MuchresearchhasfocusedonthepracticalefficiencyofinteriorpointmethodsforSDP. However, intheresearchtodate,com-putationalissueshavearisenthataremuchmorecomplexthanthoseforlinearprogramming,andthesecomputationalissuesareonlybeginningtobe wellunderstood.They probablystem from a varietyof factors, includingthe factthatSDPisnot guaranteedto havestrictlycomplementaryoptimalsolutions (as is the case in linear programming). Finally, becauseSDP is

suchanewfield,thereisnorepresentativesuiteofpracticalproblemsonwhichtotestalgorithms,i.e.,thereisnoequivalentversionofthe netlibsuiteofindustriallinearprogrammingproblems.

49

8/12/2019 MIT15_093J_F09_sdp

50/51

Agoodwebsiteforsemidefiniteprogrammingis:

50

8/12/2019 MIT15_093J_F09_sdp

51/51

MIT OpenCourseWarehttp://ocw.mit.edu

15.093J / 6.255J Optimization Methods

Fall2009

For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.

http://ocw.mit.edu/http://ocw.mit.edu/termshttp://ocw.mit.edu/termshttp://ocw.mit.edu/