conic optimisation: why, when and how should it be used. · if you can cast your problem in conic...
TRANSCRIPT
![Page 1: Conic Optimisation: Why, When and How Should it be Used. · If you can cast your problem in conic form, then it is convex! No need for long and tedious convexity proof! Plenty of](https://reader033.vdocuments.site/reader033/viewer/2022053012/5f0fb95c7e708231d4459268/html5/thumbnails/1.jpg)
Conic Optimisation:
Why, When and How Should it be Used.
A Short Trip Along a Basic yet Powerful Optimisation Tool
OSE Workshop 2015 – 17 September 2015 Glasgow (UK)
Andrea Cassioli, [email protected]
www.mosek.com
![Page 2: Conic Optimisation: Why, When and How Should it be Used. · If you can cast your problem in conic form, then it is convex! No need for long and tedious convexity proof! Plenty of](https://reader033.vdocuments.site/reader033/viewer/2022053012/5f0fb95c7e708231d4459268/html5/thumbnails/2.jpg)
Conic Optimization I
Standard form
min∑k
cTk xk +∑k
〈Ck , Xk〉
s.t. ∑k
aTikxk +∑k
〈Aik , Xik〉 = bi i = 1, . . .
xk ∈ Kk k = 1, . . .Xk ∈ Kk k = 1, . . .
where Kk , Kk are proper cones.
1 / 24
![Page 3: Conic Optimisation: Why, When and How Should it be Used. · If you can cast your problem in conic form, then it is convex! No need for long and tedious convexity proof! Plenty of](https://reader033.vdocuments.site/reader033/viewer/2022053012/5f0fb95c7e708231d4459268/html5/thumbnails/3.jpg)
Conic Optimization II
We only consider
Positive Orth. R+ = x ∈ Rn|x ≥ 0
Lorentz Q = (t, x) ∈ R× Rn|t ≥ ‖x‖2
Rotated Qr =
(z , y , x) ∈ R× R× Rn|2zy ≥ ‖x‖22, z ,w ≥ 0
PSD Matrices S+ =X ∈ Rn×n|X is PSD
Other more exotic cones can be defined, still an active researchfield.
2 / 24
![Page 4: Conic Optimisation: Why, When and How Should it be Used. · If you can cast your problem in conic form, then it is convex! No need for long and tedious convexity proof! Plenty of](https://reader033.vdocuments.site/reader033/viewer/2022053012/5f0fb95c7e708231d4459268/html5/thumbnails/4.jpg)
Conic Optimization III
Properties
• convexity guarantee,
• rich duality theory,
• (almost) avoid ill-posedness,
• polynomial time complexity,
• deeply investigated, but still active research field,
• many good implementations!
3 / 24
![Page 5: Conic Optimisation: Why, When and How Should it be Used. · If you can cast your problem in conic form, then it is convex! No need for long and tedious convexity proof! Plenty of](https://reader033.vdocuments.site/reader033/viewer/2022053012/5f0fb95c7e708231d4459268/html5/thumbnails/5.jpg)
Convexity guarantee
If you can cast your problem in conic form, then it is convex!No need for long and tedious convexity proof!
Plenty of rules and examples:
Boyd, S. and Vandenberghe, L., Convex optimization,Cambridge University Press, 2004.
Ben-Tal, A. and Nemirovski, A, Lectures on Modern ConvexOptimization: Analysis, Algorithms, and EngineeringApplications, SIAM, 2001.
Alizadeh, F. and Goldfarb, D. Second-order coneprogramming, Math. Programming 95-1, 2003.
4 / 24
![Page 6: Conic Optimisation: Why, When and How Should it be Used. · If you can cast your problem in conic form, then it is convex! No need for long and tedious convexity proof! Plenty of](https://reader033.vdocuments.site/reader033/viewer/2022053012/5f0fb95c7e708231d4459268/html5/thumbnails/6.jpg)
Conic Modeling I
What can we represent with CO?
Trivially:
• LP
• QP
• QCQP
• SOCP
• SDP
5 / 24
![Page 7: Conic Optimisation: Why, When and How Should it be Used. · If you can cast your problem in conic form, then it is convex! No need for long and tedious convexity proof! Plenty of](https://reader033.vdocuments.site/reader033/viewer/2022053012/5f0fb95c7e708231d4459268/html5/thumbnails/7.jpg)
Conic Modeling II
• Geometric mean:(x , t) ∈ Rn+1|(x1x2 . . . xn)1/n ≥ t, x ≥ 0
• Harmonic mean:
(x , t) ∈ Rn+1|(
1
2
∑x−1i
)−1≥ t, x ≥ 0
6 / 24
![Page 8: Conic Optimisation: Why, When and How Should it be Used. · If you can cast your problem in conic form, then it is convex! No need for long and tedious convexity proof! Plenty of](https://reader033.vdocuments.site/reader033/viewer/2022053012/5f0fb95c7e708231d4459268/html5/thumbnails/8.jpg)
Conic Modeling III
• Increasing powers:(x , t)|xp/q ≤ t, x ≥ 0
, p/q ≥ 1, p, q ∈ Z+
• Decrease powers:(x , t)|x−p/q ≤ t, x ≥ 0
, p, q ∈ Z+
• Power cones:(x , y) ∈ Rn
+ × R| |y | ≤∏
xαj
j
, α > 0,
∑αj = 0
7 / 24
![Page 9: Conic Optimisation: Why, When and How Should it be Used. · If you can cast your problem in conic form, then it is convex! No need for long and tedious convexity proof! Plenty of](https://reader033.vdocuments.site/reader033/viewer/2022053012/5f0fb95c7e708231d4459268/html5/thumbnails/9.jpg)
Conic Modeling IV
Eigenvalue related problems:
• largest/smallest eigenvalue
• eigenvalue spread
• spectral radius
• minimize condition number of a PSD matrix
• singular values.
8 / 24
![Page 10: Conic Optimisation: Why, When and How Should it be Used. · If you can cast your problem in conic form, then it is convex! No need for long and tedious convexity proof! Plenty of](https://reader033.vdocuments.site/reader033/viewer/2022053012/5f0fb95c7e708231d4459268/html5/thumbnails/10.jpg)
Conic Modeling - Example
(Linear) Regressions/Regularization
min φ(δ)δ = Ax − yx ∈ Ω
φ(·) is typically
• ‖ · ‖1, ‖ · ‖2, ‖ · ‖∞• 3/2−regression
• k−largest penalty
• Huber penalty
• Dead-zone linear penalty
9 / 24
![Page 11: Conic Optimisation: Why, When and How Should it be Used. · If you can cast your problem in conic form, then it is convex! No need for long and tedious convexity proof! Plenty of](https://reader033.vdocuments.site/reader033/viewer/2022053012/5f0fb95c7e708231d4459268/html5/thumbnails/11.jpg)
Conic Modeling - Example
Nearest correlation matrix
min ‖X − Y ‖FXii = 1 i = 1, . . . , nX ∈ Sn+
where ‖A‖F =√
tr(ATA) =
√∑i ,j
A2ij , and hence
min t(t, vec(X − Y )) ∈ QXii = 1 i = 1, . . . , nX ∈ Sn+
10 / 24
![Page 12: Conic Optimisation: Why, When and How Should it be Used. · If you can cast your problem in conic form, then it is convex! No need for long and tedious convexity proof! Plenty of](https://reader033.vdocuments.site/reader033/viewer/2022053012/5f0fb95c7e708231d4459268/html5/thumbnails/12.jpg)
Conic Modeling - Examples
For a polyhedron, find an ellipsoid E defined by a matrix C :
Maximal inscribed ellipsoid: Vol(E) ≈ det(C )1/n
Minimal enclosing ellipsoid: Vol(E) ≈ det(C )−1/n
Much interest in the robotic community.11 / 24
![Page 13: Conic Optimisation: Why, When and How Should it be Used. · If you can cast your problem in conic form, then it is convex! No need for long and tedious convexity proof! Plenty of](https://reader033.vdocuments.site/reader033/viewer/2022053012/5f0fb95c7e708231d4459268/html5/thumbnails/13.jpg)
Conic Modeling - Examples
Polynomial curve-fitting
For a given degree, condition on
• interpolation
• non-negativity
• monotonicity
• convexity/concavity
f2(t)
f4(t)
f8(t)12
1
32
−2 −1 0 1 2 t
12 / 24
![Page 14: Conic Optimisation: Why, When and How Should it be Used. · If you can cast your problem in conic form, then it is convex! No need for long and tedious convexity proof! Plenty of](https://reader033.vdocuments.site/reader033/viewer/2022053012/5f0fb95c7e708231d4459268/html5/thumbnails/14.jpg)
Conic Modeling
There is plenty of material out there.
Advice
Always try to cast your problem in conic form!
13 / 24
![Page 15: Conic Optimisation: Why, When and How Should it be Used. · If you can cast your problem in conic form, then it is convex! No need for long and tedious convexity proof! Plenty of](https://reader033.vdocuments.site/reader033/viewer/2022053012/5f0fb95c7e708231d4459268/html5/thumbnails/15.jpg)
CO polynomial complexity
The problem size is not just the combination of the number ofvariables and constraints, but also the number of bits to representthe problem!
Advise
Do not be afraid of transformations that introduce additionalvariables/constraints.
A larger but sparser problem can have the same complexity that asmaller but denser.
14 / 24
![Page 16: Conic Optimisation: Why, When and How Should it be Used. · If you can cast your problem in conic form, then it is convex! No need for long and tedious convexity proof! Plenty of](https://reader033.vdocuments.site/reader033/viewer/2022053012/5f0fb95c7e708231d4459268/html5/thumbnails/16.jpg)
SOCP vs. QCQP
SOCP contains QCQP, which one should one use?
• in general no significant differences in complexity
• SOCP more general, ready for extensions
• some specialized algorithms for structured QP/QCQP exist
• our experience indicates SOCP more robust.
15 / 24
![Page 17: Conic Optimisation: Why, When and How Should it be Used. · If you can cast your problem in conic form, then it is convex! No need for long and tedious convexity proof! Plenty of](https://reader033.vdocuments.site/reader033/viewer/2022053012/5f0fb95c7e708231d4459268/html5/thumbnails/17.jpg)
From QCQP to SOCP I
To convert QCQP to SOCP you need to factorize the Q matrix...
min xTQxx ∈ X
↓
min t
t ≥ xTQx = xTFFT x = ‖Fx‖2x ∈ X
↓
min t(1/2, t,Fx) ∈ Qr
x ∈ X16 / 24
![Page 18: Conic Optimisation: Why, When and How Should it be Used. · If you can cast your problem in conic form, then it is convex! No need for long and tedious convexity proof! Plenty of](https://reader033.vdocuments.site/reader033/viewer/2022053012/5f0fb95c7e708231d4459268/html5/thumbnails/18.jpg)
From QCQP to SOCP II
Forming F may sound tricky, but it can pay:
• You can exploit the structure better than the solver;
• Sometimes F already given;
• You may need to factorize to check Q ∈ S+;
• Recover from errors.
17 / 24
![Page 19: Conic Optimisation: Why, When and How Should it be Used. · If you can cast your problem in conic form, then it is convex! No need for long and tedious convexity proof! Plenty of](https://reader033.vdocuments.site/reader033/viewer/2022053012/5f0fb95c7e708231d4459268/html5/thumbnails/19.jpg)
From QCQP to SOCP III
minimize −x1 − x2subject to (x1 − x2)2 ≤ 0,
0 ≤ x1, x2 ≤ 1
What if the quadratic constraints is affected by a small error ε, i.e.
xT[
1 −1−1 1 + ε
]x ≤ 0
Observe:
• ε < 0 : The problem is not convex.
• ε = 0 : x∗1 = x∗2 = 1.
• ε > 0 : x∗1 = x∗2 = 0.
18 / 24
![Page 20: Conic Optimisation: Why, When and How Should it be Used. · If you can cast your problem in conic form, then it is convex! No need for long and tedious convexity proof! Plenty of](https://reader033.vdocuments.site/reader033/viewer/2022053012/5f0fb95c7e708231d4459268/html5/thumbnails/20.jpg)
From QCQP to SOCP III
Advise
Do try to cast a QCQP as a SOCP yourself!
19 / 24
![Page 21: Conic Optimisation: Why, When and How Should it be Used. · If you can cast your problem in conic form, then it is convex! No need for long and tedious convexity proof! Plenty of](https://reader033.vdocuments.site/reader033/viewer/2022053012/5f0fb95c7e708231d4459268/html5/thumbnails/21.jpg)
Exploit duality I
In CO duality allows to
1 Automatic dual formulation (pretty much like LPs)
2 Numeric infeasibility certificate (approximate Farkas’ Lemma)
For primal-dual algorithms there is no theoretical difference inusing primal or dual formulation, but...
20 / 24
![Page 22: Conic Optimisation: Why, When and How Should it be Used. · If you can cast your problem in conic form, then it is convex! No need for long and tedious convexity proof! Plenty of](https://reader033.vdocuments.site/reader033/viewer/2022053012/5f0fb95c7e708231d4459268/html5/thumbnails/22.jpg)
Exploit duality II
in practice:
• presolver may be affected
• caching issue from different memory setup
• linear algebra routines may be tuned better for (dual) primalformulation.
21 / 24
![Page 23: Conic Optimisation: Why, When and How Should it be Used. · If you can cast your problem in conic form, then it is convex! No need for long and tedious convexity proof! Plenty of](https://reader033.vdocuments.site/reader033/viewer/2022053012/5f0fb95c7e708231d4459268/html5/thumbnails/23.jpg)
Exploit duality
In our experience some times up to 10x speed up!
Advise
Give the dual formulation a chance!
Some framework like CVX will do it automagically for you...
22 / 24
![Page 24: Conic Optimisation: Why, When and How Should it be Used. · If you can cast your problem in conic form, then it is convex! No need for long and tedious convexity proof! Plenty of](https://reader033.vdocuments.site/reader033/viewer/2022053012/5f0fb95c7e708231d4459268/html5/thumbnails/24.jpg)
Software
Great tools/solvers for CO available:
• CVX, YALMIP, Julia Convex, PICOS, ...
• MOSEK, CPLEX, Gurobi,...
• Sedumi, SDPA, SDPT3, CVXOPT, ...
Advice
Do not re-invent the wheel!
23 / 24
![Page 25: Conic Optimisation: Why, When and How Should it be Used. · If you can cast your problem in conic form, then it is convex! No need for long and tedious convexity proof! Plenty of](https://reader033.vdocuments.site/reader033/viewer/2022053012/5f0fb95c7e708231d4459268/html5/thumbnails/25.jpg)
Conclusion
My suggestions:
1 use CO whenever possible
2 rely on proven technology even for cutting edge research
3 give the dual problem a chance
4 move to SOCP to QP/QCQP unless a special case is at hand
5 don’t be afraid of larger but sparser formulations
6 share your experiences and issues with the developers!
24 / 24