living on the edge - duke electrical and computer...
TRANSCRIPT
![Page 1: Living on the Edge - Duke Electrical and Computer Engineeringpeople.ee.duke.edu/~lcarin/SAHD_Tropp.pdf · 2013. 8. 3. · Living on the Edge, SAHD 2013, Durham, 24 July 2013 27. Inference,](https://reader034.vdocuments.site/reader034/viewer/2022051823/5fed82ef3233ee586470e8ab/html5/thumbnails/1.jpg)
Living on the Edge¦
Phase Transitions in Random Convex Programs
Joel A. TroppMichael B. McCoy
Computing + Mathematical Sciences
California Institute of Technology
Joint with Dennis Amelunxen and Martin Lotz (Manchester)
Including work of Samet Oymak and Babak Hassibi (Caltech)
Research supported in part by ONR, AFOSR, DARPA, and the Sloan Foundation 1
![Page 2: Living on the Edge - Duke Electrical and Computer Engineeringpeople.ee.duke.edu/~lcarin/SAHD_Tropp.pdf · 2013. 8. 3. · Living on the Edge, SAHD 2013, Durham, 24 July 2013 27. Inference,](https://reader034.vdocuments.site/reader034/viewer/2022051823/5fed82ef3233ee586470e8ab/html5/thumbnails/2.jpg)
...in which we dilate upon the question...
How big is a cone?
(and why you should care)
Living on the Edge, SAHD 2013, Durham, 24 July 2013 2
![Page 3: Living on the Edge - Duke Electrical and Computer Engineeringpeople.ee.duke.edu/~lcarin/SAHD_Tropp.pdf · 2013. 8. 3. · Living on the Edge, SAHD 2013, Durham, 24 July 2013 27. Inference,](https://reader034.vdocuments.site/reader034/viewer/2022051823/5fed82ef3233ee586470e8ab/html5/thumbnails/3.jpg)
.
Statistical.Dimension
Living on the Edge, SAHD 2013, Durham, 24 July 2013 3
![Page 4: Living on the Edge - Duke Electrical and Computer Engineeringpeople.ee.duke.edu/~lcarin/SAHD_Tropp.pdf · 2013. 8. 3. · Living on the Edge, SAHD 2013, Durham, 24 July 2013 27. Inference,](https://reader034.vdocuments.site/reader034/viewer/2022051823/5fed82ef3233ee586470e8ab/html5/thumbnails/4.jpg)
The Statistical Dimension of a Cone
Definition [Amelunxen, Lotz, McCoy, T 2013].
.
The statistical dimension δ(K) of a closed, convex cone K
is the quantity
δ(K) := E(‖ΠK(g)‖22
).
where
§ ΠK is the Euclidean projection onto K
§ g is a standard normal vector
Living on the Edge, SAHD 2013, Durham, 24 July 2013 4
![Page 5: Living on the Edge - Duke Electrical and Computer Engineeringpeople.ee.duke.edu/~lcarin/SAHD_Tropp.pdf · 2013. 8. 3. · Living on the Edge, SAHD 2013, Durham, 24 July 2013 27. Inference,](https://reader034.vdocuments.site/reader034/viewer/2022051823/5fed82ef3233ee586470e8ab/html5/thumbnails/5.jpg)
Statistical Dimension: The Motion Picture
K
g
ΠK(g)
0
small cone
K
g
ΠK(g)
0
big cone
Living on the Edge, SAHD 2013, Durham, 24 July 2013 5
![Page 6: Living on the Edge - Duke Electrical and Computer Engineeringpeople.ee.duke.edu/~lcarin/SAHD_Tropp.pdf · 2013. 8. 3. · Living on the Edge, SAHD 2013, Durham, 24 July 2013 27. Inference,](https://reader034.vdocuments.site/reader034/viewer/2022051823/5fed82ef3233ee586470e8ab/html5/thumbnails/6.jpg)
Basic Statistical Dimension Calculations
Cone Notation Statistical Dimension
Subspace Lj j
Nonnegative orthant Rd+ 12d
Second-order cone Ld+1 12(d+ 1)
Real psd cone Sd+ 14d(d− 1)
Complex psd cone Hd+ 12d
2
Living on the Edge, SAHD 2013, Durham, 24 July 2013 6
![Page 7: Living on the Edge - Duke Electrical and Computer Engineeringpeople.ee.duke.edu/~lcarin/SAHD_Tropp.pdf · 2013. 8. 3. · Living on the Edge, SAHD 2013, Durham, 24 July 2013 27. Inference,](https://reader034.vdocuments.site/reader034/viewer/2022051823/5fed82ef3233ee586470e8ab/html5/thumbnails/7.jpg)
Descent Cones
Definition. The descent cone of a function f at a point x is
D(f,x) := {h : f(x+ εh) ≤ f(x) for some ε > 0}
{y : f(y) ≤ f(x)}
x+ D(f,x)
x{h : f(x+ h) ≤ f(x)}
D(f,x)
0
Living on the Edge, SAHD 2013, Durham, 24 July 2013 7
![Page 8: Living on the Edge - Duke Electrical and Computer Engineeringpeople.ee.duke.edu/~lcarin/SAHD_Tropp.pdf · 2013. 8. 3. · Living on the Edge, SAHD 2013, Durham, 24 July 2013 27. Inference,](https://reader034.vdocuments.site/reader034/viewer/2022051823/5fed82ef3233ee586470e8ab/html5/thumbnails/8.jpg)
Descent Cone of `1 Norm at Sparse Vector
0 1/4 1/2 3/4 10
1/4
1/2
3/4
1
1
Living on the Edge, SAHD 2013, Durham, 24 July 2013 8
![Page 9: Living on the Edge - Duke Electrical and Computer Engineeringpeople.ee.duke.edu/~lcarin/SAHD_Tropp.pdf · 2013. 8. 3. · Living on the Edge, SAHD 2013, Durham, 24 July 2013 27. Inference,](https://reader034.vdocuments.site/reader034/viewer/2022051823/5fed82ef3233ee586470e8ab/html5/thumbnails/9.jpg)
Descent Cone of S1 Norm at Low-Rank Matrix
0 1/4 1/2 3/4 1
1/4
1/2
3/4
1
0
Living on the Edge, SAHD 2013, Durham, 24 July 2013 9
![Page 10: Living on the Edge - Duke Electrical and Computer Engineeringpeople.ee.duke.edu/~lcarin/SAHD_Tropp.pdf · 2013. 8. 3. · Living on the Edge, SAHD 2013, Durham, 24 July 2013 27. Inference,](https://reader034.vdocuments.site/reader034/viewer/2022051823/5fed82ef3233ee586470e8ab/html5/thumbnails/10.jpg)
Aside: The Gaussian Width
§ The Gaussian width w(K) of a convex cone K can be defined as
w(K) := E supx∈K∩ S
〈g, x〉
§ We have the relationship
w(K)2 ≤ δ(K) ≤ w(K)2 + 1
§ Statistical dimension is the canonical extension of the linear dimension
to the class of convex cones. Gaussian width ain’t.
Living on the Edge, SAHD 2013, Durham, 24 July 2013 10
![Page 11: Living on the Edge - Duke Electrical and Computer Engineeringpeople.ee.duke.edu/~lcarin/SAHD_Tropp.pdf · 2013. 8. 3. · Living on the Edge, SAHD 2013, Durham, 24 July 2013 27. Inference,](https://reader034.vdocuments.site/reader034/viewer/2022051823/5fed82ef3233ee586470e8ab/html5/thumbnails/11.jpg)
.
RegularizedDenoising
Living on the Edge, SAHD 2013, Durham, 24 July 2013 11
![Page 12: Living on the Edge - Duke Electrical and Computer Engineeringpeople.ee.duke.edu/~lcarin/SAHD_Tropp.pdf · 2013. 8. 3. · Living on the Edge, SAHD 2013, Durham, 24 July 2013 27. Inference,](https://reader034.vdocuments.site/reader034/viewer/2022051823/5fed82ef3233ee586470e8ab/html5/thumbnails/12.jpg)
Setup for Regularized Denoising
§ Let x\ ∈ Rd be “structured” but unknown
§ Let f : Rd → R be a convex function that reflects “structure”
§ Observe z = x\ + σw where w ∼ normal(0, I)
§ Remove noise by solving the convex program*
minimize1
2‖z − x‖22 subject to f(x) ≤ f(x\)
§ Hope: The minimizer x approximates x\
*We assume the side information f(x\) is available. This is equivalent** to knowing the
optimal choice of Lagrange multiplier for the constraint.
Living on the Edge, SAHD 2013, Durham, 24 July 2013 12
![Page 13: Living on the Edge - Duke Electrical and Computer Engineeringpeople.ee.duke.edu/~lcarin/SAHD_Tropp.pdf · 2013. 8. 3. · Living on the Edge, SAHD 2013, Durham, 24 July 2013 27. Inference,](https://reader034.vdocuments.site/reader034/viewer/2022051823/5fed82ef3233ee586470e8ab/html5/thumbnails/13.jpg)
Geometry of Denoising
{x : f(x) ≤ f(x\)}
x\ + D(f,x\)
zσw
x
error
x\
Living on the Edge, SAHD 2013, Durham, 24 July 2013 13
![Page 14: Living on the Edge - Duke Electrical and Computer Engineeringpeople.ee.duke.edu/~lcarin/SAHD_Tropp.pdf · 2013. 8. 3. · Living on the Edge, SAHD 2013, Durham, 24 July 2013 27. Inference,](https://reader034.vdocuments.site/reader034/viewer/2022051823/5fed82ef3233ee586470e8ab/html5/thumbnails/14.jpg)
The Risk of Regularized Denoising
Theorem 1. [Oymak & Hassibi 2013] Assume
§ We observe z = x\ + σw where w is standard normal
§ The vector x solves
minimize1
2‖z − x‖22 subject to f(x) ≤ f(x\)
Then
supσ>0
E ‖x− x\‖2
σ2= δ
(D(f,x\)
)
Living on the Edge, SAHD 2013, Durham, 24 July 2013 14
![Page 15: Living on the Edge - Duke Electrical and Computer Engineeringpeople.ee.duke.edu/~lcarin/SAHD_Tropp.pdf · 2013. 8. 3. · Living on the Edge, SAHD 2013, Durham, 24 July 2013 27. Inference,](https://reader034.vdocuments.site/reader034/viewer/2022051823/5fed82ef3233ee586470e8ab/html5/thumbnails/15.jpg)
.
Regularized LinearInverse Problems
Living on the Edge, SAHD 2013, Durham, 24 July 2013 15
![Page 16: Living on the Edge - Duke Electrical and Computer Engineeringpeople.ee.duke.edu/~lcarin/SAHD_Tropp.pdf · 2013. 8. 3. · Living on the Edge, SAHD 2013, Durham, 24 July 2013 27. Inference,](https://reader034.vdocuments.site/reader034/viewer/2022051823/5fed82ef3233ee586470e8ab/html5/thumbnails/16.jpg)
Setup for Linear Inverse Problems
§ Let x\ ∈ Rd be a structured, unknown vector
§ Let f : Rd → R be a convex function that reflects structure
§ Let A ∈ Rm×d be a measurement operator
§ Observe z = Ax\
§ Find estimate x by solving convex program
minimize f(x) subject to Ax = z
§ Hope: x = x\
Living on the Edge, SAHD 2013, Durham, 24 July 2013 16
![Page 17: Living on the Edge - Duke Electrical and Computer Engineeringpeople.ee.duke.edu/~lcarin/SAHD_Tropp.pdf · 2013. 8. 3. · Living on the Edge, SAHD 2013, Durham, 24 July 2013 27. Inference,](https://reader034.vdocuments.site/reader034/viewer/2022051823/5fed82ef3233ee586470e8ab/html5/thumbnails/17.jpg)
Geometry of Linear Inverse Problems
x\ + null(A)
{x : f(x) ≤ f(x\)}
x\
x\ + D(f,x\)
Living on the Edge, SAHD 2013, Durham, 24 July 2013 17
![Page 18: Living on the Edge - Duke Electrical and Computer Engineeringpeople.ee.duke.edu/~lcarin/SAHD_Tropp.pdf · 2013. 8. 3. · Living on the Edge, SAHD 2013, Durham, 24 July 2013 27. Inference,](https://reader034.vdocuments.site/reader034/viewer/2022051823/5fed82ef3233ee586470e8ab/html5/thumbnails/18.jpg)
Linear Inverse Problems with Random Data
Theorem 2. [Amelunxen, Lotz, McCoy, T 2013] Assume
§ The vector x\ ∈ Rd is unknown
§ The observation z = Ax\ where A ∈ Rm×d is standard normal
§ The vector x solves
minimize f(x) subject to Ax = z
Thenm & δ
(D(f,x\)
)=⇒ x = x\ whp
m . δ(D(f,x\)
)=⇒ x 6= x\ whp
Related work: Rudelson–Vershynin 2006, Donoho–Tanner 2008, Stojnic 2009, Chandrasekaran et al. 2010
Living on the Edge, SAHD 2013, Durham, 24 July 2013 18
![Page 19: Living on the Edge - Duke Electrical and Computer Engineeringpeople.ee.duke.edu/~lcarin/SAHD_Tropp.pdf · 2013. 8. 3. · Living on the Edge, SAHD 2013, Durham, 24 July 2013 27. Inference,](https://reader034.vdocuments.site/reader034/viewer/2022051823/5fed82ef3233ee586470e8ab/html5/thumbnails/19.jpg)
Sparse Recovery via `1 Minimization
0 25 50 75 1000
25
50
75
100
Living on the Edge, SAHD 2013, Durham, 24 July 2013 19
![Page 20: Living on the Edge - Duke Electrical and Computer Engineeringpeople.ee.duke.edu/~lcarin/SAHD_Tropp.pdf · 2013. 8. 3. · Living on the Edge, SAHD 2013, Durham, 24 July 2013 27. Inference,](https://reader034.vdocuments.site/reader034/viewer/2022051823/5fed82ef3233ee586470e8ab/html5/thumbnails/20.jpg)
Low-Rank Recovery via S1 Minimization
0 10 20 300
300
600
900
Living on the Edge, SAHD 2013, Durham, 24 July 2013 20
![Page 21: Living on the Edge - Duke Electrical and Computer Engineeringpeople.ee.duke.edu/~lcarin/SAHD_Tropp.pdf · 2013. 8. 3. · Living on the Edge, SAHD 2013, Durham, 24 July 2013 27. Inference,](https://reader034.vdocuments.site/reader034/viewer/2022051823/5fed82ef3233ee586470e8ab/html5/thumbnails/21.jpg)
.
DemixingStructured Signals
Living on the Edge, SAHD 2013, Durham, 24 July 2013 21
![Page 22: Living on the Edge - Duke Electrical and Computer Engineeringpeople.ee.duke.edu/~lcarin/SAHD_Tropp.pdf · 2013. 8. 3. · Living on the Edge, SAHD 2013, Durham, 24 July 2013 27. Inference,](https://reader034.vdocuments.site/reader034/viewer/2022051823/5fed82ef3233ee586470e8ab/html5/thumbnails/22.jpg)
Setup for Demixing Problems
§ Let x\ ∈ Rd and y\ ∈ Rd be structured, unknown vectors
§ Let f, g : Rd → R be convex functions that reflect “structure”
§ Let U ∈ Rd×d be a known orthogonal matrix
§ Observe z = x\ +Uy\
§ Demix via convex program
minimize f(x) subject to g(y) ≤ g(y\)
x+Uy = z
§ Hope: (x, y) = (x\,y\)
Living on the Edge, SAHD 2013, Durham, 24 July 2013 22
![Page 23: Living on the Edge - Duke Electrical and Computer Engineeringpeople.ee.duke.edu/~lcarin/SAHD_Tropp.pdf · 2013. 8. 3. · Living on the Edge, SAHD 2013, Durham, 24 July 2013 27. Inference,](https://reader034.vdocuments.site/reader034/viewer/2022051823/5fed82ef3233ee586470e8ab/html5/thumbnails/23.jpg)
Geometry of Demixing Problems
x\
x\ + D(f,x\)
x\ −UD(g,y\)
Success!
x\
x\ + D(f,x\)
x\ −UD(g,y\)
Failure!
Living on the Edge, SAHD 2013, Durham, 24 July 2013 23
![Page 24: Living on the Edge - Duke Electrical and Computer Engineeringpeople.ee.duke.edu/~lcarin/SAHD_Tropp.pdf · 2013. 8. 3. · Living on the Edge, SAHD 2013, Durham, 24 July 2013 27. Inference,](https://reader034.vdocuments.site/reader034/viewer/2022051823/5fed82ef3233ee586470e8ab/html5/thumbnails/24.jpg)
Demixing Problems with Random Incoherence
Theorem 3. [Amelunxen, Lotz, McCoy, T 2013] Assume
§ The vectors x\ ∈ Rd and y\ ∈ Rd are unknown
§ The observation z = x\ +Qy\ where Q is random orthogonal
§ The pair (x, y) solves
minimize f(x) subject to g(y) ≤ g(y\)
x+Qy = z
Then
δ(D(f,x\)
)+ δ(D(g,y\)
). d =⇒ (x, y) = (x\,y\) whp
δ(D(f,x\)
)+ δ(D(g,y\)
)& d =⇒ (x, y) 6= (x\,y\) whp
Living on the Edge, SAHD 2013, Durham, 24 July 2013 24
![Page 25: Living on the Edge - Duke Electrical and Computer Engineeringpeople.ee.duke.edu/~lcarin/SAHD_Tropp.pdf · 2013. 8. 3. · Living on the Edge, SAHD 2013, Durham, 24 July 2013 27. Inference,](https://reader034.vdocuments.site/reader034/viewer/2022051823/5fed82ef3233ee586470e8ab/html5/thumbnails/25.jpg)
Sparse + Sparse via `1 + `1 Minimization
0 25 50 75 1000
25
50
75
100
Living on the Edge, SAHD 2013, Durham, 24 July 2013 25
![Page 26: Living on the Edge - Duke Electrical and Computer Engineeringpeople.ee.duke.edu/~lcarin/SAHD_Tropp.pdf · 2013. 8. 3. · Living on the Edge, SAHD 2013, Durham, 24 July 2013 27. Inference,](https://reader034.vdocuments.site/reader034/viewer/2022051823/5fed82ef3233ee586470e8ab/html5/thumbnails/26.jpg)
Low-Rank + Sparse via S1 + `1 Minimization
0 7 14 21 28 350
245
490
735
980
1225
Living on the Edge, SAHD 2013, Durham, 24 July 2013 26
![Page 27: Living on the Edge - Duke Electrical and Computer Engineeringpeople.ee.duke.edu/~lcarin/SAHD_Tropp.pdf · 2013. 8. 3. · Living on the Edge, SAHD 2013, Durham, 24 July 2013 27. Inference,](https://reader034.vdocuments.site/reader034/viewer/2022051823/5fed82ef3233ee586470e8ab/html5/thumbnails/27.jpg)
To learn more...
E-mail: [email protected]
Web: http://users.cms.caltech.edu/~mccoy
http://users.cms.caltech.edu/~jtropp
Papers:
§ MT, “Sharp recovery bounds for convex deconvolution, with applications.” arXiv cs.IT
1205.1580
§ ALMT, “Living on the edge: A geometric theory of phase transitions in convex
optimization.” arXiv cs.IT 1303.6672
§ Oymak & Hassibi, “Asymptotically exact denoising in relation to compressed sensing,”
arXiv cs.IT 1305.2714
§ More to come!
Living on the Edge, SAHD 2013, Durham, 24 July 2013 27
![Page 28: Living on the Edge - Duke Electrical and Computer Engineeringpeople.ee.duke.edu/~lcarin/SAHD_Tropp.pdf · 2013. 8. 3. · Living on the Edge, SAHD 2013, Durham, 24 July 2013 27. Inference,](https://reader034.vdocuments.site/reader034/viewer/2022051823/5fed82ef3233ee586470e8ab/html5/thumbnails/28.jpg)
Inference, Control, & Optimization
New doctoral program, Department of Computing + Mathematical Sciences at Caltech
§ A unified view of inference and decision making, emphasizing methods
that are computationally efficient and theoretically grounded
§ Core faculty: Jim Beck, Venkat Chandrasekaran, John Doyle,
Babak Hassibi, Steven Low, Richard Murray, Houman Owhadi,
Joel Tropp, Adam Wierman
§ Key Research Areas: Signal processing, statistics, optimization,
control, uncertainty quantification, and their applications in science +
engineering
Living on the Edge, SAHD 2013, Durham, 24 July 2013 28