some aspects of the interplay between convex analysis …plc/sestri/combettes2014.pdf · some...
TRANSCRIPT
Some Aspects of the Interplay BetweenConvex Analysis and Monotone Operator
Theory
Patrick L. Combettes
Laboratoire Jacques-Louis LionsFaculte de Mathematiques
Universite Pierre et Marie Curie – Paris 675005 Paris, France
Sestri Levante, 12 settembre 2014
Patrick L. Combettes Convex Functions and Monotone Operators 1/17
Overview
Our objective is to discuss certain aspects of the impor-tance of the theory of monotone operators and of non-expansive operators in the analysis and the numerical so-lution of problems in inverse problems and learning theory,even when those admit a purely variational formulation.
Special emphasis is placed on the role played by duality.
Patrick L. Combettes Convex Functions and Monotone Operators 2/17
Duality in Hilbert spaces
Duality-closed classes of objects:Closed vector subspaces: V → V⊥ (Frechet?)
Closed convex cones: K → K (Fenchel; ↑ : K = V )Lower semicontinuous proper convex functions:f → f ∗ (Fenchel, Moreau; ↑ : f = ιK )Maximally monotone operators: A→ A−1 (Moreau;↑ : A = ∂f )Firmly nonexpansive operators: J → Id− J ↑ : (Minty;l J = (Id + A)−1)Nonexpansive operators: R → −R (l R = 2J − Id)
Patrick L. Combettes Convex Functions and Monotone Operators 3/17
Duality in Hilbert spaces
Duality-closed classes of objects:Closed vector subspaces: V → V⊥ (Frechet?)Closed convex cones: K → K (Fenchel; ↑ : K = V )
Lower semicontinuous proper convex functions:f → f ∗ (Fenchel, Moreau; ↑ : f = ιK )Maximally monotone operators: A→ A−1 (Moreau;↑ : A = ∂f )Firmly nonexpansive operators: J → Id− J ↑ : (Minty;l J = (Id + A)−1)Nonexpansive operators: R → −R (l R = 2J − Id)
Patrick L. Combettes Convex Functions and Monotone Operators 3/17
Duality in Hilbert spaces
Duality-closed classes of objects:Closed vector subspaces: V → V⊥ (Frechet?)Closed convex cones: K → K (Fenchel; ↑ : K = V )Lower semicontinuous proper convex functions:f → f ∗ (Fenchel, Moreau; ↑ : f = ιK )
Maximally monotone operators: A→ A−1 (Moreau;↑ : A = ∂f )Firmly nonexpansive operators: J → Id− J ↑ : (Minty;l J = (Id + A)−1)Nonexpansive operators: R → −R (l R = 2J − Id)
Patrick L. Combettes Convex Functions and Monotone Operators 3/17
Duality in Hilbert spaces
Duality-closed classes of objects:Closed vector subspaces: V → V⊥ (Frechet?)Closed convex cones: K → K (Fenchel; ↑ : K = V )Lower semicontinuous proper convex functions:f → f ∗ (Fenchel, Moreau; ↑ : f = ιK )Maximally monotone operators: A→ A−1 (Moreau;↑ : A = ∂f )
Firmly nonexpansive operators: J → Id− J ↑ : (Minty;l J = (Id + A)−1)Nonexpansive operators: R → −R (l R = 2J − Id)
Patrick L. Combettes Convex Functions and Monotone Operators 3/17
Duality in Hilbert spaces
Duality-closed classes of objects:Closed vector subspaces: V → V⊥ (Frechet?)Closed convex cones: K → K (Fenchel; ↑ : K = V )Lower semicontinuous proper convex functions:f → f ∗ (Fenchel, Moreau; ↑ : f = ιK )Maximally monotone operators: A→ A−1 (Moreau;↑ : A = ∂f )Firmly nonexpansive operators: J → Id− J ↑ : (Minty;l J = (Id + A)−1)
Nonexpansive operators: R → −R (l R = 2J − Id)
Patrick L. Combettes Convex Functions and Monotone Operators 3/17
Duality in Hilbert spaces
Duality-closed classes of objects:Closed vector subspaces: V → V⊥ (Frechet?)Closed convex cones: K → K (Fenchel; ↑ : K = V )Lower semicontinuous proper convex functions:f → f ∗ (Fenchel, Moreau; ↑ : f = ιK )Maximally monotone operators: A→ A−1 (Moreau;↑ : A = ∂f )Firmly nonexpansive operators: J → Id− J ↑ : (Minty;l J = (Id + A)−1)Nonexpansive operators: R → −R (l R = 2J − Id)
Patrick L. Combettes Convex Functions and Monotone Operators 3/17
Problems in duality
Functional setting (Fenchel, Moreau): f + g → f ∗∨ + g∗
Composite functional setting (Rockafellar):
f + g ◦ L→ f ∗∨ ◦ L∗ + g∗
Maximally monotone operator setting (Mosco 1972(for V.I.), Mercier 1980, Attouch-Thera 1996):
A + B ↔ −A−1 ◦ (−Id) + B−1.
Composite maximally monotone operator setting (Robin-son 1999, Pennanen 2000):
A + L∗BL↔ −LA−1(−L∗) + B−1
Open problem 1: (Firmly) nonexpansive operator set-ting?
Patrick L. Combettes Convex Functions and Monotone Operators 4/17
Problems in duality
Functional setting (Fenchel, Moreau): f + g → f ∗∨ + g∗
Composite functional setting (Rockafellar):
f + g ◦ L→ f ∗∨ ◦ L∗ + g∗
Maximally monotone operator setting (Mosco 1972(for V.I.), Mercier 1980, Attouch-Thera 1996):
A + B ↔ −A−1 ◦ (−Id) + B−1.
Composite maximally monotone operator setting (Robin-son 1999, Pennanen 2000):
A + L∗BL↔ −LA−1(−L∗) + B−1
Open problem 1: (Firmly) nonexpansive operator set-ting?
Patrick L. Combettes Convex Functions and Monotone Operators 4/17
Problems in duality
Functional setting (Fenchel, Moreau): f + g → f ∗∨ + g∗
Composite functional setting (Rockafellar):
f + g ◦ L→ f ∗∨ ◦ L∗ + g∗
Maximally monotone operator setting (Mosco 1972(for V.I.), Mercier 1980, Attouch-Thera 1996):
A + B ↔ −A−1 ◦ (−Id) + B−1.
Composite maximally monotone operator setting (Robin-son 1999, Pennanen 2000):
A + L∗BL↔ −LA−1(−L∗) + B−1
Open problem 1: (Firmly) nonexpansive operator set-ting?
Patrick L. Combettes Convex Functions and Monotone Operators 4/17
Problems in duality
Functional setting (Fenchel, Moreau): f + g → f ∗∨ + g∗
Composite functional setting (Rockafellar):
f + g ◦ L→ f ∗∨ ◦ L∗ + g∗
Maximally monotone operator setting (Mosco 1972(for V.I.), Mercier 1980, Attouch-Thera 1996):
A + B ↔ −A−1 ◦ (−Id) + B−1.
Composite maximally monotone operator setting (Robin-son 1999, Pennanen 2000):
A + L∗BL↔ −LA−1(−L∗) + B−1
Open problem 1: (Firmly) nonexpansive operator set-ting?
Patrick L. Combettes Convex Functions and Monotone Operators 4/17
Problems in duality
Functional setting (Fenchel, Moreau): f + g → f ∗∨ + g∗
Composite functional setting (Rockafellar):
f + g ◦ L→ f ∗∨ ◦ L∗ + g∗
Maximally monotone operator setting (Mosco 1972(for V.I.), Mercier 1980, Attouch-Thera 1996):
A + B ↔ −A−1 ◦ (−Id) + B−1.
Composite maximally monotone operator setting (Robin-son 1999, Pennanen 2000):
A + L∗BL↔ −LA−1(−L∗) + B−1
Open problem 1: (Firmly) nonexpansive operator set-ting?
Patrick L. Combettes Convex Functions and Monotone Operators 4/17
Open problem 1: Duality for nonexpansiveoperators?
Consider the inclusion 0 ∈ Ax + Bx , call P its solution setand D the dual solution set. Then:
P ={
x ∈ H | (∃u ∈ D) − u ∈ Ax and u ∈ Bx}
andD =
{u ∈ H | (∃ x ∈P) x ∈ A−1(−u) and x ∈ B−1u
}P = dom (A ∩ (−B)) and D = dom (A−1 ◦ (−Id) ∩ B−1)
Suppose that A and B are maximally monotone, setTA,B = JA(2JB − Id) + Id− JB, and let γ > 0. Then:
TA,B is firmly nonexpansiveP = JγB
(Fix TγA,γB
)D = γB
(Fix TγA,γB
)P 6= Ø⇔ D 6= Ø⇔ Fix TγA,γB 6= Ø
Patrick L. Combettes Convex Functions and Monotone Operators 5/17
Open problem 2: True Fenchel duality formonotone operators?
Suppose that 0 ∈ sri(dom f − dom g). Then
inf(f + g)(H) = −min(f∨∗ + g∗)(H)
We always have
zer(A + B) 6= Ø ⇔ zer(−A−1 ◦ (−Id) + B−1) 6= Ø.
For suitable “closure” or “enlargement” operations,can
zer(A + B
)\ 6= Ø ⇒ zer(− A−1 ◦ (−Id) + B−1)[ 6= Ø
recover Fenchel duality?
Patrick L. Combettes Convex Functions and Monotone Operators 6/17
Open problem 2: True Fenchel duality formonotone operators?
Suppose that 0 ∈ sri(dom f − dom g). Then
inf(f + g)(H) = −min(f∨∗ + g∗)(H)
We always have
zer(A + B) 6= Ø ⇔ zer(−A−1 ◦ (−Id) + B−1) 6= Ø.
For suitable “closure” or “enlargement” operations,can
zer(A + B
)\ 6= Ø ⇒ zer(− A−1 ◦ (−Id) + B−1)[ 6= Ø
recover Fenchel duality?
Patrick L. Combettes Convex Functions and Monotone Operators 6/17
Open problem 2: True Fenchel duality formonotone operators?
Suppose that 0 ∈ sri(dom f − dom g). Then
inf(f + g)(H) = −min(f∨∗ + g∗)(H)
We always have
zer(A + B) 6= Ø ⇔ zer(−A−1 ◦ (−Id) + B−1) 6= Ø.
For suitable “closure” or “enlargement” operations,can
zer(A + B
)\ 6= Ø ⇒ zer(− A−1 ◦ (−Id) + B−1)[ 6= Ø
recover Fenchel duality?
Patrick L. Combettes Convex Functions and Monotone Operators 6/17
Splitting methods: Quick overview
Traditional splitting techniques were developed in thelate 1970s for inclusions involving the sum of two maxi-mally monotone operators:
0 ∈ Ax + Bx
Open problem 3: The continuous-time dynamics lead-ing to FB are reasonably understood (−x ′ ∈ Ax + Bx);what about FBF and DR?Main algorithms:
Forward-backward method (Mercier, 1979)Douglas-Rachford method (Lions and Mercier, 1979)Forward-backward-forward algorithm (Tseng, 2000).
Until recently, general splitting methods lacked for thecomposite problem
0 ∈ Ax + L∗BLx
Patrick L. Combettes Convex Functions and Monotone Operators 7/17
Splitting methods: Composite problems
L : H → G linear and bounded, A : H → 2H, B : G → 2G
maximally monotoneSolve
0 ∈ Ax + L∗BLx
Main issue: 3 objects (A,B, L) to split... and a binaryrelation ∈ binding themWe need to reduce a problem to a 2-object problemin a larger spaceKey: recast the problem in the primal dual space
Patrick L. Combettes Convex Functions and Monotone Operators 8/17
Kuhn-Tucker set of a composite inclusion
Primal solutions: P ={
x ∈ H | 0 ∈ Ax + L∗BLx}
Dual solutions: D ={
v∗ ∈ G | 0 ∈ −L ◦ A−1(−L∗v∗) + B−1v∗}
Kuhn-Tucker set
Z ={
(x , v∗) ∈ H ⊕ G | −L∗v∗ ∈ Ax and Lx ∈ B−1v∗}
Z is a closed convex setZ 6= Ø⇔P 6= Ø⇔ D 6= ØZ ⊂P ×D
Patrick L. Combettes Convex Functions and Monotone Operators 9/17
Kuhn-Tucker set of a composite inclusion
Primal solutions: P ={
x ∈ H | 0 ∈ Ax + L∗BLx}
Dual solutions: D ={
v∗ ∈ G | 0 ∈ −L ◦ A−1(−L∗v∗) + B−1v∗}
Kuhn-Tucker set
Z ={
(x , v∗) ∈ H ⊕ G | −L∗v∗ ∈ Ax and Lx ∈ B−1v∗}
Z is a closed convex setZ 6= Ø⇔P 6= Ø⇔ D 6= ØZ ⊂P ×D
Patrick L. Combettes Convex Functions and Monotone Operators 9/17
Kuhn-Tucker set of a composite inclusion
Primal solutions: P ={
x ∈ H | 0 ∈ Ax + L∗BLx}
Dual solutions: D ={
v∗ ∈ G | 0 ∈ −L ◦ A−1(−L∗v∗) + B−1v∗}
Kuhn-Tucker set
Z ={
(x , v∗) ∈ H ⊕ G | −L∗v∗ ∈ Ax and Lx ∈ B−1v∗}
Z is a closed convex setZ 6= Ø⇔P 6= Ø⇔ D 6= ØZ ⊂P ×D
Patrick L. Combettes Convex Functions and Monotone Operators 9/17
Kuhn-Tucker set of a composite inclusion
Strategy: find a Kuhn-Tucker pair (x , v∗) by apply-ing a standard (e.g., Douglas-Rachford or forward-backward-forward) method to a monotone+skew de-composition of the problem in H⊕ G.[
00
]∈[A 00 B−1
]︸ ︷︷ ︸
M
[xv∗
]+
[0 L∗
−L 0
]︸ ︷︷ ︸
S
[xv∗
]
L. M. Briceno-Arias and PLC, A monotone+skew splittingmodel for composite monotone inclusions in duality, SIAM J.Optim., vol. 21, 2011.
Even in minimization problems, such a framework can-not be reduced to a functional setting: monotone op-erator splitting is required!Possible limitation: linear inversions (in DR) or knowl-edge of ‖L‖ (in FBF) are necessary
Patrick L. Combettes Convex Functions and Monotone Operators 10/17
Kuhn-Tucker set of a composite inclusion
Strategy: find a Kuhn-Tucker pair (x , v∗) by apply-ing a standard (e.g., Douglas-Rachford or forward-backward-forward) method to a monotone+skew de-composition of the problem in H⊕ G.[
00
]∈[A 00 B−1
]︸ ︷︷ ︸
M
[xv∗
]+
[0 L∗
−L 0
]︸ ︷︷ ︸
S
[xv∗
]
L. M. Briceno-Arias and PLC, A monotone+skew splittingmodel for composite monotone inclusions in duality, SIAM J.Optim., vol. 21, 2011.
Even in minimization problems, such a framework can-not be reduced to a functional setting: monotone op-erator splitting is required!Possible limitation: linear inversions (in DR) or knowl-edge of ‖L‖ (in FBF) are necessary
Patrick L. Combettes Convex Functions and Monotone Operators 10/17
Revisiting the proximal point algorithm
A maximally monotone, (γn)n∈N ∈ ]0,+∞[N, zer A 6= Ø,
xn+1 = JγnAxn
Classical results by Brezis&Lions (1978):If∑
n∈N γ2n = +∞, then xn ⇀ x ∈ zer A
If A = ∂f (f ∈ Γ0(H)) and∑
n∈N γn = +∞, then xn ⇀ x ∈zer A = Argmin f
Is the proximal point algorithm of any use?
Patrick L. Combettes Convex Functions and Monotone Operators 11/17
Revisiting the proximal point algorithm
A : H → 2H maximally monotone, V be a closed vec-tor subspace of HAV : partial inverse of A w.r.t. V (Spingarn, 1983)
gra AV ={
(PV x + PV⊥u,PV u + PV⊥x) | (x,u) ∈ gra A}
Then (Spingarn, 1983):AV is maximally monotonez ∈ zer AV ⇔ (PV z,PV⊥z) ∈ gra Ap = JAV z ⇔ PV p + PV⊥(z − p) = JAz
Aside – Open problem 4: Let f ∈ Γ0(H) and V = {0}.Then (∂f )V = ∂f ∗, but how to define a partial conju-gate of f ? (in general (∂f )V is not a subdifferential)
Patrick L. Combettes Convex Functions and Monotone Operators 12/17
Revisiting the proximal point algorithm
A : H → 2H maximally monotone, V be a closed vec-tor subspace of HAV : partial inverse of A w.r.t. V (Spingarn, 1983)
gra AV ={
(PV x + PV⊥u,PV u + PV⊥x) | (x,u) ∈ gra A}
Then (Spingarn, 1983):AV is maximally monotonez ∈ zer AV ⇔ (PV z,PV⊥z) ∈ gra Ap = JAV z ⇔ PV p + PV⊥(z − p) = JAz
Aside – Open problem 4: Let f ∈ Γ0(H) and V = {0}.Then (∂f )V = ∂f ∗, but how to define a partial conju-gate of f ? (in general (∂f )V is not a subdifferential)
Patrick L. Combettes Convex Functions and Monotone Operators 12/17
Revisiting the proximal point algorithm
Back to the Kuhn-Tucker set
Z ={
(x , v∗) ∈ H ⊕ G | −L∗v∗ ∈ Ax and Lx ∈ B−1v∗}
Define H = H⊕ G, A : (x , y) 7→ Ax × By , andV =
{(x , y) ∈ H ⊕ G | Lx = y
}Apply the (monotone operator) proximal point algo-rithm to AV
M. A. Alghamdi, A. Alotaibi, PLC, and N. Shahzad, A primal-dual method of partial inverses for composite inclusions, Op-tim. Lett., March 2014.
Patrick L. Combettes Convex Functions and Monotone Operators 13/17
Revisiting the proximal point algorithm
TheoremSet Q = (Id + L∗L)−1 and assume that zer(A + L∗BL) 6= Ø. Let(λn)n∈N ∈ ]0, 2[
N, such that∑
n∈N λn(2 − λn) = +∞, let x0 ∈ H,v∗0 ∈ G, and set y0 = Lx0, u0 = −L∗v∗0 , and
(∀n ∈ N)
pn ≈ JA(xn + un)qn ≈ JB(yn + v∗n )rn = xn + un − pnsn = yn + vn − qntn = Q(rn + L∗sn)wn = Q(pn + L∗qn)xn+1 = xn − λntnyn+1 = yn − λnLtnun+1 = un + λn(wn − pn)v∗n+1 = v∗n + λn(Lwn − qn).
Then xn−wn → 0, yn−Lwn → 0, un−rn+tn → 0, and v∗n−sn+Ltn → 0,(xn, v∗n )⇀ (x , v∗) ∈ K .
Patrick L. Combettes Convex Functions and Monotone Operators 14/17
A strongly convergent splitting method
We have seen that the Kuhn-Tucker set
Z ={
(x , v∗) ∈ H ⊕ G | −L∗v∗ ∈ Ax and Lx ∈ B−1v∗}
is a closed and convexThe previous methods require efficient linear inversionsschmes or knowledge of ‖L‖, neither of which may beavailable in many situations (e.g., domain decompo-sition methods). In addition they require stringent ad-ditional hypotheses to guarantee strong convergence(to an unspecified Kuhn-Tucker point)We present a method which alleviates all these limita-tions.
Aside – Open problem 5: in infinite-dimensional prob-lems is weak convergence relevant?
Patrick L. Combettes Convex Functions and Monotone Operators 15/17
A strongly convergent splitting method
We have seen that the Kuhn-Tucker set
Z ={
(x , v∗) ∈ H ⊕ G | −L∗v∗ ∈ Ax and Lx ∈ B−1v∗}
is a closed and convexThe previous methods require efficient linear inversionsschmes or knowledge of ‖L‖, neither of which may beavailable in many situations (e.g., domain decompo-sition methods). In addition they require stringent ad-ditional hypotheses to guarantee strong convergence(to an unspecified Kuhn-Tucker point)We present a method which alleviates all these limita-tions.Aside – Open problem 5: in infinite-dimensional prob-lems is weak convergence relevant?
Patrick L. Combettes Convex Functions and Monotone Operators 15/17
A strongly convergent splitting method
An abstract Haugazeau scheme:Let C be a nonempty closed convex subset of H andlet x0 ∈H. Iterate
for n = 0, 1, . . .⌊take xn+1/2 ∈H such that C ⊂ H(xn,xn+1/2)xn+1 = P
H(x0,xn)∩H(xn,xn+1/2
)x0
Suppose that, for every x ∈ H and every strictly in-creasing sequence (kn)n∈N in N, xkn ⇀ x ⇒ x ∈ C . Then(xn)n∈N is well defined and xn → PCx0.Strategy: Apply this principle to C = Z , construct xn+1/2by suitably choosing points in gra A and gra B
A. Alotaibi, PLC, and N. Shahzad, Best approximation fromthe Kuhn-Tucker set of composite monotone inclusions, J.Nonlinear Convex Anal., to appear.
Patrick L. Combettes Convex Functions and Monotone Operators 16/17
A strongly convergent splitting method
TheoremFix (x0, v∗0 ) ∈ H× G and ε ∈ ]0, 1[, and iterate for every n ∈ N
(γn, µn) ∈ [ε, 1/ε]2
an = JγnA(xn − γnL∗v∗n ), ln = Lxn, bn = JµnB(ln + µnv∗n )s∗n = γ−1
n (xn − an) + µ−1n L∗(ln − bn), tn = bn − Lan
τn = ‖s∗n‖2 + ‖tn‖2, λn ∈ [ε, 1]θn = λn
(γ−1
n ‖xn − an‖2 + µ−1n ‖ln − bn‖2)/τn
xn+1/2 = xn − θns∗n , v∗n+1/2 = v∗n − θntn
χn =⟨x0 − xn | xn − xn+1/2
⟩+⟨v∗0 − v∗n | v∗n − v∗n+1/2
⟩µn = ‖x0 − xn‖2 + ‖v∗0 − v∗n ‖2, νn = ‖xn − xn+1/2‖2 + ‖v∗n − v∗n+1/2‖2
ρn = µnνn − χ2n
if ρn > 0 and χnνn > ρn⌊xn+1 = x0 + (1 + χn/νn)(xn+1/2 − xn)v∗n+1 = v∗0 + (1 + χn/νn)(v∗n+1/2 − v∗n )
if[ ETC... easy steps]
Then (xn, v∗n )→ PK (x0, v∗0 )
Patrick L. Combettes Convex Functions and Monotone Operators 17/17