the most important concept in optimization (minimization)
DESCRIPTION
The Most Important Concept in Optimization (minimization). A point is said to be an optimal solution of a unconstrained minimization if there exists no decent direction. A point is said to be an optimal solution of a constrained minimization if there exists no - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: The Most Important Concept in Optimization (minimization)](https://reader036.vdocuments.site/reader036/viewer/2022062422/56812bb1550346895d8fed9c/html5/thumbnails/1.jpg)
The Most Important Concept in Optimization (minimization)
A point is said to be an optimal solution of a unconstrained minimization if there exists no decent direction
A point is said to be an optimal solution of a constrained minimization if there exists no feasible decent direction
There might exist decent direction but move along this direction will leave out the feasible region
![Page 2: The Most Important Concept in Optimization (minimization)](https://reader036.vdocuments.site/reader036/viewer/2022062422/56812bb1550346895d8fed9c/html5/thumbnails/2.jpg)
Decent Direction of
Move alone the decent direction for a certain stepsize will decrease the objective function value i.e.,
f (x0+ õd) < f (x0); õ 2 (0; î )
d 2 Rn is descent direction if9 î > 0; such that
r f (x0)d < 0
x0
![Page 3: The Most Important Concept in Optimization (minimization)](https://reader036.vdocuments.site/reader036/viewer/2022062422/56812bb1550346895d8fed9c/html5/thumbnails/3.jpg)
Feasible Direction of
Move alone the feasible direction from for a certain stepsize will not leave the feasible region i.e.,
x0
x0
(x0+ õd) 2 F; õ 2 (0; î )
d 2 Rn is f easible direction of x0 2 F
if 9 î > 0; such that
Fwhere is the feasible region.
![Page 4: The Most Important Concept in Optimization (minimization)](https://reader036.vdocuments.site/reader036/viewer/2022062422/56812bb1550346895d8fed9c/html5/thumbnails/4.jpg)
minx2R2
(x1à 3)2+ x22
minx2R3
(x1à 2)4+ x21x
22+ (x2à 1)2+ (x1+ x3)2
![Page 5: The Most Important Concept in Optimization (minimization)](https://reader036.vdocuments.site/reader036/viewer/2022062422/56812bb1550346895d8fed9c/html5/thumbnails/5.jpg)
Steep Decent with Exact Line Search
Start with any
x0 2 Rn . Having xi 2 Rn
stop if r f (xi) = 0(i) Steep Decent Direction :
di = à r f(xi)0
(ii) Finding Stepsize by Exact Line Search :
õã 2 argminõ>0
f (xi + õdi)
xi+1 = xi + õãdi
![Page 6: The Most Important Concept in Optimization (minimization)](https://reader036.vdocuments.site/reader036/viewer/2022062422/56812bb1550346895d8fed9c/html5/thumbnails/6.jpg)
Minimum Principle
Let f : Rn ! Rbe a convex and differentiable function
F ò Rnbe the feasible region.
xã 2 argminx2F
f (x) ( ) r f (xã)(x à xã) > 0 8x 2 F
Example:min (x à 1)2 s:t: a ô x ô b
![Page 7: The Most Important Concept in Optimization (minimization)](https://reader036.vdocuments.site/reader036/viewer/2022062422/56812bb1550346895d8fed9c/html5/thumbnails/7.jpg)
minx2R2
x21+ x2
2
x1+ x264à x1à x2 6 à 2
x1; x2>0
ï
ï
ï ï
![Page 8: The Most Important Concept in Optimization (minimization)](https://reader036.vdocuments.site/reader036/viewer/2022062422/56812bb1550346895d8fed9c/html5/thumbnails/8.jpg)
Minimization Problem vs.
Kuhn-Tucker Stationary-point Problem
r f (x) + ë0r g(x) = 0
g(x)60;
ë0g(x) = 0
ë > 0
Find x 2 Ò; ë 2 Rmsuch thatKTSP:
minx2Ò
f (x)MP: such that
g(x)60
![Page 9: The Most Important Concept in Optimization (minimization)](https://reader036.vdocuments.site/reader036/viewer/2022062422/56812bb1550346895d8fed9c/html5/thumbnails/9.jpg)
Lagrangian Function
For a fixed
L(x;ë) = f (x) + ë0g(x)
Let L(x;ë) = f (x) + ë0g(x) and ë > 0 If f (x);g(x)are convex thenL (x;ë)is convex.
ë > 0, if x 2 argminf L (x;ë)j x 2 Rngthen
@x@L(x;ë)
ìììx=x
= r f (x) +ë0r g(x) = 0
Above result is a sufficient condition ifL (x;ë)
is convex.
![Page 10: The Most Important Concept in Optimization (minimization)](https://reader036.vdocuments.site/reader036/viewer/2022062422/56812bb1550346895d8fed9c/html5/thumbnails/10.jpg)
KTSP with Equality Constraints?
(Assumeh(x) = 0are linear functions)
h(x) = 0 , h(x)60 and à h(x)60
KTSP:
r f (x) + ë0r g(x) + (ì + à ì à )0r h
g(x)60;
ë0g(x) = 0;
ë>;
Find x 2 Ò; ë 2 Rk; ì +; ì à 2 Rmsuch that
(x) = 0
(ì +)0h(x) = 0; (ì à )0(à h(x)) = 0
h(x) = 0
ì +; ì à > 0
![Page 11: The Most Important Concept in Optimization (minimization)](https://reader036.vdocuments.site/reader036/viewer/2022062422/56812bb1550346895d8fed9c/html5/thumbnails/11.jpg)
KTSP with Equality Constraints
KTSP: Find x 2 Ò; ë 2 Rk; ì 2 Rm such that
r f (x) + ë0r g(x) + ì r h
g(x)60;ë0g(x) = 0;
ë>0
(x) = 0
h(x) = 0
If ì = ì + à ì à and ì +; ì à > 0 then
ì is free variable
![Page 12: The Most Important Concept in Optimization (minimization)](https://reader036.vdocuments.site/reader036/viewer/2022062422/56812bb1550346895d8fed9c/html5/thumbnails/12.jpg)
Generalized Lagrangian FunctionL(x;ë; ì ) = f (x) + ë0g(x) + ì 0h(x)
For fixed ë>0;ì , if x 2 argminf L (x;ë;ì )j x 2 Rngthen
Let and ë > 0L(x;ë; ì ) = f (x) + ë0g(x) + ì 0h(x)
L (x;ë;ì )
If f (x);g(x)are convex and is linear thenh(x)is convex.
@x@L(x;ë;ì )
ìììx=x
= r f (x) +ë0r g(x) + ì 0r h(x) = 0
Above result is a sufficient condition if
is convex.
L (x;ë;ì )
![Page 13: The Most Important Concept in Optimization (minimization)](https://reader036.vdocuments.site/reader036/viewer/2022062422/56812bb1550346895d8fed9c/html5/thumbnails/13.jpg)
Lagrangian Dual Problem
maxë;ì
minx2Ò
L(x;ë; ì )
subject to ë > 0
![Page 14: The Most Important Concept in Optimization (minimization)](https://reader036.vdocuments.site/reader036/viewer/2022062422/56812bb1550346895d8fed9c/html5/thumbnails/14.jpg)
Lagrangian Dual Problem
maxë;ì
minx2Ò
L(x;ë; ì )
subject to ë > 0
maxë;ì
ò(ë; ì )
subject to ë > 0where ò(ë; ì ) = inf
x2ÒL(x;ë; ì )
![Page 15: The Most Important Concept in Optimization (minimization)](https://reader036.vdocuments.site/reader036/viewer/2022062422/56812bb1550346895d8fed9c/html5/thumbnails/15.jpg)
Weak Duality Theorem
Let x 2 Òbe a feasible solution of the primal
problem and(ë; ì )a feasible solution of the
dual problem. Then f (x)>ò(ë; ì )
Corollary: supfò(ë; ì )j ë>0g
6 inff f (x)j g(x) 6 0; h(x) = 0g
ò(ë; ì ) = infx2Ò
L(x;ë; ì ) ô L (xà;ë; ì )
![Page 16: The Most Important Concept in Optimization (minimization)](https://reader036.vdocuments.site/reader036/viewer/2022062422/56812bb1550346895d8fed9c/html5/thumbnails/16.jpg)
Weak Duality Theorem
06ë ? g(x)60
Corollary: ëã>0If f (xã) = ò(ëã; ì ã)where
and g(xã)60;h(xã) = 0 , then xã and(ëã; ì ã)
solve the primal and dual problem respectively.In this case,
![Page 17: The Most Important Concept in Optimization (minimization)](https://reader036.vdocuments.site/reader036/viewer/2022062422/56812bb1550346895d8fed9c/html5/thumbnails/17.jpg)
Saddle Point of Lagrangian
Let xã 2 Ò;ëã>0; ì ã 2 Rmsatisfying
L (xã;ë; ì )6L (xã;ëã; ì ã) 6L(x;ëã; ì ã);
8 x 2 Ò; ë>0: Then (xã;ëã; ì ã) is called
The saddle point of the Lagrangian function
![Page 18: The Most Important Concept in Optimization (minimization)](https://reader036.vdocuments.site/reader036/viewer/2022062422/56812bb1550346895d8fed9c/html5/thumbnails/18.jpg)
Dual Problem of Linear Program
minx2R n
p0x
subject to Ax > b; x>0
Primal LP
Dual LP maxë2R m
b0ë
subject to A0ë6p; ë>0
※ All duality theorems hold and work perfectly!
![Page 19: The Most Important Concept in Optimization (minimization)](https://reader036.vdocuments.site/reader036/viewer/2022062422/56812bb1550346895d8fed9c/html5/thumbnails/19.jpg)
Application of LP Duality (I) Farkas’ Lemma
For any matrixA 2 Rmâ nand any vectorb2 Rm;either
Ax60; b0x > 0 has a solution
or
A0ë = b; ë>0 has a solution
but never both.
![Page 20: The Most Important Concept in Optimization (minimization)](https://reader036.vdocuments.site/reader036/viewer/2022062422/56812bb1550346895d8fed9c/html5/thumbnails/20.jpg)
Application of LP Duality (II) LSQ-Normal Equation Always Has a Solution
For any matrixA 2 Rmâ nand any vectorb2 Rm;
consider minx2R n
jjAx à bjj22
xã 2 argminf jjAx à bjj22g , A0Axã = A0b
Claim: A0Ax = A0balways has a solution.
![Page 21: The Most Important Concept in Optimization (minimization)](https://reader036.vdocuments.site/reader036/viewer/2022062422/56812bb1550346895d8fed9c/html5/thumbnails/21.jpg)
Dual Problem of Strictly Convex Quadratic Program
minx2R n
21x0Qx + p0x
subject to Ax6 b
Primal QP
With strictly convex assumption, we have
Dual QP
max à 21(p0+ ë0A)Qà 1(A0ë + p) à ë0b
subject to ë>0