introduction to unconstrained optimization - direct search
TRANSCRIPT
![Page 1: Introduction to unconstrained optimization - direct search](https://reader030.vdocuments.site/reader030/viewer/2022020917/61bd78a885f9f97d9a6a28b9/html5/thumbnails/1.jpg)
Introduction to
unconstrained optimization
- direct search methods
Jussi Hakanen
Post-doctoral researcher [email protected]
spring 2014 TIES483 Nonlinear optimization
![Page 2: Introduction to unconstrained optimization - direct search](https://reader030.vdocuments.site/reader030/viewer/2022020917/61bd78a885f9f97d9a6a28b9/html5/thumbnails/2.jpg)
Structure of optimization methods
Typically โ Constraint handling
converts the problem to (a series of) unconstrained problems
โ In unconstrained optimization a search direction is determined at each iteration
โ The best solution in the search direction is found with line search
spring 2014 TIES483 Nonlinear optimization
Constraint handling
method
Unconstrained
optimization
Line
search
![Page 3: Introduction to unconstrained optimization - direct search](https://reader030.vdocuments.site/reader030/viewer/2022020917/61bd78a885f9f97d9a6a28b9/html5/thumbnails/3.jpg)
Group discussion
1. What kind of optimality conditions there exist for
unconstrained optimization (๐ฅ โ ๐ ๐)?
2. List methods for unconstrained optimization?
โ what are their general ideas?
Discuss in small groups (3-4) for 15-20 minutes
Each group has a secretary who writes down the
answers of the group
At the end, we summarize what each group found
spring 2014 TIES483 Nonlinear optimization
![Page 4: Introduction to unconstrained optimization - direct search](https://reader030.vdocuments.site/reader030/viewer/2022020917/61bd78a885f9f97d9a6a28b9/html5/thumbnails/4.jpg)
Reminder: gradient and hessian Definition: If function ๐: ๐ ๐ โ ๐ is differentiable, then the
gradient ๐ป๐(๐ฅ) consists of the partial derivatives ๐๐(๐ฅ)
๐๐ฅ๐ i.e.
๐ป๐ ๐ฅ =๐๐(๐ฅ)
๐๐ฅ1, โฆ ,
๐๐(๐ฅ)
๐๐ฅ๐
๐
Definition: If ๐ is twice differentiable, then the matrix
๐ป ๐ฅ =
๐2๐(๐ฅ)
๐๐ฅ1๐๐ฅ1โฏ
๐2๐(๐ฅ)
๐๐ฅ1๐๐ฅ๐
โฎ โฑ โฎ๐2๐(๐ฅ)
๐๐ฅ๐๐๐ฅ1โฏ
๐2๐(๐ฅ)
๐๐ฅ๐๐๐ฅ๐
is called the Hessian (matrix) of ๐ at ๐ฅ
Result: If ๐ is twice continuously differentiable, then ๐2๐(๐ฅ)
๐๐ฅ๐๐๐ฅ๐=
๐2๐(๐ฅ)
๐๐ฅ๐๐๐ฅ๐
spring 2014 TIES483 Nonlinear optimization
![Page 5: Introduction to unconstrained optimization - direct search](https://reader030.vdocuments.site/reader030/viewer/2022020917/61bd78a885f9f97d9a6a28b9/html5/thumbnails/5.jpg)
Reminder: Definite Matrices
Definition: A symmetric ๐ ร ๐ matrix ๐ป is positive semidefinite if โ ๐ฅ โ โ๐
๐ฅ๐๐ป๐ฅ โฅ 0.
Definition: A symmetric ๐ ร ๐ matrix ๐ป is positive definite if
๐ฅ๐๐ป๐ฅ > 0 โ 0 โ ๐ฅ โ โ๐
Note: If โฅ โ โค (> โ <), then ๐ป is negative semidefinite (definite). If ๐ป is neither positive nor negative semidefinite, then it is indefinite.
Result: Let โ โ ๐ โ ๐ ๐ be open convex set and ๐: ๐ โ ๐ twice differentiable in ๐. Function ๐ is convex if and only if ๐ป(๐ฅโ) is positive semidefinite for all ๐ฅโ โ ๐.
![Page 6: Introduction to unconstrained optimization - direct search](https://reader030.vdocuments.site/reader030/viewer/2022020917/61bd78a885f9f97d9a6a28b9/html5/thumbnails/6.jpg)
Unconstraint problem
min ๐ ๐ฅ , ๐ . ๐ก. ๐ฅ โ ๐ ๐
Necessary conditions: Let ๐ be twice differentiable in ๐ฅโ. If ๐ฅโ is a local minimizer, then
โ ๐ป๐ ๐ฅโ = 0 (that is, ๐ฅโ is a critical point of ๐) and
โ ๐ป ๐ฅโ is positive semidefinite.
Sufficient conditions: Let ๐ be twice differentiable in ๐ฅโ. If
โ ๐ป๐ ๐ฅโ = 0 and
โ ๐ป(๐ฅโ) is positive definite,
then ๐ฅโ is a strict local minimizer.
Result: Let ๐: ๐ ๐ โ ๐ is twice differentiable in ๐ฅโ. If ๐ป๐ ๐ฅโ = 0 and ๐ป(๐ฅโ) is indefinite, then ๐ฅโ is a saddle point.
![Page 7: Introduction to unconstrained optimization - direct search](https://reader030.vdocuments.site/reader030/viewer/2022020917/61bd78a885f9f97d9a6a28b9/html5/thumbnails/7.jpg)
Unconstraint problem
Adopted from Prof. L.T. Biegler (Carnegie Mellon
University)
![Page 8: Introduction to unconstrained optimization - direct search](https://reader030.vdocuments.site/reader030/viewer/2022020917/61bd78a885f9f97d9a6a28b9/html5/thumbnails/8.jpg)
Descent direction
Definition: Let ๐: ๐ ๐ โ ๐ . A vector ๐ โ ๐ ๐ is a descent
direction for ๐ in ๐ฅโ โ ๐ ๐ if โ ๐ฟ > 0 s.t.
๐ ๐ฅโ + ๐๐ < ๐(๐ฅโ) โ ๐ โ (0, ๐ฟ].
Result: Let ๐: ๐ ๐ โ ๐ be differentiable in ๐ฅโ. If โ๐ โ ๐ ๐
s.t. ๐ป๐ ๐ฅโ ๐๐ < 0 then ๐ is a descent direction for ๐ in
๐ฅโ.
spring 2014 TIES483 Nonlinear optimization
![Page 9: Introduction to unconstrained optimization - direct search](https://reader030.vdocuments.site/reader030/viewer/2022020917/61bd78a885f9f97d9a6a28b9/html5/thumbnails/9.jpg)
Model algorithm for unconstrained
minimization
Let ๐ฅโ be the current estimate for ๐ฅโ
1) [Test for convergence.] If conditions are satisfied, stop. The solution is ๐ฅโ.
2) [Compute a search direction.] Compute a non-zero vector ๐โ โ ๐ ๐ which is the search direction.
3) [Compute a step length.] Compute ๐ผโ > 0, the step length, for which it holds that ๐ ๐ฅโ + ๐ผโ๐โ < ๐(๐ฅโ).
4) [Update the estimate for minimum.] Set ๐ฅโ+1 = ๐ฅโ + ๐ผโ๐โ, โ = โ + 1 and go to step 1.
spring 2014 TIES483 Nonlinear optimization
From Gill et al., Practical Optimization, 1981, Academic Press
![Page 10: Introduction to unconstrained optimization - direct search](https://reader030.vdocuments.site/reader030/viewer/2022020917/61bd78a885f9f97d9a6a28b9/html5/thumbnails/10.jpg)
On convergence
Iterative method: a sequence {๐ฅโ} s.t. ๐ฅโ โ ๐ฅโ when โ โ โ
Definition: A method converges โ linearly if โ๐ผ โ [0,1) and ๐ โฅ 0 s.t. โ โ โฅ ๐
๐ฅโ+1 โ ๐ฅโ โค ๐ผ ๐ฅโ โ ๐ฅโ ,
โ superlinearly if โ๐ โฅ 0 and for some sequence ๐ผโ โ 0 it holds that โโ โฅ ๐
๐ฅโ+1 โ ๐ฅโ โค ๐ผโ ๐ฅโ โ ๐ฅโ ,
โ with degree ๐ if โ๐ผ โฅ 0, ๐ > 0 and ๐ โฅ 0 s.t. โโ โฅ ๐ ๐ฅโ+1 โ ๐ฅโ โค ๐ผ ๐ฅโ โ ๐ฅโ ๐
.
If ๐ = 2 (๐ = 3), the convergence is quadratic (cubic).
spring 2014 TIES483 Nonlinear optimization
![Page 11: Introduction to unconstrained optimization - direct search](https://reader030.vdocuments.site/reader030/viewer/2022020917/61bd78a885f9f97d9a6a28b9/html5/thumbnails/11.jpg)
Summary of group discussion for
methods
1. Newtonโs method 1. Utilizes tangent
2. Golden section method 1. For line search
3. Downhill Simplex
4. Cyclic coordinate method 1. One coordinate at a time
5. Polytopy search (Nelder-Mead) 1. Idea based on geometry
6. Gradient descent (steepest descent) 1. Based on gradient information
spring 2014 TIES483 Nonlinear optimization
![Page 12: Introduction to unconstrained optimization - direct search](https://reader030.vdocuments.site/reader030/viewer/2022020917/61bd78a885f9f97d9a6a28b9/html5/thumbnails/12.jpg)
Direct search methods
Univariate search, coordinate descent, cyclic
coordinate search
Hooke and Jeeves
Powellโs method
spring 2014 TIES483 Nonlinear optimization
![Page 13: Introduction to unconstrained optimization - direct search](https://reader030.vdocuments.site/reader030/viewer/2022020917/61bd78a885f9f97d9a6a28b9/html5/thumbnails/13.jpg)
Coordinate descent
spring 2014 TIES483 Nonlinear optimization
Fro
m M
iett
ine
n: N
on
line
ar
op
tim
iza
tio
n, 2
00
7 (
in F
inn
ish
)
๐ ๐ฅ = 2๐ฅ12 + 2๐ฅ1๐ฅ2 + ๐ฅ2
2 + ๐ฅ1 โ ๐ฅ2
![Page 14: Introduction to unconstrained optimization - direct search](https://reader030.vdocuments.site/reader030/viewer/2022020917/61bd78a885f9f97d9a6a28b9/html5/thumbnails/14.jpg)
Idea of pattern search
spring 2014 TIES483 Nonlinear optimization
Fro
m M
iett
ine
n: N
on
line
ar
op
tim
iza
tio
n, 2
00
7 (
in F
inn
ish
)
![Page 15: Introduction to unconstrained optimization - direct search](https://reader030.vdocuments.site/reader030/viewer/2022020917/61bd78a885f9f97d9a6a28b9/html5/thumbnails/15.jpg)
Hooke and Jeeves
spring 2014 TIES483 Nonlinear optimization
Fro
m M
iett
ine
n: N
on
line
ar
op
tim
iza
tio
n, 2
00
7 (
in F
inn
ish
)
๐ ๐ฅ = ๐ฅ1 โ 2 4 + x1 โ 2x22
![Page 16: Introduction to unconstrained optimization - direct search](https://reader030.vdocuments.site/reader030/viewer/2022020917/61bd78a885f9f97d9a6a28b9/html5/thumbnails/16.jpg)
Hooke and Jeeves with fixed step length
spring 2014 TIES483 Nonlinear optimization
Fro
m M
iett
ine
n: N
on
line
ar
op
tim
iza
tio
n, 2
00
7 (
in F
inn
ish
)
๐ ๐ฅ = ๐ฅ1 โ 2 4 + x1 โ 2x22
![Page 17: Introduction to unconstrained optimization - direct search](https://reader030.vdocuments.site/reader030/viewer/2022020917/61bd78a885f9f97d9a6a28b9/html5/thumbnails/17.jpg)
Powellโs method
Most efficient pattern search method
Differs from Hooke and Jeeves so that for
each pattern search step one of the coordinate
directions is replaced with previous pattern
search direction.
spring 2014 TIES483 Nonlinear optimization