mathematical concepts - geradmathematical concepts mth8418 s. le digabel, polytechnique montr eal...

Properties of functions Optimization definitions Directions References

Mathematical concepts

MTH8418

S. Le Digabel, Polytechnique Montreal

Winter 2020(v4)

MTH8418: Concepts 1/40


Plan

Properties of functions

Optimization definitions

Directions

References





Directions

References



Properties of a function

I We consider a function f : Rn → R

I We define some properties of f at x, a point of its domain

I We say that these properties apply near x if the property issatisfied on some open neighborhood of x

I We can also consider some properties on a domain X ⊂ Rn

I f is continuous at x ∈ Rn if the limit limy→x

f(y) exists and is

equal to f(x)



DifferentiabilityI We consider the function f : Rn → RI f is differentiable at x ∈ Rn if there exists g ∈ Rn such that

limy→x

f(y)− f(x)− g>(y − x)

‖y − x‖= 0

I If this g exists, it is unique and is called the gradient of f atx, denoted ∇f(x)

I If f is differentiable at x, then f is continuous at x

I Partial derivatives:∂f(x)∂xi

= limh→0

1h (f(x1, . . . , xi + h, . . . , xn)− f(x)) (if the limit exists)

I Gradient and partial derivatives:

∇f(x) =

(∂f(x)

∂x1,∂f(x)

∂x2, . . . ,

∂f(x)

∂xn

)MTH8418: Concepts 5/40


Example 1

Illustrate that all partial derivatives of a function may exist, even ifthe function is not differentiable.



Directional derivatives

I If it exists, the directional derivative at x in the directionv ∈ Rn is defined by

f ′(x; v) = limt↓0

f(x+ tv)− f(x)

t

I If the ith partial derivative exists, then ∂f(x)∂xi

= f ′(x; ei) whereei is the ith unit vector

I If f is differentiable at x, then f ′(x; v) = v>∇f(x) for allv ∈ Rn, and f ′(x;−d) = −f ′(x; d)

I All directional derivatives of a function may exist, even if thefunction is not differentiable.



Example 2

Consider f(x) = ‖x‖ =√x>x

I What is the directional derivative of f at the origin and in thedirection v ∈ Rn?

I What is the gradient of f at the origin?



Differentiability classes, smoothness

I A function f : Rn → R is said of class Ck, denoted f ∈ Ck,with 0 ≤ k ≤ ∞, if all the possible partial derivatives of theform

∂kf

∂xi1∂xi2 · · · ∂xikexist and are continuous, where i` ∈ 1, 2, . . . , n for all` ∈ 1, 2, . . . , k

I C0: Continuous functions

I C1: Continuously differentiable functions

I C∞: Smooth functions



Strict differentiability

Let f be differentiable at x ∈ Rn. f is said strictly differentiable atx if for all v ∈ Rn:

limy→x,t↓0

f(y + tv)− f(y)

t= lim

y→xf ′(y; v) = f ′(x; v) = v>∇f(x)



Convex sets

I A set in a vector space is convex if for every pair of pointswithin the set, every point on the straight line segment thatjoins them is also within the object:

U convex ⇔ tx+ (1− t)y ∈ U for all t ∈ [0; 1] and (x, y) ∈ U

I If the set U is convex, any convex combination of elements ofU belongs to U :

U convex ⇔p∑i=1

λiui ∈ U with ui ∈ U , λi ≥ 0 for

i ∈ 1, 2, . . . , p, andp∑i=1

λi = 1



Convex functions

I The epigraph of f : Rn → R is the set of points lying aboveits graph:

epi(f) = (x, z) : x ∈ Rn, z ∈ R : f(x) ≤ z ⊆ Rn+1

I f is convex if epi(f) is a convex set

I With n = 1, if f ′′ exists and f ′′(x) ≥ 0 for all x ∈ R, then fis convex

I With convex functions, local optimality = global optimality



Lipschitz functions

I f is Lipschitz on the set X ⊂ Rn if there exists a scalarK > 0 such that

|f(x)− f(y)| ≤ K‖x− y‖ for all x, y ∈ X

I K is called the Lipschitz constant

I Example of non-Lipschitz functions: f(x) =√x for x ≥ 0,

any discontinuous function



Generalized derivatives

I Let f be Lipschitz near x

I The Clarke generalized derivative [Clarke, 1983] of f at x inthe direction v ∈ Rn is

f(x; v) = lim supy→x, t↓0

f(y + tv)− f(y)

t

I If f is convex, the standard directional derivatives and theClarke generalized derivatives are identical

I f is regular at x if for every direction v ∈ Rn, f ′(x; v) existsand equals f(x; v)



Generalized gradient

I The generalized gradient [Clarke, 1983] of f at x is the set

∂f(x) =ξ ∈ Rn : f(x; v) ≥ v>ξ for all v ∈ Rn

I f(x; v) = maxξ∈∂f(x)

v>ξ

I If ∂f(x) reduces to ξ, then f is strictly differentiable at xand ∇f(x) = ξ



Example 3

Consider f(x) = |x|. We have:

I f(0; d) = |d|

I f is regular

I ∂f(0) = [−1; 1]

Note that f(x) = −|x| is not regular: At x = 0, for every direction,the Clarke derivative is > 0 while the function is decreasing



Example 4 (1/2)

f(x) =

x2(2 + sin(πx )

)if x 6= 0

0 if x = 0

−1 −0.5 0 0.5 10

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

y = x2(2+sin(π/x))

x

y



Example 4 (2/2)I Single global optimizer at x = 0I Infinitely many local optima in any open neighborhood of 0I f is differentiable everywhere as

∇f(x) = f ′(x) =

2x(2 + sin(πx )

)− π cos(πx ) if x 6= 0

0 if x = 0

I f is Lipschitz near every x ∈ RI The derivative is not continuous at 0 since ∇f(0) = 0 and∇f(1/2k) = 2/k − π for every integer k 6= 0

I f is not strictly differentiable at x = 0 because the generalizedgradient is ∂f(0) = [−π;π]

I f is not regular at x = 0 since the Clarke generalizedderivatives f(0;±1) = π differ from the standard directionalderivatives f ′(0;±1) = 0



Types of functions: Summaryf : X ⊆ Rn → R is...

Continuous near x Lipschitz near x

Regular at x

Differentiable at x

Continuously differentiable at xConvex on X

Strictly differentiable at x





Directions

References



Optimization problem

We consider the optimization problem:

minx∈Ω

f(x)

I Ω =x ∈ X ⊆ Rn : cj(x) ≤ 0, j ∈ J = 1, 2, . . . ,m

I n variables, m general constraints

I Typically X contains bounds, a-priori constraints,nonquantifiable constraints (defined in Lecture #9)



Some types of problems

I Linear Optimization: f linear, Ω = X , and X contains onlybounds and linear constraints

I Nonlinear Optimization: Functions f and cj , j ∈ J , arenonlinear

I Derivative-Free Optimization: Derivatives are unavailable

I Discrete or combinatorial Optimization: X * Rn and some orall the variables are integers or booleans. Metaheuristicsexploiting the structure of the problem are best fitted



Types of optimality

I x is a feasible solution if x ∈ Ω

I x /∈ Ω⇔ x is infeasible

I x∗ ∈ Ω is a global optimum if f(x∗) ≤ f(x) for all x ∈ Ω.Also noted x∗ ∈ arg min

x∈Ωf(x)

I x∗ ∈ Ω is a local optimum if there exists ε > 0 such thatf(x∗) ≤ f(x) for all x ∈ Ω and ‖x− x∗‖ ≤ ε

I Global optimum = Optimum = Optimizer.In the minimization context: replace “opt” with “min”

I With convexity, local optimality = global optimality



Convergence analysis (1/2)

I An optimization algorithm is not considered as a heuristicwhen it is backed by a convergence analysis which ensuressome properties at the resulting solution x

I This analysis typically depends on some assumptions madeabout the nature of the problem. For example:differentiability of f , convexity of Ω, etc.

I Usually, these properties are given as necessary or sufficientoptimality conditions

I There is global convergence when the properties of the resultare independent of the starting solution(s)



Convergence analysis (2/2)

I In DFO, we expect global convergence to solutions satisfyingsome local and necessary optimality conditions, when thefunction is supposed Lipschitz

I However, a blackbox has no exploitable property and cannotbe proven Lipschitz

I But consider the following choice between two algorithms toapply to such a problem:I Algorithm A is a heuristic. It can give you ∇f(x) 6= 0 when f

is differentiableI Algorithm B guarantees ∇f(x) = 0 when f is differentiable

The choice is obvious



Optimality conditions: Unconstrained case

If all the directional derivatives at x exist, then:

I x is a local minimum ⇒ all the directional derivatives arenon-negative

I If in addition f is differentiable, then f ′(x; d) = 0 for alld ∈ Rn and ∇f(x) = 0 since f ′(x;−d) = −f ′(x; d) for alld ∈ Rn



Active set

I Feasible set:

Ω =x ∈ Rn : cj(x) ≤ 0, j ∈ J

(with X = Rn)

I For any x ∈ Rn, the active set A(x) is the set of indices ofthe constraints satisfied to equality at x:

A(x) = j ∈ J : cj(x) = 0



Cones

I Cones are used to state the properties of the feasible region

I K ⊆ Rn is a cone if λd ∈ K for all scalar λ > 0 and all d ∈ K

I The polar of cone K ⊆ Rn isK∗ = d ∈ Rn : d>v ≤ 0 for all v ∈ K

I If the cj functions, j ∈ J , are differentiable, then the normal

cone is NΩ(x) =

∑j∈A(x)

λj∇cj(x) : λj ≥ 0, j ∈ A(x)

I Polar of normal cone = tangent cone: N∗Ω(x) = TΩ(x)

I Tangent cone ' cone of feasible directions



Optimality conditions: Graphical interpretation

&%'$.

...............................

............................

.........................

......................

....................

..................

................

..............

........

....

........

...

......................

............. ................ .................. ..................... .......................

.............................

.........................

...........................................

................. .............. ............. ............ ........................

...............

........

........

.

........

........

...

. ........................... .................................................

...........................................

....................

..................

....................

.....................

......................

........................

.........................

...........................

. ...................................... ........................................................................

.................................................................

..............................

.............................

...........................

..........................

..........................

............................

.............................

..............................

................................

.................................

..................................

....................................

. ..............................................................................

......................................

......................................

.....................................

.....................................

....................................

....................................

...................................

.....................................

........................................

..........................................

............................................

..............................................

.................................................

..........................................

..........................................

...........................................

............................................

.............................................

..............................................

...............................................

................................................

.................................................

....................................................

.......................................................

..........................................................

.............................................................

.

..................................................

................................................

..............................................

............................................

..........................................

........................................

......................................

...................................

................................................................

............................. ........................... . ..................... ........................................

....................

....................

.....................

.......................

..........................

............................

...............................

.................................

....................................

......................................

.

.................................................

...............................................

.............................................

...........................................

.........................................

.......................................

........................................................................

................................. ............................... ............................... ..........................................................

..... ............... .............. .............. ..............

...............

...............

..................

.....................

........................

...........................

..............................

.................................

....................................

.......................................

.........................................

............................................

.............................

..............................

..................................................................

.......................................................................... ........................................ .......................................... ............................................. ................................................ . .................... ................... ...................

...................

...................

....................

......................

.........................

............................

...............................

.................................

....................................

.......................................

..........................................

............................................

...............................................

5

6

76

5 43

2

1c2(x) = 0

c1(x) = 0

Ω

..........................................

........................................

...............................................................................

............................................................................ ..................................... .................................... ..................................... ...................................... ........................................ .........................................

..........................................

............................................

.............................................

..............................................

............................................

.....

......................................

....................................

.................................. ................................. ............................... ............................. ............................ ..........................................................

............................................................

.

....................................

.................................

...............................

.............................

..........................

.............

...........

.............

.........

.....................

.....................

....................

....................

...................

.....................

........................ .......................... ............................ .............................. ................................. ................................... ..................................... . .......................................................... ....................................................... .................................................... ................................................. ............................................... ............................................. ....................................................................................

............................................................................

....................................................................

...............................

.............................

............................

...........................




&%'$.

...............................

............................

.........................

......................

....................

..................

................

..............

........

....

........

...

......................

............. ................ .................. ..................... .......................

.............................

.........................

...........................................

................. .............. ............. ............ ........................

...............

........

........

.

........

........

...

. ........................... .................................................

...........................................

....................

..................

....................

.....................

......................

........................

.........................

...........................

. ...................................... ........................................................................

.................................................................

..............................

.............................

...........................

..........................

..........................

............................

.............................

..............................

................................

.................................

..................................

....................................

. ..............................................................................

......................................

......................................

.....................................

.....................................

....................................

....................................

...................................

.....................................

........................................

..........................................

............................................

..............................................

.................................................

..........................................

..........................................

...........................................

............................................

.............................................

..............................................

...............................................

................................................

.................................................

....................................................

.......................................................

..........................................................

.............................................................

.

..................................................

................................................

..............................................

............................................

..........................................

........................................

......................................

...................................

................................................................

............................. ........................... . ..................... ........................................

....................

....................

.....................

.......................

..........................

............................

...............................

.................................

....................................

......................................

.

.................................................

...............................................

.............................................

...........................................

.........................................

.......................................

........................................................................

................................. ............................... ............................... ..........................................................

..... ............... .............. .............. ..............

...............

...............

..................

.....................

........................

...........................

..............................

.................................

....................................

.......................................

.........................................

............................................

.............................

..............................

..................................................................

.......................................................................... ........................................ .......................................... ............................................. ................................................ . .................... ................... ...................

...................

...................

....................

......................

.........................

............................

...............................

.................................

....................................

.......................................

..........................................

............................................

...............................................

5

6

76

5 43

2

1c2(x) = 0

c1(x) = 0

Ω

..........................................

........................................

...............................................................................

............................................................................ ..................................... .................................... ..................................... ...................................... ........................................ .........................................

..........................................

............................................

.............................................

..............................................

............................................

.....

......................................

....................................

.................................. ................................. ............................... ............................. ............................ ..........................................................

............................................................

.

....................................

.................................

...............................

.............................

..........................

.............

...........

.............

.........

.....................

.....................

....................

....................

...................

.....................

........................ .......................... ............................ .............................. ................................. ................................... ..................................... . .......................................................... ....................................................... .................................................... ................................................. ............................................... ............................................. ....................................................................................

............................................................................

....................................................................

...............................

.............................

............................

...........................

• xa




&%'$.

...............................

............................

.........................

......................

....................

..................

................

..............

........

....

........

...

......................

............. ................ .................. ..................... .......................

.............................

.........................

...........................................

................. .............. ............. ............ ........................

...............

........

........

.

........

........

...

. ........................... .................................................

...........................................

....................

..................

....................

.....................

......................

........................

.........................

...........................

. ...................................... ........................................................................

.................................................................

..............................

.............................

...........................

..........................

..........................

............................

.............................

..............................

................................

.................................

..................................

....................................

. ..............................................................................

......................................

......................................

.....................................

.....................................

....................................

....................................

...................................

.....................................

........................................

..........................................

............................................

..............................................

.................................................

..........................................

..........................................

...........................................

............................................

.............................................

..............................................

...............................................

................................................

.................................................

....................................................

.......................................................

..........................................................

.............................................................

.

..................................................

................................................

..............................................

............................................

..........................................

........................................

......................................

...................................

................................................................

............................. ........................... . ..................... ........................................

....................

....................

.....................

.......................

..........................

............................

...............................

.................................

....................................

......................................

.

.................................................

...............................................

.............................................

...........................................

.........................................

.......................................

........................................................................

................................. ............................... ............................... ..........................................................

..... ............... .............. .............. ..............

...............

...............

..................

.....................

........................

...........................

..............................

.................................

....................................

.......................................

.........................................

............................................

.............................

..............................

..................................................................

.......................................................................... ........................................ .......................................... ............................................. ................................................ . .................... ................... ...................

...................

...................

....................

......................

.........................

............................

...............................

.................................

....................................

.......................................

..........................................

............................................

...............................................

5

6

76

5 43

2

1c2(x) = 0

c1(x) = 0

Ω

..........................................

........................................

...............................................................................

............................................................................ ..................................... .................................... ..................................... ...................................... ........................................ .........................................

..........................................

............................................

.............................................

..............................................

............................................

.....

......................................

....................................

.................................. ................................. ............................... ............................. ............................ ..........................................................

............................................................

.

....................................

.................................

...............................

.............................

..........................

.............

...........

.............

.........

.....................

.....................

....................

....................

...................

.....................

........................ .......................... ............................ .............................. ................................. ................................... ..................................... . .......................................................... ....................................................... .................................................... ................................................. ............................................... ............................................. ....................................................................................

............................................................................

....................................................................

...............................

.............................

............................

...........................

• xa∇f(xa)=0




&%'$.

...............................

............................

.........................

......................

....................

..................

................

..............

........

....

........

...

......................

............. ................ .................. ..................... .......................

.............................

.........................

...........................................

................. .............. ............. ............ ........................

...............

........

........

.

........

........

...

. ........................... .................................................

...........................................

....................

..................

....................

.....................

......................

........................

.........................

...........................

. ...................................... ........................................................................

.................................................................

..............................

.............................

...........................

..........................

..........................

............................

.............................

..............................

................................

.................................

..................................

....................................

. ..............................................................................

......................................

......................................

.....................................

.....................................

....................................

....................................

...................................

.....................................

........................................

..........................................

............................................

..............................................

.................................................

..........................................

..........................................

...........................................

............................................

.............................................

..............................................

...............................................

................................................

.................................................

....................................................

.......................................................

..........................................................

.............................................................

.

..................................................

................................................

..............................................

............................................

..........................................

........................................

......................................

...................................

................................................................

............................. ........................... . ..................... ........................................

....................

....................

.....................

.......................

..........................

............................

...............................

.................................

....................................

......................................

.

.................................................

...............................................

.............................................

...........................................

.........................................

.......................................

........................................................................

................................. ............................... ............................... ..........................................................

..... ............... .............. .............. ..............

...............

...............

..................

.....................

........................

...........................

..............................

.................................

....................................

.......................................

.........................................

............................................

.............................

..............................

..................................................................

.......................................................................... ........................................ .......................................... ............................................. ................................................ . .................... ................... ...................

...................

...................

....................

......................

.........................

............................

...............................

.................................

....................................

.......................................

..........................................

............................................

...............................................

5

6

76

5 43

2

1c2(x) = 0

c1(x) = 0

Ω

..........................................

........................................

...............................................................................

............................................................................ ..................................... .................................... ..................................... ...................................... ........................................ .........................................

..........................................

............................................

.............................................

..............................................

............................................

.....

......................................

....................................

.................................. ................................. ............................... ............................. ............................ ..........................................................

............................................................

.

....................................

.................................

...............................

.............................

..........................

.............

...........

.............

.........

.....................

.....................

....................

....................

...................

.....................

........................ .......................... ............................ .............................. ................................. ................................... ..................................... . .......................................................... ....................................................... .................................................... ................................................. ............................................... ............................................. ....................................................................................

............................................................................

....................................................................

...............................

.............................

............................

...........................

•xb




&%'$.

...............................

............................

.........................

......................

....................

..................

................

..............

........

....

........

...

......................

............. ................ .................. ..................... .......................

.............................

.........................

...........................................

................. .............. ............. ............ ........................

...............

........

........

.

........

........

...

. ........................... .................................................

...........................................

....................

..................

....................

.....................

......................

........................

.........................

...........................

. ...................................... ........................................................................

.................................................................

..............................

.............................

...........................

..........................

..........................

............................

.............................

..............................

................................

.................................

..................................

....................................

. ..............................................................................

......................................

......................................

.....................................

.....................................

....................................

....................................

...................................

.....................................

........................................

..........................................

............................................

..............................................

.................................................

..........................................

..........................................

...........................................

............................................

.............................................

..............................................

...............................................

................................................

.................................................

....................................................

.......................................................

..........................................................

.............................................................

.

..................................................

................................................

..............................................

............................................

..........................................

........................................

......................................

...................................

................................................................

............................. ........................... . ..................... ........................................

....................

....................

.....................

.......................

..........................

............................

...............................

.................................

....................................

......................................

.

.................................................

...............................................

.............................................

...........................................

.........................................

.......................................

........................................................................

................................. ............................... ............................... ..........................................................

..... ............... .............. .............. ..............

...............

...............

..................

.....................

........................

...........................

..............................

.................................

....................................

.......................................

.........................................

............................................

.............................

..............................

..................................................................

.......................................................................... ........................................ .......................................... ............................................. ................................................ . .................... ................... ...................

...................

...................

....................

......................

.........................

............................

...............................

.................................

....................................

.......................................

..........................................

............................................

...............................................

5

6

76

5 43

2

1c2(x) = 0

c1(x) = 0

Ω

..........................................

........................................

...............................................................................

............................................................................ ..................................... .................................... ..................................... ...................................... ........................................ .........................................

..........................................

............................................

.............................................

..............................................

............................................

.....

......................................

....................................

.................................. ................................. ............................... ............................. ............................ ..........................................................

............................................................

.

....................................

.................................

...............................

.............................

..........................

.............

...........

.............

.........

.....................

.....................

....................

....................

...................

.....................

........................ .......................... ............................ .............................. ................................. ................................... ..................................... . .......................................................... ....................................................... .................................................... ................................................. ............................................... ............................................. ....................................................................................

............................................................................

....................................................................

...............................

.............................

............................

...........................

•xb

∇f(xb)=−λ∇c1(xb)

∇c1(xb)




&%'$.

...............................

............................

.........................

......................

....................

..................

................

..............

........

....

........

...

......................

............. ................ .................. ..................... .......................

.............................

.........................

...........................................

................. .............. ............. ............ ........................

...............

........

........

.

........

........

...

. ........................... .................................................

...........................................

....................

..................

....................

.....................

......................

........................

.........................

...........................

. ...................................... ........................................................................

.................................................................

..............................

.............................

...........................

..........................

..........................

............................

.............................

..............................

................................

.................................

..................................

....................................

. ..............................................................................

......................................

......................................

.....................................

.....................................

....................................

....................................

...................................

.....................................

........................................

..........................................

............................................

..............................................

.................................................

..........................................

..........................................

...........................................

............................................

.............................................

..............................................

...............................................

................................................

.................................................

....................................................

.......................................................

..........................................................

.............................................................

.

..................................................

................................................

..............................................

............................................

..........................................

........................................

......................................

...................................

................................................................

............................. ........................... . ..................... ........................................

....................

....................

.....................

.......................

..........................

............................

...............................

.................................

....................................

......................................

.

.................................................

...............................................

.............................................

...........................................

.........................................

.......................................

........................................................................

................................. ............................... ............................... ..........................................................

..... ............... .............. .............. ..............

...............

...............

..................

.....................

........................

...........................

..............................

.................................

....................................

.......................................

.........................................

............................................

.............................

..............................

..................................................................

.......................................................................... ........................................ .......................................... ............................................. ................................................ . .................... ................... ...................

...................

...................

....................

......................

.........................

............................

...............................

.................................

....................................

.......................................

..........................................

............................................

...............................................

5

6

76

5 43

2

1c2(x) = 0

c1(x) = 0

Ω

..........................................

........................................

...............................................................................

............................................................................ ..................................... .................................... ..................................... ...................................... ........................................ .........................................

..........................................

............................................

.............................................

..............................................

............................................

.....

......................................

....................................

.................................. ................................. ............................... ............................. ............................ ..........................................................

............................................................

.

....................................

.................................

...............................

.............................

..........................

.............

...........

.............

.........

.....................

.....................

....................

....................

...................

.....................

........................ .......................... ............................ .............................. ................................. ................................... ..................................... . .......................................................... ....................................................... .................................................... ................................................. ............................................... ............................................. ....................................................................................

............................................................................

....................................................................

...............................

.............................

............................

...........................•xc




&%'$.

...............................

............................

.........................

......................

....................

..................

................

..............

........

....

........

...

......................

............. ................ .................. ..................... .......................

.............................

.........................

...........................................

................. .............. ............. ............ ........................

...............

........

........

.

........

........

...

. ........................... .................................................

...........................................

....................

..................

....................

.....................

......................

........................

.........................

...........................

. ...................................... ........................................................................

.................................................................

..............................

.............................

...........................

..........................

..........................

............................

.............................

..............................

................................

.................................

..................................

....................................

. ..............................................................................

......................................

......................................

.....................................

.....................................

....................................

....................................

...................................

.....................................

........................................

..........................................

............................................

..............................................

.................................................

..........................................

..........................................

...........................................

............................................

.............................................

..............................................

...............................................

................................................

.................................................

....................................................

.......................................................

..........................................................

.............................................................

.

..................................................

................................................

..............................................

............................................

..........................................

........................................

......................................

...................................

................................................................

............................. ........................... . ..................... ........................................

....................

....................

.....................

.......................

..........................

............................

...............................

.................................

....................................

......................................

.

.................................................

...............................................

.............................................

...........................................

.........................................

.......................................

........................................................................

................................. ............................... ............................... ..........................................................

..... ............... .............. .............. ..............

...............

...............

..................

.....................

........................

...........................

..............................

.................................

....................................

.......................................

.........................................

............................................

.............................

..............................

..................................................................

.......................................................................... ........................................ .......................................... ............................................. ................................................ . .................... ................... ...................

...................

...................

....................

......................

.........................

............................

...............................

.................................

....................................

.......................................

..........................................

............................................

...............................................

5

6

76

5 43

2

1c2(x) = 0

c1(x) = 0

Ω

..........................................

........................................

...............................................................................

............................................................................ ..................................... .................................... ..................................... ...................................... ........................................ .........................................

..........................................

............................................

.............................................

..............................................

............................................

.....

......................................

....................................

.................................. ................................. ............................... ............................. ............................ ..........................................................

............................................................

.

....................................

.................................

...............................

.............................

..........................

.............

...........

.............

.........

.....................

.....................

....................

....................

...................

.....................

........................ .......................... ............................ .............................. ................................. ................................... ..................................... . .......................................................... ....................................................... .................................................... ................................................. ............................................... ............................................. ....................................................................................

............................................................................

....................................................................

...............................

.............................

............................

...........................•xc∇c2(xc)

∇c1(xc)

∇f(xc)= −λ1∇c1(xc)−λ2∇c2(xc)




I ∇f(xa) = 0, A(xa) = ∅, NΩ(xa) = ∅, TΩ(xa) = R2

I A(xb) = 1, NΩ(xb) reduces to the half-line in the direction∇c1(xb), TΩ(xb) union of 0 and the open half-spaceorthogonal to ∇c1(xb)

I A(xc) = 1, 2,NΩ(xc) = λ1∇c1(xc) + λ2∇c2(xc) : λ1, λ2 ≥ 0



Optimality conditions: Differentiable case

If x ∈ Ω is a local minimizer of a differentiable function f subjectto differentiable constraints, then

−∇f(x) ∈ NΩ(x)

andf ′(x; d) ≥ 0 ∀ d ∈ TΩ(x)

where

f ′(x; d) = limt→0

f(x+ td)− f(x)

t= d>∇f(x)



Optimality conditions: Generalization

If x ∈ Ω is a local minimizer of a Lipschitz function f over a setΩ ⊂ Rn, then

f(x; d) ≥ 0 ∀ d ∈ THΩ (x)

where

I f(x; d) is the generalized directional derivative

I THΩ (x) is the hypertangent cone, a generalization of thetangent cone





Directions

References



Descent directions

Let f : Rn → R be a differentiable function at x

I A descent direction from x is a nonzero vector v ∈ Rn suchthat f ′(x; v) ≤ 0

I A strict descent direction from x is a nonzero vector v ∈ Rnsuch that f ′(x; v) < 0

I If ∇f(x) 6= 0:

I There is a closed half space H ⊆ Rn such that f ′(x; v) ≥ 0 ifand only if v ∈ H

I For any direction v ∈ Rn, either v or −v is a descent direction



Positive spanning sets and bases

I A positive spanning set (or generating set) for Rn is a finiteset of vectors whose nonnegative linear combinations span Rn

I A positive basis is a positive spanning set such that no propersubset is a positive spanning set

I A basis of Rn is not a positive basis for Rn

I Positive spanning sets contain at least one element in everyopen half-space

I If f is differentiable at a non-stationary point x, there is atleast one element of any positive spanning set that is a strictdescent direction



Example 5

6

HHHj

(a)

6

HHHj

(b)

6

HHHj

?

HHHY

(c)

6

HHHj

?

HHHY

(d)

A basis, two positive bases and a positive spanning set of R2



Cosine measure

I D: set of directions in Rn

I The cosine measure [Kolda et al., 2003] is defined by

κ(D) = minv∈Rn

maxd∈D

v>d

‖v‖‖d‖= min

v∈Rnmaxd∈D

cos〈v, d〉

I It is the cosine of the largest angle between an arbitrary vectorv and the closest direction in D

I It identifies the “largest hole” in the directions

I If D is a positive spanning set, κ(D) > 0

I Sets with higher κ span the space better



Example 6 (cosine measure)

With ei the ith coordinate vector for i ∈ 1, 2, . . . , n, give thevalue of κ(D) for the following sets of directions with n = 2:

1. D = e1

2. D = e1, e2

3. D = e1, e2,−e2

4. D = e1, e2,−e1,−e2

5. D = e1, e2,−e1 − e2





Directions

References



References I

Audet, C. and Hare, W. (2017).

Derivative-Free and Blackbox Optimization.Springer Series in Operations Research and Financial Engineering. Springer International Publishing, Berlin.

Clarke, F. (1983).

Optimization and Nonsmooth Analysis.John Wiley & Sons, New York.Reissued in 1990 by SIAM Publications, Philadelphia, as Vol. 5 in the series Classics in Applied Mathematics.

Kolda, T., Lewis, R., and Torczon, V. (2003).

Optimization by direct search: New perspectives on some classical and modern methods.SIAM Review, 45(3):385–482.

Nocedal, J. and Wright, S. (2006).

Numerical Optimization.Springer Series in Operations Research and Financial Engineering. Springer, Berlin, second edition.


mathematical concepts - geradmathematical concepts mth8418 s. le digabel, polytechnique montr eal...

Documents