Nonlinear Optimization

Claudia Schillings

HU Berlin - 14. October 2015

based on material by Michael Hintermuller, HU Berlin, Thomas Surowiec, HU Berlin

C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 1 / 12

Contents

Notions of solutions

Optimality conditions for unconstrained problems

Unconstrained optimization by descent methods with step sizestrategies

Convergence rates

Gradient based methods

Conjugate gradient method

Newton’s method

Quasi-Newton methods

Optimality conditions forconstrained problems

Algorithms forconstrained problems

C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 2 / 12

Contents

Notions of solutions

Optimality conditions for unconstrained problems

Unconstrained optimization by descent methods with step sizestrategies

Convergence rates

Gradient based methods

Conjugate gradient method

Newton’s method

Quasi-Newton methods

Optimality conditions forconstrained problems

Algorithms forconstrained problems

C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 2 / 12

Applications

Biological applications

Engineering systems

Environmental systems

Physical systems

...

Source: Chen et al.

...almost everywhere

C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 3 / 12

Applications

Biological applications

Engineering systems

Environmental systems

Physical systems

...

Copyright DLR.

...almost everywhere

C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 3 / 12

Applications

Biological applications

Engineering systems

Environmental systems

Physical systems

...

Source: Muggeridge et al.

...almost everywhere

C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 3 / 12

Applications

Biological applications

Engineering systems

Environmental systems

Physical systems

...

Copyright photonics.com.

...almost everywhere

C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 3 / 12

Applications

Biological applications

Engineering systems

Environmental systems

Physical systems

...

Copyright photonics.com.

...almost everywhere

C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 3 / 12

Organizational Stuff

Lectures:

Wednesdays 13-15 RUD25 1.013

Thursdays 13-15 RUD25 1.115

Exercise classes:

Thursdays 15-17 RUD25 2.006.

First problem sheet will be available tomorrow (15.10.15) for downloadon our course homepage.

The first exercise class will take place on Thursday, October 22nd.

Course homepage:

www.mathematik.hu-berlin.de/de/forschung/schillings-lehre/schillings-non-opt/

C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 4 / 12

Organizational Stuff

Assessment:

Final Exam

Problems to be solved on the board in the exercise classes.

Coding exercises (MATLAB).

Lecture notes:

Lecture notes will be provided as the module progresses.

Contact details:

claudia.schillings@hu-berlin.de

Rudower Chaussee 25 , Room 2.408

Office hours: by appointment

www.mathematik.hu-berlin.de/de/personen/mitarb-vz/schillings-claudia

C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 4 / 12

Organizational Stuff

Literature:D. Bertsekas, Nonlinear Programming, Athena Scientific Publisher, Belmont,Massachusetts, 1995.

A. R. Conn, N. I. M. Gould, P. L. Toint, Trust-Region Methods, SIAM, Philadelphia, 2000.

J. E. Dennis, R. B. Schnabel, Numerical Methods for Unconstrained Optimization andNonlinear Equations, SIAM Philadelphia, 1996.

R. Fletcher, Practical Methods of Optimization, Wiley & Sons Publisher, New York, 1980.

C. Geiger, C. Kanzow, Numerische Verfahren zur Loesung unrestringierterOptimierungsaufgaben, Springer-Verlag, Berlin, 1999.

C. T. Kelley, Iterative Methods for Optimization, Frontiers in Applied Mathematics, SIAM,Philadelphia, 1999.

J. Nocedal and S. J. Wright, Numerical Optimization, Springer-Verlag, Berlin, 1999.

Prerequisites:

Linear Algebra, Analysis I + II, Analytic geometry

C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 4 / 12

Chapter 1: Introduction

Definition of a finite dimensional minimization problemLet X ⊂ Rn an arbitrary set and f : X → R a continuous function.The problem is to find an x∗ ∈ X such that

(1.1) f (x∗) ≤ f (x) for all x ∈ X .

Alternate formulations:

min f (x) s.t. x ∈ X ,

or

(1.2) minx∈X

f (x) .

C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 5 / 12

Chapter 1: Introduction

Definition of a finite dimensional minimization problemLet X ⊂ Rn an arbitrary set and f : X → R a continuous function.The problem is to find an x∗ ∈ X such that

(1.1) f (x∗) ≤ f (x) for all x ∈ X .

Alternate formulations:

min f (x) s.t. x ∈ X ,

or

(1.2) minx∈X

f (x) .

C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 5 / 12

Examples of Optimization ProblemsBasic IdeaGiven observations or measurements of a system of interest, how canwe determine certain intrinsic properties?

Undamped harmonic oscillatorLet M be a point of mass with mass m fixed to the end of avertical spring.

At equilibrium, M is located at the origin.

K is the restoring force which tries to replace M in itsequilibrium position.

For small (vertical) displacements y, the force K can bemodeled by Hooke’s law

K = −ky ,

where k denotes the (positive) spring constant.

http://people.seas.harvard.edu/jones/cscie129/nu lectures.

C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 6 / 12

Examples of Optimization Problems

Undamped harmonic oscillator

k denotes the unknown (positive) spring constant.

y(t) := position of M at time t.

Ignoring damping and friction, Newton’s law states:

(1.3) my = −ky ,

i.e. mass m times acceleration y equals the opposing force of the spring −ky.

(1.3) is called the undamped harmonic oscillator equation.

Usually, friction and damping forces behave proportionally to the velocity of M,i.e. −ry with fixed r > 0.

Together with (1.3), we obtain

my + ry + ky = 0 .

Setting c := r/m and k := k/m, we get

(1.4) y + cy + ky = 0 .

C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 7 / 12

Examples of Optimization ProblemsUndamped harmonic oscillator

Assume at time

(1.5) y(0) = y0 , y(0) = 0 .

Given endtime T > 0, we consider the initial boundary value problem (IVP) onthe interval [0, T].

The objective is to determine x = (c, k)> with the help of measurements.

For j = 1, . . . ,N, we are given measurements {yj}Nj=1 of the spring deviation at

time tj = (j− 1)T/(N − 1).

Let y(x; t) be the solution of the IVP for a given x. By solving the unconstrainedoptimization problem

(1.6) minx∈R2

f (x) :=12

N∑j=1

|y(x; tj)−yj|2 ,

we seek to determine the spring constant k and damping factor c.

Note that y(·; t) is differentiable w.r.t. x provided c2 − 4k 6= 0 .

(1.6) is a nonlinear least squares problem.

C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 8 / 12

Examples of Optimization Problems

Basic IdeaDeciding product capacity based on fixed and variable costs.

x output quantity.

Kv(x) variable costs, Kf (x) = c > 0 fixed costs.

K(x) := Kv(x) + Kf (x), x ∈ R total costs.

Normally one looks for an x∗, which minimizes total costs K(x), i.e.

(1.7) x∗ = argmin{Kv(x)+Kf (x) : x ∈ R} = argmin{Kv(x) : x ∈ R} .

In general, x∗ is not unique. We therefore write:

x∗ ∈ argmin{Kv(x) : x ∈ R} .

C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 9 / 12

Constrained OptimizationParticularly in the previous example, one often has constraints on x.

When X 6= Rn, we often have

X = X1 ∩ X2 ∩ X3

with sets

X1 = {x ∈ Rn : ci(x) = 0, i ∈ I1} ,X2 = {x ∈ Rn : ci(x) ≤ 0, i ∈ I2} ,X3 = {x ∈ Rn : xi ∈ Z, i ∈ I3} .

Ii ⊂ N, i = 1, 2, 3, are called (finite) index sets.

X1,X2,X3 are called equality, inequality and integer constraints.

X set of discrete points→ discrete or combinatorial optimization.

Otherwise, continuous optimization.

f , ci for any i is non differentiable→ nonsmooth optimization.

C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 10 / 12

Constrained OptimizationParticularly in the previous example, one often has constraints on x.

When X 6= Rn, we often have

X = X1 ∩ X2 ∩ X3

with sets

X1 = {x ∈ Rn : ci(x) = 0, i ∈ I1} ,X2 = {x ∈ Rn : ci(x) ≤ 0, i ∈ I2} ,X3 = {x ∈ Rn : xi ∈ Z, i ∈ I3} .

Ii ⊂ N, i = 1, 2, 3, are called (finite) index sets.

X1,X2,X3 are called equality, inequality and integer constraints.

X set of discrete points→ discrete or combinatorial optimization.

Otherwise, continuous optimization.

f , ci for any i is non differentiable→ nonsmooth optimization.

C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 10 / 12

Notions of Solutions

Definition 1.1Let f : X → R with X ⊂ Rn. The point x∗ ∈ X is called a

(i) (strict) global minimizer of f (on X), if

f (x∗) ≤ f (x) (f (x∗) < f (x)) ∀x ∈ X \ {x∗} .

The optimal objective value f (x∗) is called (strict) global minimum.

(ii) (strict) local minimizer of f (on X) if there exists a neighborhood of U of x∗ suchthat

f (x∗) ≤ f (x) (f (x∗) < f (x)) ∀x ∈ (X ∩ U) \ {x∗} .The optimal objective value f (x∗) is called (strict) local minimum.

C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 11 / 12

Stationary Points

Let X ⊂ Rn be an open set and f : X → R be a differentiable function. We denoteits gradient by

∇f (x) =(∂f∂x1

(x), . . . ,∂f∂xn

(x))>

.

If f : X → R is directionally differentiable, then its directional derivative at x ∈ X indirection d ∈ Rn is denoted by

f ′(x; d) := limα↓0

f (x + αd)− f (x)α

.

Definition 1.2Let X ⊂ Rn be an open set and f : X → R be a continuously differentiable function.The point x∗ ∈ X is called a stationary point of f , if

∇f (x∗) = 0

holds true.

C. Schillings (HU Berlin) Nonlinear Optimization HU Berlin - 14. October 2015 12 / 12