numerical optimization in matlab - andrii parkhomenko · 2017-03-31 · optimization typical...

Numerical Optimization in MATLAB

Andrii Parkhomenko

Universitat Autonoma de Barcelona and Barcelona GSE

Spring 2017

Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 0 / 24

Optimization

Typical problem in economics: find an optimum

Case 1:

maxx

f (x) = 0.5x − 0.02x2

Easy: just solve for x that satisfies f ′(x) = 0

Case 2:

maxx

f (x) = 0.5 ln x − 0.02x2 +1

(1 + x)53

Can’t solve analytically ⇒ need to use numerical methods


Numerical Optimization

Continuous differentiable objective function: derivative-based methods

Arbitrary objective function: derivative-free methods


Numerical Optimization in MATLAB

These slides cover three functions contained in MATLAB’s Optimization

Toolbox:

fminunc: unconstrained optimization, derivative-based

fmincon: constrained optimization, derivative-based

fminsearch: unconstrained optimization, derivative-free


fminunc

Finds parameters that minimize a given function

Syntax:x = fminunc(objfun,x0)

I x0: starting guessI objfun: objective function specified in a different scriptI x: solution


fminunc

Example 1:

maxx

f (x) = 0.5 ln x − 0.02x2 +1

(1 + x)53

0 1 2 3 4 5−1.4

−1.2

−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

x

f(x)


fminunc

Example 1:

maxx

f (x) = 0.5 ln x − 0.02x2 +1

(1 + x)53

Step 1. Main code

x0 = 1;

x = fminunc(@objfun,x0)

Step 2. Function

function y = objfun(x)

y = 0.5*log(x) - 0.02*x^2 + (1+x)^(-5/3);

y = -y;

end


fminunc

fminunc uses the BFGS Quasi-Newton method:

xk+1 = xk + αkpk

1 Compute direction pk

pk = −B−1k ∇fk

Bk+1 = Bk −Bksks

′kBk

s ′kBksk+

yky′k

y ′ksk

where sk ≡ xk+1 − xk and yk ≡ fk+1 − fk

2 Compute stepsize αk . Use line search to approximately solve

minαk

f (xk + αkpk)


fminunc

Introduce options:

options = optimset(’Opt1’,Opt1Val,’Opt2’,Opt2Val,...);

x = fminunc(@objfun,x0, options)

Important algorithm options:

Display:

optimset(’Display’, ’off’)

Tolerance level:

optimset(’TolFun’, 1e-6)

optimset(’TolX’, 1e-6)

Maximum number of iterations or evaluations

optimset(’MaxIter’, 400)

optimset(’MaxFunEvals’, 100)


fminunc

Caution:

Objective function must be continuous

Local minima and flat regions are big issues


fminunc

Example 2. Discontinuous objective function:

maxx

f (x) =

{0.5 ln x − 0.02x2 + 1

(1+x)53, 0 < x ≤ 4

0.5 ln x − 0.02x2 + 1

(1+x)53

+ 1, 4 < x

0 1 2 3 4 5−1.5

−1

−0.5

0

0.5

1

1.5

x

f(x)


fminunc

Example 3. Objective function with flat regions:

maxx

f (x) =

{0.5 ln x − 0.02x2 + 1

(1+x)53, 0 < x ≤ 4

0.441546, 4 < x

0 1 2 3 4 5−1.4

−1.2

−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

x

f(x)


fminunc

Example 4. Objective function with local maxima:

maxx

f (x) = sin(3x) + x − 0.25x2

0 1 2 3 4 5−1

−0.5

0

0.5

1

1.5

2

x

f(x)


fminunc

To obtain more details on the workings of fminunc:

Read documentation: Help → Type ’fminunc’

Read the code: Type ’open fminunc’


fmincon

Finds parameters that minimize a given function, subject to a

constraint

Same algorithm as fminunc

Syntax:x = fmincon(objfun,x0,A,b,Aeq,beq,lb,ub,nlcon,options)

I x0: starting guessI objfun: objective function specified in a different scriptI A, b: linear inequality constraint: Ax ≤ bI Aeq, beq: linear equality constraint: Ax = bI lb, ub: lower and upper bounds: x ∈ [lb, ub]I nlcon: nonlinear constraintI x: solution


fmincon

Example 1:

maxx

f (x) = 0.5 ln x − 0.02x2 +1

(1 + x)53

s.t. x ≤ a

0 1 2 3 4 5−1.4

−1.2

−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

x

f(x)


fmincon

Sometimes it’s better to reformulate a constrained problem as anunconstrained

I Can use a formulation with Lagrange multipliers


fminsearch

Finds parameters that minimize a given function

Derivative-free algorithm (Nelder-Mead)

Syntax:x = fminsearch(objfun,x0,options)

I x0: starting guessI objfun: objective function specified in a different scriptI x: solution


fminsearch

Nelder-Mead algorithm:

1 select 2 points: x1 and x2

2 order s.t. x = x1 and x = x2 if f (x) ≥ f (x); and x = x2 and x = x1

otherwise

3 reflection: xR = x + (x − x)

4 if f (xR) > f (x) and f (xR) > f (x), expansion: xE = x + α(xR − x), α > 1if f (xR) ≤ f (x) and f (xR) > f (x), update x = xR and go to step 2if f (xR) ≤ f (x) and f (xR) ≤ f (x), contraction: xC = x + β(x − x), β < 1

2

5 if f (xE ) > f (xR), update x = xE and go to step 2if f (xE ) < f (xR), update x = xR and go to step 2

6 if f (xC ) > f (x), update x = xC and go to step 2if f (xC ) < f (x), shrink: replace x = x + γ(x − x) and go to step 2, γ < 1

Iterate until convergence: x ≈ x and/or f (x) ≈ f (x)


fminsearch

Example 1:

maxx

f (x) = 0.5 ln x − 0.02x2 +1

(1 + x)53

0 1 2 3 4 5−1.4

−1.2

−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

x

f(x)


fminsearch

Example 2. Discontinuous objective function:

maxx

f (x) =

{0.5 ln x − 0.02x2 + 1

(1+x)53, 0 < x ≤ 4

0.5 ln x − 0.02x2 + 1

(1+x)53

+ 1, 4 < x

0 1 2 3 4 5−1.5

−1

−0.5

0

0.5

1

1.5

x

f(x)


fminsearch

Example 3. Objective function with flat regions:

maxx

f (x) =

{0.5 ln x − 0.02x2 + 1

(1+x)53, 0 < x ≤ 4

0.441546, 4 < x

0 1 2 3 4 5−1.4

−1.2

−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

x

f(x)


fminsearch

Example 4. Objective function with local maxima:

maxx

f (x) = sin(3x) + x − 0.25x2

0 1 2 3 4 5−1

−0.5

0

0.5

1

1.5

2

x

f(x)


fminsearch

Example 5. Objective function with many discontinuities:

maxx

f (x) = bxc+ 0.2x − 0.04x2

0 5 10 15 200

1

2

3

4

5

6

7

8

9

f(x)

x


fminunc vs fminsearch

Rule of thumb: use fminunc for continuous differentiable functions,

fminsearch otherwise

No option guarantees that you find a (global) optimum

For complicated cases:I understand the problem wellI try different algorithmsI try different starting points


numerical optimization in matlab - andrii parkhomenko · 2017-03-31 · optimization typical...

Documents