Numerical Optimization in MATLAB

Andrii Parkhomenko

Universitat Autònoma de Barcelona and Barcelona GSE

Spring 2017

Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 0 / 24

Optimization

Typical problem in economics: find an optimum

Case 1:
max f (x) = 0.5x − 0.02x 2
x

Easy: just solve for x that satisfies f 0 (x) = 0

Case 2:
1
max f (x) = 0.5 ln x − 0.02x 2 + 5
x (1 + x) 3
Can’t solve analytically ⇒ need to use numerical methods

Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 1 / 24

Numerical Optimization

Continuous differentiable objective function: derivative-based methods

Arbitrary objective function: derivative-free methods

Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 2 / 24

Numerical Optimization in MATLAB

These slides cover three functions contained in MATLAB’s Optimization

Toolbox:
fminunc: unconstrained optimization, derivative-based
fmincon: constrained optimization, derivative-based
fminsearch: unconstrained optimization, derivative-free

Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 3 / 24

fminunc

Finds parameters that minimize a given function

Syntax:
x = fminunc(objfun,x0)
I x0: starting guess
I objfun: objective function specified in a different script
I x: solution

Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 4 / 24

fminunc

Example 1:
1
max f (x) = 0.5 ln x − 0.02x 2 + 5
x (1 + x) 3

0.6

0.4

0.2

−0.2
f(x)

−0.4

−0.6

−0.8

−1

−1.2

−1.4
0 1 2 3 4 5
x

Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 5 / 24

fminunc

Example 1:
1
max f (x) = 0.5 ln x − 0.02x 2 + 5
x (1 + x) 3

Step 1. Main code

x0 = 1;
x = fminunc(@objfun,x0)

Step 2. Function
function y = objfun(x)
y = 0.5*log(x) - 0.02*x^2 + (1+x)^(-5/3);
y = -y;
end

Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 6 / 24

fminunc

fminunc uses the BFGS Quasi-Newton method:

xk+1 = xk + αk pk

1 Compute direction pk

pk = −Bk−1 ∇fk
Bk sk sk0 Bk yk yk0
Bk+1 = Bk − +
sk0 Bk sk yk0 sk
where sk ≡ xk+1 − xk and yk ≡ fk+1 − fk

2 Compute stepsize αk . Use line search to approximately solve

min f (xk + αk pk )
αk

Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 7 / 24

fminunc

Introduce options:
options = optimset(’Opt1’,Opt1Val,’Opt2’,Opt2Val,...);
x = fminunc(@objfun,x0, options)

Important algorithm options:

Display:
optimset(’Display’, ’off’)
Tolerance level:
optimset(’TolFun’, 1e-6)
optimset(’TolX’, 1e-6)
Maximum number of iterations or evaluations
optimset(’MaxIter’, 400)
optimset(’MaxFunEvals’, 100)
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 8 / 24
fminunc

Caution:
Objective function must be continuous
Local minima and flat regions are big issues

Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 9 / 24

fminunc

Example 2. Discontinuous objective function:

0.5 ln x − 0.02x 2 + 1 5 , 0<x ≤4
(
(1+x) 3
max f (x) =
x 0.5 ln x − 0.02x 2 + 1 5 + 1, 4<x
(1+x) 3

1.5

0.5
f(x)

−0.5

−1

−1.5
0 1 2 3 4 5
x

Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 10 / 24

fminunc

Example 3. Objective function with flat regions:

0.5 ln x − 0.02x 2 + 1 5 , 0 < x ≤ 4
(
max f (x) = (1+x) 3
x 0.441546, 4<x

0.6

0.4

0.2

−0.2
f(x)

−0.4

−0.6

−0.8

−1

−1.2

−1.4
0 1 2 3 4 5
x

Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 11 / 24

fminunc

Example 4. Objective function with local maxima:

max f (x) = sin(3x) + x − 0.25x 2

x

1.5

1
f(x)

0.5

−0.5

−1
0 1 2 3 4 5
x

Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 12 / 24

fminunc

To obtain more details on the workings of fminunc:

Read documentation: Help → Type ’fminunc’
Read the code: Type ’open fminunc’

Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 13 / 24

fmincon

Finds parameters that minimize a given function, subject to a

constraint
Same algorithm as fminunc
Syntax:
x = fmincon(objfun,x0,A,b,Aeq,beq,lb,ub,nlcon,options)
I x0: starting guess
I objfun: objective function specified in a different script
I A, b: linear inequality constraint: Ax ≤ b
I Aeq, beq: linear equality constraint: Ax = b
I lb, ub: lower and upper bounds: x ∈ [lb, ub]
I nlcon: nonlinear constraint
I x: solution

Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 14 / 24

fmincon

Example 1:
1
max f (x) = 0.5 ln x − 0.02x 2 + 5 s.t. x ≤ a
x (1 + x) 3

0.6

0.4

0.2

−0.2
f(x)

−0.4

−0.6

−0.8

−1

−1.2

−1.4
0 1 2 3 4 5
x
Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 15 / 24
fmincon

Sometimes it’s better to reformulate a constrained problem as an

unconstrained
I Can use a formulation with Lagrange multipliers

Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 16 / 24

fminsearch

Finds parameters that minimize a given function

Derivative-free algorithm (Nelder-Mead)
Syntax:
x = fminsearch(objfun,x0,options)
I x0: starting guess
I objfun: objective function specified in a different script
I x: solution

Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 17 / 24

fminsearch
Nelder-Mead algorithm:
1 select 2 points: x1 and x2
2 order s.t. x̂ = x1 and x̄ = x2 if f (x̂) ≥ f (x̄); and x̂ = x2 and x̄ = x1
otherwise
3 reflection: x R = x̂ + (x̂ − x̄)
4 if f (x R ) > f (x̂) and f (x R ) > f (x̄), expansion: x E = x̂ + α(x R − x̂), α > 1
if f (x R ) ≤ f (x̂) and f (x R ) > f (x̄), update x̄ = x R and go to step 2
if f (x R ) ≤ f (x̂) and f (x R ) ≤ f (x̄), contraction: x C = x̂ + β(x̄ − x̂), β < 21
5 if f (x E ) > f (x R ), update x̄ = x E and go to step 2
if f (x E ) < f (x R ), update x̄ = x R and go to step 2
6 if f (x C ) > f (x̄), update x̄ = x C and go to step 2
if f (x C ) < f (x̄), shrink: replace x̄ = x̂ + γ(x̄ − x̂) and go to step 2, γ < 1
Iterate until convergence: x̂ ≈ x̄ and/or f (x̂) ≈ f (x̄)

Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 18 / 24

fminsearch

Example 1:
1
max f (x) = 0.5 ln x − 0.02x 2 + 5
x (1 + x) 3

0.6

0.4

0.2

−0.2
f(x)

−0.4

−0.6

−0.8

−1

−1.2

−1.4
0 1 2 3 4 5
x

Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 19 / 24

fminsearch

Example 2. Discontinuous objective function:

0.5 ln x − 0.02x 2 + 1 5 , 0<x ≤4
(
(1+x) 3
max f (x) =
x 0.5 ln x − 0.02x 2 + 1 5 + 1, 4<x
(1+x) 3

1.5

0.5
f(x)

−0.5

−1

−1.5
0 1 2 3 4 5
x

Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 20 / 24

fminsearch

Example 3. Objective function with flat regions:

0.5 ln x − 0.02x 2 + 1 5 , 0 < x ≤ 4
(
max f (x) = (1+x) 3
x 0.441546, 4<x

0.6

0.4

0.2

−0.2
f(x)

−0.4

−0.6

−0.8

−1

−1.2

−1.4
0 1 2 3 4 5
x

Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 21 / 24

fminsearch

Example 4. Objective function with local maxima:

max f (x) = sin(3x) + x − 0.25x 2

x

1.5

1
f(x)

0.5

−0.5

−1
0 1 2 3 4 5
x

Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 22 / 24

fminsearch

Example 5. Objective function with many discontinuities:

max f (x) = bxc + 0.2x − 0.04x 2

x

5
f(x)

0
0 5 10 15 20
x

Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 23 / 24

fminunc vs fminsearch

Rule of thumb: use fminunc for continuous differentiable functions,

fminsearch otherwise
No option guarantees that you find a (global) optimum
For complicated cases:
I understand the problem well
I try different algorithms
I try different starting points

Andrii Parkhomenko (UAB & Barcelona GSE) Numerical Optimization in MATLAB 24 / 24