ME 310

Numerical Methods

Optimization

These presentations are prepared by

Dr. Cuneyt Sert
Mechanical Engineering Department
Middle East Technical University
Ankara, Turkey
[email protected]

They can not be used without the permission of the author

1
 Optimization
• Optimization is similar to root finding. Both involve guessing and searching for a point on a function.

Global maximum
f(x) f (x) = 0 , f (x) < 0

a b x

Global minimum
f (x) = 0 , f (x) > 0

• Optimum is the point where f (x) = 0. f (x) indicates whether it is a minimum or a maximum.
• In this range there can be only one global minimum and one global maximum. These can be at the
end points of the interval.
• There can be several local minimums and local maximums.
• Practical optimization problems are complicated and include several constraints.
• Minimization of cost of a manufactured part (time, quality, quantity, etc.).
• Maximization of efficiency of a cooling unit (size, material, safety, ergonomics, cost, etc.).
• Transportation problem (manage the shipping between several sources and several warehouses).
2
• We will study 1D (f=f(x)), unconstrained optimization using the following methods.
• Bracketing methods (Golden Section Search, Quadratic Interpolation)
• Open Methods (Newton’s Method)

Bracketing Methods
• Consider finding the maximum of a function f(x) in the interval [a,b].
• Consider a function that has only one maximum (or minimum) in [a,b].
• Similar to the bracketing methods used for root finding, we will iteratively narrow the interval [a,b]
to locate the minimum.
• Remember that in root finding (for example in the Bisection method), only one intermediate point
was enough to narrow the interval.
• In finding a maximum we need two intermediate points (x1 and x2).
• If f(x1) > f(x2) than the maximum is between [x2,b]. Otherwise it is between [a,x1]

a x2 x1 b a x2 x1 b a x2 x1 b a x2 x1 b 3
 Golden Section Search
• There are several different ways in selecting the two intermediate points x1 and x2.
• In Golden Section Search these two points are selected as

x1 = a + d
x2 = b – d

a x2 x1 b where d = R (b-a).
d

5 1
• R  0.618034 . . . is called the golden-ratio. It is the positive root of r2 + r – 1 = 0.
2
• If f(x1) > f(x2) than continue with the interval [x2,b]. Otherwise continue with [a,x1]. This works to
locate a maximum. To locate a minimum do the opposite.
• Calculate two new intermediate points in the narrowed interval and iterate like this.
• At each iteration, the interval drops by a factor of R ( d i+1 = R d i ).
• Stop when (x1-x2) drops below the specified tolerance. See page 347 for an alternative stopping
criteria.
4
 Golden Ratio
• What is the importance of the golden ratio, R = 0.618034 ?
• Consider the following case. Superscripts show the iteration number.

Iteration 0 • At iteration 0,
5 1 0 5 1 0
x 10  a 0  (b  a 0 ) , x 02  b 0  (b  a 0 )
2 2

• f(x10) < f(x20) , therefore continue with [x20,b0].

a0 x20 x10 b0

d0
d0 • At iteration 1, a1= x20 , b1= b0

5 1 1
x 11  a1  (b  a1 )
2
Iteration 1 5 1 1
x 12  b1  (b  a1 )  x 10
2

• Therefore there is no need to calculate x21 and f(x21).

This saves computations.
a1 x21 x11 b1

d1 Exercise: Show that x21 = x10

d1
5
Pseudocode for the Golden Section Search
R = 0.618033988
READ a, b, maxIter, tolerance
CALCULATE fa, fb

LOOP k from 1 to maxIter Exercise 21: This pseudocode is written to

x1 = a + R(b-a) ; f1 = func(x1) find a maximum. Modify it so that it can find a
minimum too.
x2 = b - R(b-a) ; f2 = func(x2)

IF ( f1 > f2) THEN Exercise 22: Change the stopping criteria

a = x2 ; x2 = x1 ; f2 = f1 with the one given in the book.
x1 = a + R(b-a) ; f1 = func(x1)
ELSE Exercise 23: What happens if
b = x1 : x1 = x2 ; f1 = f2 (i) the initial interval [a,b] contains more than
x2 = b - R(b-a) ; f2 = func(x2) one min or max?
ENDIF (ii) a is the maximum and b is the minimum,
or vice versa?
WRITE k, x1, x2
IF ( (x1 – x2) < tolerance) STOP

ENDLOOP 6
 Quadratic Interpolation
• Based on the fact that a quadratic (2nd order) polynomial often provides a good approximation of a
function near an optimum point.

f(x)
quadratic polynomial

x0 x1 x3 x2

• Select 3 points (x0, x1 and x2) that contains only 1 optimum point of a function.
• Only one quadratic will pass through these points. Find the equation of this quadratic.
• Equate its first derivative to zero and find its optimum point, x3.

f (x 0 )( x 12  x 22 )  f (x 1 )( x 22  x 20 )  f (x 2 )( x 20  x 12 )
x3 
2 f (x 0 )( x 1  x 2 )  2 f (x 1 )( x 2  x 0 )  2f (x 2 )( x 0  x 1 )

• Similar to the Golden Section Search, narrow the interval by discarding one of the points.
• Continue with the remaining 3 points and calculate a new optimum (x3).
• Iterate like this and stop when the approximate relative error drops below the tolerance value. 7
 Newton’s Method
f (x i )
• Recall that Newton-Raphson method is used to find the root of f(x) =0 as xi1  xi 
f (x i )

f (x i )
• Similarly the optimum points of f(x) can be found by applying N-R to f (x) = 0. x i1  x i 
f (x i )

• This open method requires only one starting point.

• It also requires the 1st and 2nd derivative of f(x).

• It converges fast, but convergence is not guaranteed.

• At the end one can check the sign of f (x) to determine whether the optimum point is a minimum
or a maximum.

• If the derivatives are not known than their approximations can be used. This is similar to the
Secant method that we learned in root finding.

• To avoid divergence, it is a good idea to use this method when we are close enough to the
optimum point. So we can use a hybrid technique, where we start with a bracketing method and
safely narrow the interval and than continue with the Newton’s method.

8
Example 23:

Find the maximum of f(x) = 2 x - 1.75 x2 + 1.1 x3 – 0.25 x4 using

(a) Golden section search (a = -2 , b = 4, es = 1%)
(b) Quadratic interpolation (x0 = -1.75 , x1 = 2 , x2 = 2.25 , perform 5 iterations)
(c) Newton’s method (x0 = 2.5 , es = 1%)

(a) Golden Section Search

iter 1: a = -2 , b=4, x1 = a + R(b-a) = 1.708 , x2 = b - R(b-a) = 0.292

f(x1) = 1.664 , f(x2) = 0.460 f(x1) > f(x2) than continue with [x2 , b].
f(x1) > f(x2) than xopt = x1 = 1.708 , ea = (1-R) * (b-a) / |xopt|* 100 = 134 %

iter 2: a = 0.292 , b=4, x1 = a + R(b-a) = 2.584 , x2 = 1.708

f(x1) = 1.316 , f(x2) = 1.664 f(x1) < f(x2) than continue with [a , x1 ].
f(x1) < f(x2) than xopt = x2 = 1.708 , ea = (1-R) * (b-a) / |xopt|* 100 = 83 %

iter 3: a = 0.292 , b = 2.584 , x1 = 1.708 , x2 = b - R(b-a) = 1.167

f(x1) = 1.664 , f(x2) = 1.235 f(x1) > f(x2) than continue with [x2 , b].
f(x1) > f(x2) than xopt = x1 = 1.708 , ea = (1-R) * (b-a) / |xopt|* 100 = 51 %
......

iter 11: xopt = 2.073 , ea = 0.9 %

9
Example 23 (cont’d)

(b) Quadratic Interpolation

iter 1: x0 = -1.75 , x1 = 2.0 , x2 = 2.25

Calculate x3 = 2.0617

iter 2: x0 = 2.0 , x1 = 2.0617 , x2 = 2.25

Calculate x3 = 2.0741

iter 3: x0 = 2.0617 , x1 = 2.0741 , x2 = 2.25

Calculate x3 = 2.0779

iter 4: x0 = 2.0741 , x1 = 2.0779 , x2 = 2.25

Calculate x3 = 2.0791

iter 5: x0 = 2.0779 , x1 = 2.0791 , x2 = 2.25

Calculate x3 = 2.0786
ea = | (x3present - x3previous) / x3present |* 100 = 0.02 %

10
Example 23 (cont’d)

(c) Newton’s Method

f (x) = 2 – 3.5 x + 3.3 x2 – x3 , f (x) = -3.5 +6.6 x – 3 x2

x0 = 2.25

iter 1: x1 = x0 - f (x0) / f (x0) = 2.19565

ea = | (x1 - x0) / x1 |* 100 = 13.9 %

iter 2: x2 = x1 - f (x1) / f (x1) = 2.0917

ea = | (x2 - x1) / x2 |* 100 = 5.0 %

iter 3: x3 = x2 - f (x2) / f (x2) = 2.07951

ea = | (x3 - x2) / x3 |* 100 = 0.6 %

11