

6BV04
DOE: Optimization
Response Surface Methods

 department of mathematics and computer science 1

  Contents
• Optimisation steps
• Box method
• Steepest ascent method
• Practical example
• Response surface designs
• Multiple responses
• EVOP
• Software
• Literature

 department of mathematics and computer science 2

 
Optimisation steps
Optimisation is achieved by going through the
following phases:

• screening (determine which factors really influence

the outcome; tool: screening designs like fractional
factorial)
• improvement (approach optimum by repeated
change of factor settings; tools: Box/simplex or
steepest ascent approach)
• determination of optimum (find optimal settings of
factor settings; tool: response surface designs like
CCD or Box-Behnken + analysis of response surface
using eigenvalues)

 department of mathematics and computer science 3

 
optimum

improvement

current settings

 department of mathematics and computer science 4

 
Regression models used in optimisation
Statistical techniques for optimisation assume the following (often
reasonably satisfied in practice):
“Far away” from the optimum a first order model often suffices.
for example:

Y = ß0 + ß1x1 + ß2x2 + 

“Near” the optimum often a quadratic (second order) model

suffices. For example:

Y = ß0 + ß1x1 + ß2x2 + ß12x1x2 + ß11x12 + ß22x22 +

Lack-of-fit techniques must be applied in order to check whether

these models are appropriate, since we cannot directly see
whether we are near the optimum (cf. next slides).

 department of mathematics and computer science 5

 
Models
Far away from
optimum:
first order
model

 department of mathematics and computer science 6

 
Models

Near optimum:
fitting a first
order model
shows lack-of-
fit
(curvature)

 department of mathematics and computer science 7

 
Models

Near optimum:
second order
model

 department of mathematics and computer science 8

  Improvement
In order to efficiently move from current factor settings to
factor setting that yield near-optimal values, 2 methods are
available:
• Box/Simplex method
– idea: form new full factorials in direction of largest increase
in current full factorial
– simple; no statistics needed for implementation
– not efficient
• Steepest ascent/descent method
– idea: use 1st order regression model from fractional factorial
to obtain direction of largest increase (“steepest ascent”)
– perform single runs in direction of largest increase until
increase stops
– advanced
– recommended since it is more efficient

 department of mathematics and computer science 9

  Box method
direction of
largest increase 40.6 41.9 41.2

direction of
largest increase

40.0 41.5 41.8 41.3

stop if one has
39.3 40.9 to return to
previous
settings
 department of mathematics and computer science 10
 
Steepest ascent method

perpendicular direction of
to contour line steepest ascent

contour lines of
first-order model

region where
1eorder-model


has been determined
department of mathematics and computer science 11
 
Optimization scheme
start end

screening accept
stationary point

no yes

full factorial no stationary yes stationary

1st order RSM design
+ centre points fit 2 nd
order model point point
model OK? (CCD, ...)
optimum? nearby?
yes
no
single observation
in direction go to
steepest ascent stationary point
yes

no
better
observation?

 department of mathematics and computer science 12

 Practical example
goal: maximise yield of chemical reactor

significant factors obtained after screening

experiment:
• reaction time
• reaction temperature

current factor setting: time = 35 min. temp = 155 °C

current yield: 40 %

 department of mathematics and computer science 13

 Steepest ascent
22-design with 5 centre points:
time: 30 - 40 min; temp: 150 - 160 °C

results: montgomery14-1.sfx

• there is no significant interaction

• there is no significant lack-of-fit
• the regression model is significant

Hence, we are not near the optimum.

 department of mathematics and computer science 14

 
Steepest ascent path
outcome analysis of measurement:
yield = 24.94 + 0.155*time + 0.065*temp

with coding:
x1= (time-35)/5 x2 = (temp-155)/5

yield = 40.44 + 0.775*x1 + 0.325*x2

�0.775 �
direction path: normal vector�0.325 �
� �
step size: 5 min reaction time (choice of chemical engineer!
5 0.325
= ޴ coded step size temp (= 2.1°C)
0.42
5 0.775
 department of mathematics and computer science 15
 
Steepest ascent path experiments

Further experiments with factor settings of experiment nr. 10.

 department of mathematics and computer science 16

 Near the optimum
Settings experiment 10:
• time = 85 min
• temperature = 175 °C

A 22 design with 5 centre points is executed.

results: montgomery14-4.sfx

Lack-of-fit indicates curvature. Hence, we now are

probably near the optimum.

 department of mathematics and computer science 17

 Quadratic models
In order to fit a quadratic model (suitable when we
are near the optimum), we must vary the factors at
3 levels.
A 2p-design with centre points does not suffice,
because then all quadratic factors are confounded.

A 3p-design is possible, but not to be recommended:

• number of runs grows fast
• uses more runs than necessary to fit quadratic
model.

 department of mathematics and computer science 18

 
Response surface designs
The following designs are widely used for fitting a
quadratic model:
• Central Composite Design (uniform precision of effect
estimates)
• Box-Behnken Design (almost uniform precision of effect
estimates, but usually fewer runs required than for CCD)

The choice between these models is usually

decided by the availability of these designs for a
given number of runs and number of factors.
Note that there are other suitable designs (usually
available in statistical software that supports DOE).

 department of mathematics and computer science 19

 
Central Composite Design

A CCD consists of 3 parts:

• factorial points
• centre points
• axial points

A CCD is often executed by adding

points to an already performed
2p-design (highly efficient, but beware
of blocking!).

 department of mathematics and computer science 20

  Rotatability
In a CCD there are 2 possible choices:
• number of centre points
• location axial points
By choosing the axial points at the locations (,0,…,0) etc.
with  = (# factorial points)¼ , the design becomes
rotatable, i.e. the precision (variance) of the model depends
on the distance to the origin only. In other words, one has
the same precision for all factor estimates.

 department of mathematics and computer science 21

 
Box-Behnken designs
These are designs that
consists of
combinations from 2p-designs.

Properties:
• efficient (few runs)
• (almost) rotatable
• no corner points of
hypercube
(these are extreme
conditions
which are often hard to set)

 department of mathematics and computer science 22

 Stationary point
Near the optimum usually a quadratic model
suffices:

k k
Y =  0    i xi    ii xi2  i  j   ij xi x j  
i =1 i =1

How do find the optimum after we correctly

estimated the parameters using a response
surface design (CCD or Box-Behnken)?
The next slides show the tools to derive optimal
settings and the pitfalls that have to be avoided.
 department of mathematics and computer science 23
 
Recap: optimisation in dimension1
• necessary condition for extremum: 1st derivative = 0
• not sufficient:
“point of inflection”

• extra sufficient condition: 2nd derivative  0

 department of mathematics and computer science 24

 
Zero first derivatives: saddlepoint vs.
maximum

saddle point (unfavourable) maximum (favourable)

x -y
2 2
-( x  y
2 2
)
 department of mathematics and computer science 25
 
Determination of type of optimum
• Graphically: make contourplot (if 2
factors)
• Analytically:
  k  k  
Y =  0    i xi    x  i j   ij xi x j
2
ii i
i =1 i =1

 
matrix notation: Y =  0  x T
b  x T
Bx
Note: B must be chosen �� � �
�
as symmetric matrix, 
� 11  12 / 2  13 / 2 �
�� � � �
see example: � 12 / 2  22  23 / 2 �
�� � � �
� 13 / 2  23 / 2  33 �
� �

 department of mathematics and computer science 26

 
Stationarity and matrix analysis
stationary point (zero first-order derivatives):

Y 1 -1
= b  2 Bx = 0  x0 = - B b
x 2

characterisation through eigenvalues of matrix B:

Bx = x • all eigenvalues positive: min

• all eigenvalues negative: max
• eigenvalues different signs: saddle point

the ’s are sometimes called “parameters of canonical form”

 department of mathematics and computer science 27
 
Stationarity and matrix analysis
n StatGraphics:
augment design
add star points
 Please note that additional centre points are added
and a block variable.
 We can remove the centre points from the data set
and ignore the block variable in the analysis.

StatGraphics results: montgomery14-6.sfx

 department of mathematics and computer science 28

 
Stationarity and matrix analysis
se Matlab to avoid manual computations:
create matrix B and vector b
compute eigenvalues and location of stationary point
>> B = [-2.75247 0.5/2 ; 0.5/2 -2.00253] / 2
Analysis Summary
----------------
File name: D:\MyDocs\2DS01\collegesheets\montgomery14-6.sfx
Comment: Montgomery table 14-4

Estimated effects for opbrengst

----------------------------------------------------------------------
average = 79.94 +/- 0.11896
A:tijd = 1.98994 +/- 0.188092
B:temperatuur = 1.03033 +/- 0.188093
AA = -2.75247 +/- 0.201705
AB = 0.5 +/- 0.266003
BB = -2.00253 +/- 0.20171
----------------------------------------------------------------------


Standard errors are based on total error with 7 d.f.

department of mathematics and computer science 29

 
Stationarity and matrix analysis
n Matlab:
create matrix B and vector b
compute eigenvalues and location of stationary point
>> B = [-2.75247 0.5/2 ; 0.5/2 -2.00253] / 2

B =

-1.3762 0.1250
0.1250 -1.0013

 department of mathematics and computer science 30

 
Stationarity and matrix analysis
n Matlab:
create matrix B and vector b
compute eigenvalues and location of stationary point
>> eig(B)

 department of mathematics and computer science 31

 
Stationarity and matrix analysis
n Matlab:
create matrix B and vector b
compute eigenvalues and location of stationary point
>> eig(B)

ans =

-1.4141 both negative → maximum

-0.9634

 department of mathematics and computer science 32

 
Stationarity and matrix analysis
n Matlab:
create matrix B and vector b
compute eigenvalues and location of stationary point
>> b = [1.98994 ; 1.03033] /2
Analysis Summary
----------------
File name: D:\MyDocs\2DS01\collegesheets\montgomery14-6.sfx
Comment: Montgomery table 14-4

Estimated effects for opbrengst

----------------------------------------------------------------------
average = 79.94 +/- 0.11896
A:tijd = 1.98994 +/- 0.188092
B:temperatuur = 1.03033 +/- 0.188093
AA = -2.75247 +/- 0.201705
AB = 0.5 +/- 0.266003


BB = -2.00253 +/- 0.20171
----------------------------------------------------------------------
Standard errors are based on total error with 7 d.f. 33
department of mathematics and computer science
 
Stationarity and matrix analysis
n Matlab:
create matrix B and vector b
compute eigenvalues and location of stationary point
>> b = [1.98994 ; 1.03033] /2

b =

0.9950
0.5152

 department of mathematics and computer science 34

 
Stationarity and matrix analysis
n Matlab:
create matrix B and vector b
compute eigenvalues and location of stationary point
>> spcoded = -0.5 * inv(B) * b

 department of mathematics and computer science 35

 
Stationarity and matrix analysis
n Matlab:
create matrix B and vector b
compute eigenvalues and location of stationary point
>> spcoded = -0.5 * inv(B) * b

spcoded =

0.3893 < 1.414 (distance star point)

0.3059 → inside experimental region

 department of mathematics and computer science 36

 
Stationarity and matrix analysis
n Matlab:
create matrix B and vector b
compute eigenvalues and location of stationary point
>> sporiginal = spcoded .* [5 ; 5] + [85 ; 175]

time - 85 temp - 175

x1 = , x2 =
5 5
time = 5 x1  85, temp = 5 x2  175
 department of mathematics and computer science 37
 
Stationarity and matrix analysis
n Matlab:
create matrix B and vector b
compute eigenvalues and location of stationary point
>> sporiginal = spcoded .* [5 ; 5] + [85 ; 175]

sporiginal =

86.9463
176.5293

 department of mathematics and computer science 38

 
Optimization scheme
start end

screening accept
stationary point

no yes

full factorial no stationary yes stationary

1st order RSM design
+ centre points fit 2 nd
order model point point
model OK? (CCD, ...)
optimum? nearby?
yes
no
single observation
in direction go to
steepest ascent stationary point
yes

no
better
observation?

 department of mathematics and computer science 39

 
Multiple responses
If more than 1 response variable needs to be
optimised, then a graphical way of optimising may
be achieved by overlaying contour plots in case
there are only 2 independent variables.

Overlay Plot
te m p e r a tu u r

180
molecular_weight
178 opbrengst
viscositeit
176

174

172

170
80 82 84 86 88 90
tijd

 department of mathematics and computer science 40

 
Evolutionary Operation (EVOP)

Optimisation of a running production process is not

always possible or may not be allowed because of costs:
• involves interruption
• may (temporarily) yield low quality products

An alternative is Evolutionary Operation:

• experimentation within running operation
• frequent execution of 2k-designs, starting at current
settings
• high and low setting of factors are close to each other,
thus no risk of low quality products

 department of mathematics and computer science 41

  Software
• StatLab optimisation:
http://www.win.tue.nl/statlab
Interactive software for teaching DOE through cases
• Box: http://www.win.tue.nl/~marko/box/box.html
Game-like demonstration of Box method
• Matlab virtual reactor: Statistics toolbox -> Demos ->
Empirical Modeling -> RSM demo
• Statgraphics:
menu choice Special -> Experimental Design
– design experiment with pre-defined catalogue
– analysis of experiments with ANOVA

 department of mathematics and computer science 42

  Literature
• J. Trygg and S. Wold. Introduction to Experimental
Design – What is it? Why and Where is it Useful?,
Homepage of Chemometrics, editorial August 2002:
http://www.acc.umu.se/~tnkjtg/Chemometrics/editori
al/aug2002.html

• DOE booklet from Umetrics: http://www.umetrics.com

/pdfs/books/DOEbook.pdf
• Introduction to DOE from moresteam.com http://
www.moresteam.com/toolbox/t408.cfm
• StatSoft Electronic Statistics Textbook, chapter on
experimental design
• NIST Engineering Statistics Handbook:
http://www.itl.nist.gov/div898/handbook/
 department of mathematics and computer science 43