Chapter 13

Design of Experiments (DoE)

Prof. Dr. Mesut Güneş ▪ Ch. 13 Design of Experiments 13.1

 Contents
•  Introduction to DoE
•  Types of experimental designs
•  2k Factorial design
•  2kr Factorial design with replications
•  2k-p Fractional factorial design

Prof. Dr. Mesut Güneş ▪ Ch. 13 Design of Experiments 13.2

 Introduction to DoE

Prof. Dr. Mesut Güneş ▪ Ch. 13 Design of Experiments 13.3

 Design of Experiments
•  Example: Study the performance of a system in respect to
particular parameters
• System: routing algorithm for a MANET
• Parameters:
•  Number of nodes: N = {10, 20, 50, 100, 1000, 10000}
•  Mobility: M = {1 m/s, 3 m/s, 5 m/s, 10 m/s}
•  Packet size: P = {64 byte, 256 byte, 512 byte, 1024 byte}
•  Number of parallel flows: F = {1, 3, 5, 7, 10}
• Parameter space: N x M x P x F = 6 x 4 x 4 x 5 = 480

•  Question: how to perform the experiments to understand

the effects of the parameters?

Prof. Dr. Mesut Güneş ▪ Ch. 13 Design of Experiments 13.4

 Design of Experiments
•  Answer: Design of Experiments (DoE)

• The goal is to obtain

maximum information
with the
minimum number of experiments

Prof. Dr. Mesut Güneş ▪ Ch. 13 Design of Experiments 13.5

 Terminology

Response variable: The outcome of an experiment

Factor: Each variable that affects the response variable
and has several alternatives
Level: The values that a factor can assume
Primary Factor: The factors whose effects need to be quantified
Secondary Factor: Factors that impact the performance but whose
impact we are not interested in quantifying
Replication: Repetition of all or some experiments
Experimental Unit: Any entity that is used for the experiment
Interaction: Two factors A and B interact if the effect of one
depends upon the level of the other

Prof. Dr. Mesut Güneş ▪ Ch. 13 Design of Experiments 13.6

 Interaction of factors

B1 A1
No Interaction
B2 A2

A1 A2 B1 B2

B1 A1

B2
Interaction A2

A1 A2 B1 B2

Prof. Dr. Mesut Güneş ▪ Ch. 13 Design of Experiments 13.7

 Design
•  Design: An experimental design consists of specifying the
number of experiments, the factor level combinations for
each experiment, and the number of replications.

•  In planning an experiment, you have to decide

1.  what measurement to make (the response)
2.  what conditions to study
3.  what experimental material to use (the units)

•  Example
1.  Measure goodput and overhead of a routing protocol
2.  Network with n nodes in chain
3.  Routing protocol, type of nodes, type of links, traffic

Prof. Dr. Mesut Güneş ▪ Ch. 13 Design of Experiments 13.8

 Types of experimental designs

Prof. Dr. Mesut Güneş ▪ Ch. 13 Design of Experiments 13.9

 Types of experimental designs: Simple design
•  Simple design
• Start with a configuration and vary one factor at a time
• Given k factors and the i-th factor having ni levels
• The required number of experiments
k
n = 1 + ∑ (ni − 1)
i =1

• Example:
•  k=3, {n1=3, n2=4, n3=2}
•  n = 1+ (2 + 3 + 1) = 7

Prof. Dr. Mesut Güneş ▪ Ch. 13 Design of Experiments 13.10

 Types of experimental designs: Full factorial design
•  Full factorial design
• Use all possible combinations at all levels of all factors
• Given k factors and the i-th factor having ni levels
• The required number of experiments
k
n = ∏ ni
i =1

• Example:
•  k=3, {n1=3, n2=4, n3=2}
•  n = 3×4×2 = 24

Prof. Dr. Mesut Güneş ▪ Ch. 13 Design of Experiments 13.11

 Types of experimental designs
Fractional factorial design
•  Fractional factorial design
• When full factorial design results in a huge number of
experiments, it may be not possible to run all
• Use subsets of levels of factors and the possible combinations
of these
• Given k factors and the i-th factor having ni levels, and
selected subsets of levels mi ≤ ni .
• The required number of experiments
k
n = ∏ mi
i =1

• Example:
•  k=3, {n1=3, n2=4, n3=2}, but use {m1=2, m2=2, m3=1}
•  n = 2×2×1 = 4

Prof. Dr. Mesut Güneş ▪ Ch. 13 Design of Experiments 13.12

 Types of experimental designs
•  Comparison of the design types

Design Type Factors Number of

experiments
Simple design k=3, {n1=3, n2=4, n3=2} 7
Full factorial design 24
Fractional factorial design Use subset 4
{m1=2, m2=2, m3=1}

Prof. Dr. Mesut Güneş ▪ Ch. 13 Design of Experiments 13.13

 2k Factorial Designs

Prof. Dr. Mesut Güneş ▪ Ch. 13 Design of Experiments 13.14

 2k Factorial Designs
•  A 2k factorial design is used to determine the effect of k
factors
• Each factor has two levels
•  Advantages
• It is easy to analyze
• Helps to identify important factors
Æreduce the number of factors
• Often effect of a factor is unidirectional, i.e., performance
increase or decrease
• Begin by experimenting at the minimum and maximum level
of a factor Æ two levels

Prof. Dr. Mesut Güneş ▪ Ch. 13 Design of Experiments 13.15

 2k Factorial Designs
Example for k=2
•  Study impact of memory Factor 1
and cache on performance
of a workstation Memory Size

•  Memory size, two levels 4 MB 16 MB

•  Cache size, two levels Cache 1 15

-1,-1 45
1,-1
Factor 2 Size 2 25
-1,1 1,175

•  Performance of
workstation as regression
model ⎧− 1 if 4MB memory
x A = ⎨
⎩ 1 if 16MB memory
y = q0 + q A x A + qB xB + q AB x A xB
⎧− 1 if 1kb cache
xB = ⎨
⎩ 1 if 2kb cache

Prof. Dr. Mesut Güneş ▪ Ch. 13 Design of Experiments 13.16

 2k Factorial Designs
Example for k=2
•  Regression model
y = q0 + q A x A + qB xB + q AB x A xB Experiment A B y AB

•  Substitute the results into 1 -1 -1 y1 1

the model 2 1 -1 y2 -1
y1 = q0 − q A − qB + q AB 3 -1 1 y3 -1
y2 = q0 + q A − qB − q AB 4 1 1 y4 1
y3 = q0 − q A + qB − q AB
y4 = q0 + q A + qB + q AB

•  Solve equantions for qi

q0 = 14 ( y1 + y2 + y3 + y4 )
q A = 14 (− y1 + y2 − y3 + y4 )
y = 40 + 20xA + 10xB + 5xA xB
qB = 14 (− y1 − y2 + y3 + y4 )
q AB = 14 ( y1 − y2 − y3 + y4 )

Prof. Dr. Mesut Güneş ▪ Ch. 13 Design of Experiments 13.17

 2k Factorial Designs
Example for k=2: Sign table method
•  Sign table contains the effect of factors

I A B AB y
1 -1 -1 1 15
1 1 -1 -1 45
1 -1 1 -1 25
1 1 1 1 75
160 80 40 20 Total
40 20 10 5 Total/4 Result

Prof. Dr. Mesut Güneş ▪ Ch. 13 Design of Experiments 13.18

 2k Factorial Designs
Example for k=2: Allocation of variation
•  Determine the importance of a factor
• Calculate the variance
22

∑(y i − y)2
s y2 = i =1
22 − 1
• Sum of squares total (SST): Total variation of y
22
y = SST = ∑ ( yi − y ) 2
i =1

• For 22 design, the variation is given by

SST = 2 2 q A2 + 2 2 qB2 + 22 q AB
2
  
SSA SSB SSAB

• SSA: part explained by factor A

• Fraction of variation explained by A: SSA/SST

Prof. Dr. Mesut Güneş ▪ Ch. 13 Design of Experiments 13.19

 2k Factorial Designs
The General Case
•  In the general case there are k factors, each factor has
two levels
•  A total of 2k experiments are required
•  Analysis produces 2k effects (results)
•  k main effects
•  ⎛⎜⎝ k2 ⎞⎟⎠ two-factor interactions
•  ⎛⎜⎝ k3 ⎞⎟⎠ three-factor interactions
• …

•  Sign table method is used!

Prof. Dr. Mesut Güneş ▪ Ch. 13 Design of Experiments 13.20

 2k Factorial Designs
The General Case
•  Sign table, example for k=3

I A1 A2 A3 A1A2 A1A3 A2A3 A1A2A3 y

+ - - - + + + - y1
+ + - - - - + + y2
+ - + - - + - + y3
+ + + - + + - - y4
+ - - + + + - + y5
+ + - + - - - - y6
+ - + + - - + - y7
+ + + + + + + + y8

Prof. Dr. Mesut Güneş ▪ Ch. 13 Design of Experiments 13.21

 2k Factorial Designs
The General Case
•  Sign table

I A1 A2 A3 … A1A2 A1A3 … A1A2A3 … y

1 -1 y1
1 1 y2
1 -1 y3
… … …

SumI Total
SumI/2k Total/2k

Prof. Dr. Mesut Güneş ▪ Ch. 13 Design of Experiments 13.22

 2kr Factorial Design with Replications

Prof. Dr. Mesut Güneş ▪ Ch. 13 Design of Experiments 13.23

 2kr Factorial Design with Replications
•  Problem with 2k factorial design is that it does not provide
the estimation of experimental errors, since no repetitions
•  Solution: Repeat an experiment r times Æ replication
• If each of the 2k experiments is repeated r times
Æ 2kr factorial design with replications
•  Extended model
y = q0 + q A x A + qB xB + q AB x A xB + e

Experimental error

Prof. Dr. Mesut Güneş ▪ Ch. 13 Design of Experiments 13.24

 2kr Factorial Design with Replications
•  For analysis, the same method is used, except for y, the
mean of the replications is used.
I A B AB y y
1 -1 -1 1 (15,18,12) 15
1 1 -1 -1 (45,48,51) 48
1 -1 1 -1 (25,28,19) 24
1 1 1 1 (75,75,81) 77
164 86 38 20 Total
41 21.5 9.5 5 Total/4

•  Experimental error is given: eij = yij − y

•  Sum of squared errors (SSE) and the standard deviation
of errors: 22 r
SSE
SSE = ∑∑ eij2 se =
i =1 j =1 2 2 ( r − 1)

Prof. Dr. Mesut Güneş ▪ Ch. 13 Design of Experiments 13.25

 2k-p Fractional Factorial Design

Prof. Dr. Mesut Güneş ▪ Ch. 13 Design of Experiments 13.26

 2k-p Fractional Factorial Design
•  When the number of factors is large, a full factorial design
requires a large number of experiments
•  In that case fractional factorial design can be used
• Requires fewer experiments, e.g., 2k-1 requires half of the
experiments as a full factorial design

Prof. Dr. Mesut Güneş ▪ Ch. 13 Design of Experiments 13.27

 2k-p Fractional Factorial Design
•  Preparing the sign table
• Choose k-p factors and prepare a complete sign table.
ÆSign table with 2k-p rows and 2k-p columns
• The first column will be marked I and consists of all 1s
• The next k-p columns will be marked with the k-p factors that
were chosen
• The remaining columns are simply products of these factors

Prof. Dr. Mesut Güneş ▪ Ch. 13 Design of Experiments 13.28

 2k-p Fractional Factorial Design
•  Sign table, example for k =7, p =4 Æ27-4=23
k-p chosen factors products of chosen factors

I F1 F2 F3 F1F2 F1F3 F2F3 F1F2F3

+ - - - + + + -
+ + - - - - + +
+ - + - - + - +
+ + + - + + - -
2k-p rows
+ - - + + + - +
+ + - + - - - -
+ - + + - - + -
+ + + + + + + +

2k-p columns

Prof. Dr. Mesut Güneş ▪ Ch. 13 Design of Experiments 13.29

 2k-p Fractional Factorial Design
•  Confounding
• with fractional factorial design some of the effects can not be
determined
• only combined effects of several factors can be computed
•  A fractional factorial design is not unique

•  Design resolution
• The resolution of a design is measured by the order of
effects that are confounded
• The order of effect is the number of factors included in it
I = ABC order of 3 ÆResolution RIII
I = ABCD order of 4 ÆResolution RIV
• A design of higher resolution is considered a better design.

Prof. Dr. Mesut Güneş ▪ Ch. 13 Design of Experiments 13.30

 Summary
•  Design of experiments provides a method for planned
experiments
•  Goal: Obtain maximum information with minimum
experiments
•  Basic techniques
• Factorial design
• Factorial design with replications
• Fractional factorial design

Prof. Dr. Mesut Güneş ▪ Ch. 13 Design of Experiments 13.31