Analysis of variance

(ANOVA)

What is analysis of variance? Explain with an example.

The total variation present in a set of observed quantities may under contain
circumstances be partitioned into a number of disjoint components associated
with the nature of classification of the data. The systematic methodology
by which one can partition the cause of variation into several components is
called analysis of variance.
Let us consider yield of paddy. Suppose we have three different kinds of
seeds. So, the yield variation occurs due to variation of seed and also due
to some random error. This is an example of one-way fixed effect layout of
ANOVA.
What is Gauss-Markov Linear model?
Distinguish between fixed, mixed and random effects model. Also distinguish
between ANOVA, Regression and ANCOVA model:
Consider ‘n’ independent random observations y1 , y2 , . . . , yn where

E(yi ) = ai1 β1 + ai2 β2 + · · · + aap βp

V (yi ) = σ 2
cov(yi , yi ) = 0 ∀ i 6= j

where β1 , β2 , . . . , βp are model parameters and ai1 , ai2 , . . . , aip are coefficients
which are known.

1
 Define,
 
a11 a12 ... a1p
 
 a21 a22 . . . a2p 
n×p
X = .
 
 .. .. .. 
 . . 

an2 an2 . . . anp
 
y1
 
 y2 
y= . 
 
.
e  .
yn
   
E(y1 ) a11 β1 + a12 β2 + · · · + a1p βp
   
 E(y2 )   a21 β1 + a22 β2 + · · · + a2p βp 
E(y ) =  .  = 
   
. .. 
 .   .
e    

E(yn ) an1 β1 + an2 β2 + · · · + anp βp
  
a11 a12 . . . a1p β1
  
 a21 a22 . . . a2p  β2 
= .
  
 .. .. ..  .
.
 . .  . 
an1 an2 . . . anp βp
= X n×p β p×1
e
 
β1
 
β2 
where β =  .  .
 
.
e  .
βp
So finally we can write y = Xβ + e, where E(e) = 0 and Disp (e) = σ 2 In .
e e e e
This model is called Gauss-Markov linear model:
e

If we assume that β1 , β2, . . . , βp themselves are random sample from the

distribution of β̃1 , β̃2 , β̃p , where β̃j is a random variable whose realised
value is βj , then the model is called random effects model.
If all the parameters are fixed quantity and some are random then the
model is called Mixed effects model.
Let us consider the Gauss Markov linear model Y = Xβ + e, where X
e e e
2
is called design matrix, which contains coefficients corresponding to model
parameters.
If the value of the coefficients are binary i.e., 0 or 1, then the model is
called Analysis of variance model.
By an ANOVA model we test whether an effects is absent or present.
When the value of the coefficients come from the value of some other
independent co-variates i.e., coefficients have usual continuous value, then
the model is called a regression model.
ANOVA

One-way Two-way
layout layout

Fixed effects Random effects One-observation m-observation

model model per cell per cell

Fixed Random Mixed Fixed Random Mixed

It some of the coefficient value is binary and some are continuous then the
model is called Analysis of covariance (ANCOVA) model.

One way fixed effects model:

Motivation: Let us consider there are three types of seeds of paddy. So yield
of paddy is influenced by only one single factor i.e., seeds which has three
fixed levels.
So yield of paddy can be modelled by an ANOVA model which is called
one way fixed effect layout.
Suppose there be a single factor A with ‘k’ fixed levels A1 , A2 , . . . , Ak , say.

A 



10 mg 



15 mg levels


25 mg 




50 mg 

3
 Factor A
A1 y1 y2 . . . y1n1
A2 y21 y22 . . . y2n2
.. .. .. ..
. . . .
Ak yk1 yk2 . . . ynk

k
X
ni = n one way fixed effect model is given by
i=1

yij = bi + eij
yij = response corresponding to jth observation of ith level of A

j = 1, 2, . . . , ni
and i = 1, 2, . . . , k

One way fixed effects model is given by

yij = ui + eij → error part.

Next we reparametrise the model as follows

yij = µ + (µi − µ) + eij

= µ + αi + eij
αi = additional effect due to ith level of A
µ = general effect
eij = error in model

Assumption:
k
X k
X k
X k
X
1. ni αi = ni (µi − µ) = ni µ i − µ ni = nµ − nµ
i=1 i=1 i=1 i=1
h X i
=0 µ= ni µi = grand mean of effects

a
2. eij ∼ N (0, σ 2 )

4
Estimation of the model parameter:
XX
E= e2ij
i j

∂E ∂ XX
=0⇒ (yij − µ − αi )2 = 0
∂µ ∂µ
i j
XX
⇒ (yij − µ − αi )(−1) = 0
i j
XX XX XX
⇒ yij = µ+ αi
i j i j i j

XX k
X
⇒ yij = nµ − ni αi
i j i=1
| {z }
0
k X
ni
1 X 0
⇒ µ̂ = yij =
n y00
i=1 j=1

k ni
∂ XX
(yij − µ − αi )2 = 0
∂αi
i=1 j=1
ni
X
⇒ (yij − µ − αi )(−1) = 0 [here we differentiate on w.r.t. αi
j=1
(a specific value of i = 1(1)k) so
summation varikes]
ni
X ni
X ni
X
⇒ yij − µ− αi = 0
j=1 j=1 j=1
ni
X
⇒ yij − ni µ = ni αi
j=1
ni
1 X
⇒ α̂i = −µ̂ = ȳi0 − ȳ00
ni
j=1

i.e., yij = ȳ00 + (ȳi0 − ȳ00 ) + eij (1)

Now
eij = yij − α̂i − µ̂ = yij − (ȳi0 − ȳ00 ) − ȳ00
= yij − ȳi0

5
i.e., from (1)

yij = ȳ00 + (ȳi0 − ȳ00 ) + eij

⇒ (yij − ȳ00 ) = (ȳi0 − ȳ00 ) + (yij − ȳi0 )

Squaring and summing over i and j we get

k X
X ni k X
X ni k X
X ni
(yij − ȳ00 )2 = (ȳi0 − ȳ00 )2 + (yij − ȳi0 )2
i=1 j=1 i=1 j=1 i=1 j=1

(product term vanishes)

k
X k X
X ni
= ni (ȳi0 − ȳ00 )2 + (yij − ȳio )2
i=1 i=1 j=1

where

k X
X ni
(yij − ȳ00 )2 = Sum of squares due to the factor A. (SSA)
i=1 j=1
k X
X ni
(yij − ȳi0 ) = sum of squares due to the error (SSE)
i=1 j=1

i.e., T SS = SSA + SSE

This is called orthogonal splitting of total sum of square.
Hypothesis: Here we want to test whether all levels of A have similar
effect or not H0 : αi = 0 vs. Hi : inequality in H0 .

yij = µi + eij
α i = µi − µ = 0
[µ1 = µ2 = · · · = µk ]

Expectation of sum of squares:

k
X
SSA = ni (ȳi0 − ȳ00 )2
i=1

6
 Note that yij = µ + αi + eij

ni ni
1 X 1 X
ȳi0 = yij = (µ + αi + eij )
ni ni
j=1 j=1
ni ni ni
1 X 1 X 1 X
= µ+ αi + eij
ni ni ni
j=1 j=1 j=1
| {z }
ēi0

= µ + αi + ēi0
i k n i k n
1 XX 1 XX
ȳ00 = yij = (µ + αi + eij )
n n
i=1 j=1 i=1 j=1
ni
k X ni
k X i k n
1 X 1 X 1 XX
= µ+ αi + eij
n n n
i=1 j=1 i=1 j=1 i=1 j=1
| {z }
0
= µ + ē00
k
X
i.e., SSA = 1 ni {(µ + αi + ēi0 ) − (µ + ē00 )}2
i=1
k
X
= ni {αi + ēi0 − ē00 }2
i=1
Xk
ni αi2 + (ēi0 − ē00 )2 + 2αi (ēi0 − ē00 )

=
i=1

eij ’s are linearly independent N (0, σ 2 )

k
X k
X k
X
2
∴ E(SSA) = ni αi2 + ni E (ēio − ē00 ) + 2 ni αi E (ēi0 − ē00 )
| {z }
i=1 i=1 i=1 0

E(ēi0 ) = 0
E(ē00 ) = 0

7
Therefore,
k
X k
X
E(SSA) = ni αi2 + ni E (ēi0 − ē00 )2
i=1 i=1
2
E (ēi0 − ē00 ) = E(ē2i0 ) + E(ē200 ) − 2E(ēi0 ēi0 )
= V (ēi0 ) + V (ē00 ) − 2 cov(V (ēi0 , V (ē00 )
σ2 σ2 σ2 σ2 σ2
= + −2 = −
ni n n ni n
k
!
1 X
[cov(ēi0 , ē00 ) = cov ēi0 , ni ēi0
n
i=1
1 1 σ2 σ2
= ni V (ēi0 ) = · ni = ]
n n ni n

Therefore,
k k
X X  ni 
E(SSA) = ni αi2 + ni σ 2 − σ 2
n
i=1 i=1
k k
X σ2 X
= + kσ −ni αi2 ·n= 2
ni αi2 + (k − 1)σ 2
n
i=1 i=1
 
X n
k Xi

E(SSE) = E  (yij − ȳi0 )2 

i=1 j=1

yij = µ + αi + eij
ȳi0 = µ + αi + ēi0
 
Xk X ni
E(SSE) = E  (eij − ēi0 )2 
i=1 j=1
 
X ni
k X
E(e2ij ) + E(ē2i0 ) − 2E(eij ēi0 ) 

=E
i=1 j=1

E(e2ij ) = V (eij ) = σ 2
σ2
E(ē2i0 ) = V (ēi0 ) =
ni
 
ni
1 X 1 σ2
E(eij ēi0 ) = E eij , eij  = E(e2ij ) =
ni ni ni
j=1

8
 Therefore,
ni 
k X k
σ2 σ2
X  X
σ2 + ni σ 2 − σ 2


E(SSE) = −2 =
ni ni
i=1 j=1 i=1
k
X k
X
= σ2 ni − σ 2 1 = σ 2 (n − 1)
i=1 i=1

Note that
k
X
E(SSA) = ni αi2 + (k − 1)σ 2
i=1
  k
SSA 1 X
⇒E = ni αi2 + σ 2
k−1 k−1
i=1
k
1 X
⇒ E(M SA) = ni αi2 + σ 2
k−1
i=1

(k − 1) degrees of freedom corresponding to SSA.

Here, MSA = Mean square due to A.
Again,

E(SSE) = (n − k)σ 2
 
SSE
⇒E = σ 2 (MSE : Mean square error)
n−k
⇒ E(M SE) = σ 2

(n − k) degrees of freedom due to error under H0 ,

E(M SA) = σ 2 = E(M SE)

As we deviate away from H0 ,

E(M SA) ≥ E(M SE)

M SA
We define the following test statistic F = so the large value of F
M SE
indicates the reject of H0 so the right tailed test based on F statistic under
H0 , +
SSA ∼ χ2k−1
independent
SSE ∼ χ2n−k

9
 SSA/k − 1
F = ∼ Fk−1,n−k
SSE/n − k
i.e., we reject H0 at level α if

F > Fαjk−1,n−k .

Paired comparison:

H0 : µ1 = µ2 = · · · = µk v/s H1 : at least one inequality in H0 .

If H0 is rejected then different class means are different from each other, so
a paired comparison is required. [let m hypothesis to be tested simultaneously.
Type-1 error of each test is α.

P [at least one false rejection] = 1 − P No false rejection

= 1 − (1 − α)m

as m → ∞, P [at least one false rejection] → 1

This is problem of multiple testing]
From comparing ith class with i0 th class we have the following hypothesis,

H0 : µi = µ0i vs H1 : µi 6= µ0i

The corresponding test statistic is,

ȳi0 − ȳi0 0
t= r   ∼ tn−k .
M SE n1i + 1
n0i

We reject H0 at level α if
s  
1 1
|ȳi0 − ȳ |
i0 0 M SE + t i n − k,
ni n0i α/2

this quantity called east significant.

Two way fixed effect model: (one observation different per cell)
Motivation: Suppose in statistics tuition market there are two teachers
the first teacher has two fixed batches and the second teacher has three fixed
batches. Also suppose that only one student is present in the intersection of
the batches of two teachers. Then marks of students is varied by two factors,

10
i.e., the two teachers and it is modelled by ANOVA two way fixed effects
model one observation per cell.

B→
B1 B2 . . . Bq
A↓
A1 y11 y12 . . . y1q
A2 y21 y22 . . . y2q
.. .. .. ..
. . . .
Ap yp1 yp2 . . . ypq

yij = observation corresponding to ith level of A and jth level of B.

Here the model is : yij = µij + eij
Reparametrise:

yij = µ + (µi0 − µ) + (µij − µ) + (µij − µi0 − µ0j + µ) + eij

= µ + αi + βj + γij + eij

µ = general effect
αi = additional effect due to ith level of A.
βj = additional effect due to jth level of B.
γij = Interaction effect due to ith level of A and jth level of B.
For one observation per cell γij = 0 as γij can not be estimated so the
model is yij = µ + αi + βj + eij .
Assumption:
p q
ii∂
X X
(i) αi = 0 (ii) βj = 0 (iii) eij ∼ N (0, σ 2 ).
i=1 j=1

Model parameter estimator:

p X
X q p X
X q
E= e2ij = (yij − µ − αi − βj )2
i=1 j=1 i=1 j=1

11
 p X
q
∂F X
=0⇒ (yij − µ − αi − βj )(−1) = 0
∂µ
i=1 j=1
Xp X q q X
X p q X
X p
⇒ yij − pqµ − αi − βj = 0
i=1 j=1 j=1 j=1 j=1 j=1
| {z } | {z }
0 0
p X
q
1 X
⇒ µ̂ = yij = ȳ00
pq
i=1 j=1
p q
∂E ∂ XX
=0⇒ (yij − µ − αi − βj )2 = 0
∂αi ∂αi
i=1 j=1
q
X q
X
⇒ yij = qµ + qαi + βj
j=1 j=1
|{z}
0
q
1X
⇒ α̂i = yij − µ̂ = ȳi0 − ȳ00
q
j=1

1P p
Similarly, β̂j = ȳ0j − ȳ00 where ȳ0j = yij orthogonal splitting of total
p i=1
sum of squares:
Note that yij = µ̂ + µ̂i + β̄j + eij

i.e., yij = ȳ00 + (ȳi0 − ȳ00 ) + (ȳ0j − ȳ00 ) + (yij − ȳi0 − ȳ0j + ȳ00 )

⇒ (yij − ȳ00 ) = (ȳi0 − ȳ00 ) + (ȳ0j + ȳ00 ) + (yij − ȳi0 − ȳ0j + ȳ00 )
p X
X q p
X q
X
⇒ (yij − ȳ00 )2 = q (ȳi0 − ȳ00 )2 + p (ȳ0j − ȳ00 )2
i=1 j=1 i=1 j=1
X p X q
+ (yij − ȳi0 − ȳ0j + ȳ00 )
i=1 j=1

(product term vanishes)

T SS = SSA + SSB +SSE→Sum of square due to error

↓ ↓ ↓
Total sumSum of square Sum of square
of square due to A due to B

12
 Degrees of freedom corresponding to TSS → pq − 1
Degrees of freedom corresponding to SSA → p − 1
Degrees of freedom corresponding to SSB → q − 1
Degrees of freedom corresponding to error → (p − 1) × (q − 1)

Expectation of SS:
 
Xp X
q
E(SSE) = E  (yij − ȳi0 − ȳ0j − ȳ00 )2 
i=1 j=1

yij = µ + αi + βj + eij
q q
1X 1X
ȳi0 = yij = (µ + αi + βj + eij )
q q
j=1 j=1
q q
1 X 1X
= µ + αi + βj + eij
q q
j=1 j=1

= µ + αi + ēi0

Similarly,

ȳ0j = µ + βj + ē0j
ȳ00 = µ + ē00

SSE:

p X
X q
(µ + αi + βj + eij − µ − αi − ēi0 − µ − βj − ē0j + µ + ē00 )2
i=1 j=1
Xp X q
= (eij − ēi0 − ē0j + ē00 )2
i=1 j=1
Xp X q
= (e2ij + ē2i0 + ē20j + ē200 − 2eij ēi0 − 2eij ē0j + 2eij ē00
i=1 j=1

+ 2ēi0 ē0j − 2ēi0 ē00 − 2ē0j ē00 )

13
 p X
X q
E[SSE] = [E(e2ij ) + E(ē2i0 ) + E(ē20j ) + E(ē200 ) − 2 cov(eij , ēi0 )
i=1 j=1

+ 2 cov(eij , ē00 ) + 2 cov(ēi0 , ē0j ) − 2 cov(ēi0 , ē00 ) − 2 cov(ē0i , ē00 )]

 
q
1X 1 1 σ2
cov(eij , ēi0 ) = cov eij , eij  = E(e2ij ) = V (eij ) =
q q q q
j=1
p
!
qX 1 σ2
cov(eij , ē0j ) = cov eij eij = V (eij ) =
p p p
i=1
 
p q
1 XX  1 σ2
cov(eij , ē00 ) = cov eij ,
 eij = V (eij ) =
pq pq pq
i=1 j=1
 
q p
1X 1X  q σ2
cov(ēi0 , ē0j ) = cov  eij , eij = V (eij ) =
q p pq pq
j=1 i=1
p
!
1X 1 σ2 σ2
cov(ēi0 , ē00 ) = cov ēi0 , ēi0 = =
p p q pq
i=1

Similarly,
 
q
q X 1 σ2 σ2
cov(ē0j , ē00 ) = cov ē0j , ē0j  = =
q q p pq
j=1

p X q 
σ2 σ2 σ2 σ2 σ2 σ2 σ2 σ2
X 
E(SSE) = σ2 + + −2 −2 +2 +2 −2 −2
p pq q p pq pq pq pq
i=1 j=1
p X q 
σ2 σ2 σ2
X 
= σ2 − − −
p q pq
i=1 j=1
pqσ 2 (p − 1)(q − 1)
= = σ 2 (p − 1)(q − 1)
pq


2 SSE
i.e., E[SSE] = σ (p − 1)(q − 1) ⇒ E = σ2
(p − 1)(q − 1)
⇒ E[M SE] = σ 2

14
 p
" # " p #
X X
2 2
E[SSA] = E q (ȳi0 − ȳ00 ) = E q (µ + αi + ēi0 − µ − ē00 )
i=1 i=1
p
" #
X
=E q (αi + ei0 − ē00 )2
i=1
p
" #
X
=E q (αi2 + ē2i0 + ē200 − 2ēi0 ē00 + 2αi ēi0 − 2αi ē00 )
i=1
p
X  2
αi + E(ē2i0 ) + E(ē200 ) − 2 cov(ēi0 , ē00 )

=q
i=1
p  p
σ2 σ2
X  X
=q αi2 + − =q αi2 + pσ 2 − σ 2
q pq
i=1 i=1
p
X p
X
=q αi2 + pσ 2 − σ 2 = q αi2 + (p − 1)σ 2
i=1 i=1
  p
SSA q X
E = αi2 + σ 2
p−1 p−1
i=1
p
q X
⇒ E[M SA] = αi2 + σ 2
p−1
i=1

q Pq
Similarly, E[M SB] = β 2 + σ2
p − 1 j=1 j
Here we want to test,

(i) H01 : αi = 0 ∀ i vs H1 : at least one inequality in H01

↓
The factor A has no effect

(ii) H02 : βj = ∀ j vs H1 : at least one inequality in H02

↓
Factor B has no effect.

Under H0 ,
E(M SA) = σ 2 As we deviate away from H01
E(M SA) ≥ σ 2 = E(M SE)
M SA
So intuitively a large value of M SE indicates the rejection of H0 .

15
 Under H0 , +
SSA ∼ χ2p−1
independent
SSE ∼ χ2(p−1)(q−1)

Now
SSA/p − 1
∼ Fp−1 , (p − 1)(q − 1)
SSE/(p − 1)(q − 1)
F1 say
M SA
F1 = ∼ Fp−1,(p−1)(q−1)
M SE

So we reject H01 at size α if

F1 > Fα j p−1,(p−1)(q−1)

M SB
Define F2 = ∼ Fq−1,(p−1)(q−1)
M SE
We reject H02 at size α if

F2 > Fαj q−1,(p−1)(q−1)

Two way fixed effects model

(m-observations per cell)

Motivation: Suppose in statistics tuition market there are two teachers.
The first teacher has two fixed batches and the second teacher has three fixed
batches. Also suppose that ‘m’ students are present in the inter section of
the batches of two teachers. Then marks of students in varied by two factors
i.e., the two teachers and it is modelled by ANOVA two way fixed effects
layout m observation per cell.
Model: yijk = µ + αi + βj + eijk
yijk = kth observation corresponding to ith level of A and jth level of B.
µ = general effect
αi = additional level due to ith level of A.
βj = additional effect due to jth level of B.
γij = interaction effect due to ith level of A and jth level of B.
eijk = error in the model.
Assumption:

16
 p
X q
X
(i) αi = 0 (ii) βj = 0
i=1 j=1
p q
iid
X X
(iii) γij = γij = 0 (iv) eijk ∼ N (0, σ 2 )
i=1 j=1

Estimation of model parameter:

p q m
1 XXX
µ̂ = ȳ00 = yijk
pqm
i=1 j=1 k=1

q m
1 XX
α̂i = ȳi00 − ȳ00 , ȳi00 = yijk
qm
j=1 k=1
p X m
1 X
β̂j = ȳ0j0 − ȳ00 , ȳ0j0 = yijk
pm
i=1 k=1

p X
X q X
m
E= (yijk − µ − αi − βj − γij )2
i=1 j=1 k=1
m
∂E X
= (−2)(yijk − µ − αi − βj − γij ) = 0
∂γij
k=1

m
X
⇒ yijk = mµ̂ + mα̂i + mβ̂j + mγ̂kj
k=1
m
1 X
⇒ γ̂ij = yijk − (ȳ00 − ŷ00 ) − (ȳ0j0 − ȳ000 ) − ȳ000
m
k=1
⇒ γ̂ij = ȳij0 − ȳi00 − ȳ0j0 + ȳ000

Orthogonal splitting of total sum of square

yijk = ȳ000 + (ȳi00 − ȳ000 ) + (ȳ0j0 − ȳ000 )

+ (ȳij0 − ȳi00 − ȳ0j0 + ȳ000 ) + (yijk − ȳijo )
⇒ (yijk − ȳ000 ) = (ȳi00 − ȳ000 ) + (ȳ0j0 − ȳ000 )
+ (ȳij0 − ȳi00 − ȳ0j0 + ȳ000 ) + (yijk − ȳij0 )

17
squaring and taking sum in both sides we get

p X
X q X
m p
X q
X
(yijk − ȳ000 )2 = qm (ȳi00 − ȳ000 ) + pm (ȳ0j0 − ȳ000 )
i=1 j=1 k=1 i=1 j=1
p X
X q
+m (ȳij0 − ȳi00 − ȳ0j0 + ȳ000 )2
i=1 j=1
p
XX q X m
+ (yijk − ȳij0 )2
i=1 j=1 k=1

T SS = SSA + SSB + SS(AB) + SSE

degrees of freedom of TSS = pqm − 1

degrees of freedom of SSA = p − 1

degrees of freedom of SSB = q − 1

degrees of freedom of SS(AB) = (p − 1)(q − 1)

degrees of freedom of SSE = pq(m − 1)

Expectation of SS:

p X
X q X
m
SSE = (yijk − ȳij0 )
i=1 j=1 k=1

yijk = µ + αi + βj + γij + eijk

m
1 X
ȳij0 = (µ + αi + βj + γij + eijk )
m
k=1
= µ + αi + βj + γij + ēijo

18
 p X
X q X
m
E(SSE) = E(eijk − ēij0 )2
i=1 j=1 k=1
Xp X q X m
E e2ijk + ē2ij0 − 2ēij0 eijk
 
=
i=1 j=1 k=1
Xp X q X m
E(e2ijk ) + E(ē2ij0 ) − 2 cov(eijk , ēij0 )
 
=
i=1 j=1 k=1
p X q X m m
" !#
X σ2 1 X
= σ2 + − 2 cov eijk , eijk
m m
i=1 j=1 k=1 k=1
p X q X m 
σ2 σ2
X 
2
= σ + −2
m m
i=1 j=1 k=1
p X q X m
σ2
X  
= σ2 −
m
i=1 j=1 k=1
(m − 1)pqmσ 2
= = (m − 1)pqσ 2
m
p X
X q
E(SS(AB)) = m (ȳij0 − ȳi00 − ȳ0j0 + ȳ00 )2
i=1 j=1

yijk = µ + αi + βj + γij + eijk

⇒ ȳij0 = µ + αi + βj + γij + ēij0
ȳi00 = µ + ēi00 + αi
ȳ0j0 = µ + ē0j0 + βj
ȳ00 = µ + ē00
 
p X
X q
E[SS(AB)] = m E (ēij0 − ēi00 − ē0j0 + ē000 + γij )2 
i=1 j=1
p X
X q
=m {E(ēij0 − ēi00 − ē0j0 + ē000 )2 + E(γij
2
)
i=1 j=1

+ 2γij E(ēij0 − ēi00 − ē0j0 + ē000 )}
p X
X q p X
X q
2
=m γij +m E(ēij0 − ēi00 − ē0j0 + ēi00 )2
i=1 j=1 i=1 j=1

19
Note that

E(ēij0 − ēi00 − ē0j0 + ē000 )2

= E(ē2ij0 + ē2i00 + ē20j0 + ē2000 − 2ēij0 ēi00 − 2ēij0 ē0j0
+ 2ēij0 ē000 + 2ē0j0 − 2ēi00 ē000 − 2ē0j0 ē000 )
= E(ē2ij0 ) + E(ē2i00 ) + E(ē0j0 )2 + E(ē2000 )
− 2 cov(ēij0 , ēi00 ) − 2 cov(ēij0 , ē0j0 ) + 2 cov(ēi00 , ē0j0 )
+ 2 cov(ē000 ) − 2 cov(ēi00 , ē000 ) − 2 cov(ē0j0 , ē000 )

 
q
1X 1 σ2
cov(ēij0 , ēi00 ) = cov ēij0 , ēij0  = V (ēij0 ) =
q q mq
j=1
 
p
1X 1 σ2
cov(ēij0 , ē0j0 ) = cov ēij0 ,
 ēij0  = V (ēij0 ) =
p p mp
j=1
 
q X m p X
m
1 X 1 X
cov(ēi00 , ē0j0 ) = cov  eijk , eijk 
qm pm
j=1 k=1 i=1 k=1
m
1 X 1 2 σ2
= V (eijk ) = mσ =
pqm2 pqm2 pqm
k=1
p
!
1X 1 σ2
cov(ēi00 , ē000 ) = cov ēi00 , ēi00 = V (ēi00 ) =
p p pqm
i=1

Similarly,

σ2
cov(ē0j0 , ē000 ) =
pqm
p X
q X
m  2
X σ σ2 σ2 σ2
= + + +
m qm pm pqm
i=1 j=1 k=1
σ2
σ2 σ2 σ2 σ2 σ2

−2 +2 −2 +2 −2 −2
mq mp pqm pqm pqm pqm
p q m
X X X σ2 σ2 σ2 σ2
 
= − − +
m mp qm pqm
i=1 j=1 k=1
mpq(pq − p − q + 1)
= σ2 = σ 2 (p − 1)(q − 1)
mpq

20
 Therefore,
p X
X q
2
E[SS(AB)] = m γij + (p − 1)(q − 1)σ 2
i=1 j=1
Xp
E(SSA) = qm αi2 + (p − 1)σ 2
i=1
Xq
E(SSB) = pm βj2 + (q − 1)σ 2
i=1

From previous model, we already done.

Random effects model:

Motivation: The yield of paddy in India in influenced by different states.

So state is a single factor causing the variation. Since India has over thirty
states. The yield data can not be obtained from all the states due to time
and cost constraints. So a sample of states are randomly chosen. Hence the
factor “State” has random number of levels. Thus yield is modelled as one
way random effects model.
Let us consider a single factor A having ‘k’ random levels, where the
levels are so chosen at random from larger number of levels. There are ‘r’
observations corresponding to each levels.
So total number of observations are, n = rk.
A ANOVA model is given by

yij = µ + ai + eij

where µ = general effect

ai = additional random effect correspond to ith level of A
eij = error in the model
yij = jth observation corresponding to ith level of A

Assumption:
iid
(i) ai ∼ N (0, σa2 )
iid
(ii) eij ∼ N (0, σe2 )

21
(iii) ai and eij are independent.

Orthogonal splitting of TSS:

k X
X r k
X k X
X r
(yij − ȳ00 )2 = r (ȳi0 − ȳ00 )2 + (yij − ȳi0 )2
i=1 j=1 i=1 i=1 j−1

T SS = SSA + SSE

degrees of freedom of T SS = (n − 1)

degrees of freedom of SSA = (k − 1)

degrees of freedom of SSE = (n − k)

Expectation of SS:

k k
" #
X X
E[SSA] = E r (ȳi0 − ȳ00 )2 = r E(ȳi0 − ȳ00 )2
i=1 i=1
k k
" # " #
X X
E(SSA) = E r (ȳi0 − ȳ00 )2 = E r (µ + ai + ēi0 − µ − ā − ē00 )2
i=1 i=1

yij = µ + ai + eij
r
1X
ȳi0 = yij = µ + ai + ēi0
r
j=1

ȳ00 = µ + ā + ē00

22
Now by (*)

k
" #
X
(ai − ā)2 + (ēi0 − ē00 )2 + 2(ai − ā)(ēi0 − ē00 )

E(SSA) = E r
i=1
k
X k
X k
X
=r (ai − ā)2 + r E(ēi0 − ē00 )2 + 2 (ai − ā)(ēi0 − ē00 )
i=1 i=1 i=1
| {z }
0
k
X
E(ai − ā)2 + E(ēi0 − ē00 )2

=r
i=1
Xk
E(a2i ) + E(σ̄ 2 ) − 2 cov(ai , ā)

=r
i=1
k
X
E(ē2i0 ) + E(ē200 ) − 2 cov(ēi0 , ē00 )

+r
i=1
k k  2
σa2 σa2 σa σe2 σe2
X   X 
= r σa2 + −2 +r + −2
k k r rk rk
i=1 i=1
k  k  2
σa2 σe2
 
X X σe
= r σa2 − +r −
k r rk
i=1 i=1
= r(k − 1)σa2 + (k − 1)σe2 = (k − 1)(rσa2 + σe2 )

h i Pk
SSA
E k−1 = rσa2 + σe2 cov(ēi0 , ē00 ) = cov(ēi0 , k1 i=1 ēi0 )
σe2
⇒ E[M SA] = rσa2 + σe2 = k1 var(ēi0 = rk )

 
Xk X
r
E[SSE] = E  (yij − ȳi0 )2 
i=1 j=1
 
Xk X
r
=E (µ + ai + eij − µ − ai − ēi0 )2 
i=1 j=1
 
Xk X
r
=E (eij − ēi0 )2 
i=1 j=1

23
 E(eij − ēi0 )2 = E(e2ij ) + E(ē2i0 ) − 2 cov(eij , ēi0 )
 
2 r
σ 1 X
= σe2 + e − 2 cov eij , eij 
r r
j=1

σe2 σe2σ2
= σe2 +
−2 = σe2 − e
r r r
k r  2

XX σ
E(SSE) = σe2 − e
r
i=1 j=1

= σnσe2 − kσe2 = (n − k)σe2

⇒ E(M SE) = σe2

Here we want to test

H0 : σa2 = 0 vs H1 : σa2 > 0

so under H0 , E(M SA) = E(M SE) = σe2

As we deviate away from new E(M SA) ≥ E(M SE)
M SA M SA
So a right tailed test based on is appropriate under H0 , f = ∼
M SE M SE
Fk−1,n−k so we reject H0 at level α if f > Fα j k−1,n−k .

Random effects model: Two way ‘m’ observations per cell

Motivation: Total yield of crop in India is varied due to different states as

well different types of crops eg. paddy, wheat. Since types of crops as well as
number of states is very high, data collection becomes tough. So a sample of
crops and sample of states are chosen. So both the factors have randomly
chosen levels. Also in the intersection of different levels. We have fixed m
observation. Hence yield of crop is modelled by random effects model, two
way m-observations per cell.

yijk = µ + ai + bj + cij + eijk

i = 1(1)p, j = 1(1)q and k = 1(1)m

yijk = kth observation corresponding to ith level of A and jth level of B.

ai = random effect due to ith level of A.
bj = random effect due to jth level of B.
cij = random interaction effect due to ith level of A and jth level of B.

24
Orthogonal splitting of TSS:

p X
X q X
m p
X q
X
(yijk − ȳ000 )2 = mq (ȳi00 − ȳ000 )2 + pm (ȳ0j0 − ȳ000 )2
i=1 j=1 k=1 i=1 j=1
p X
X q
+m (ȳij0 − ȳi00 − ȳ0j0 + ȳ000 )2
i=1 j=1
p
XX q X m
+ (yijk − ȳij0 )2
i=1 j=1 k=1

T SS SSA SSB SS(AB) SSE

⇒ = + + +
mpq − 1 p − 1 q − 1 (p − 1)(q − 1) pq(m − 1)

Assumptions:
iid iid
(i) eijk ∼ N (0, σe2 ) 2)
(ii) ai ∼ N (0, σA
iid 2) iid 2 )
(iii) bj ∼ N (0, σB (iv) cij ∼ N (0, σAB

 
XXX
E[SS(AB)] = E  (ȳij0 − ȳi00 − ȳ0j0 + ȳ00 )2 
i j k
XX
=m E(ȳij0 − ȳi00 − ȳ0j0 + ȳ000 )2
i j

yijk = µ + ai + bj + cij + eijk

ȳij0 = µ + ai + bj + cij + ēij0
ȳi00 = µ + ai + b̄ + c̄i0 + ēi00
ȳ0j0 = µ + ā + bj + c̄0j + ē0j0
ȳ000 = µ + ā + b̄ + c̄00 + ē000

25
 XX 
E[SS(AB)] = m E cij − ci0 µ + ai + bj + cij + ēij0
j j

− µ + ai − b̄j − c̄i0 − ēi00 − µ − ā − bj − c̄0j − ē0j0

+ µ + ā + b̄ + c̄00 + ē000
XX
=m E {(cij − c̄i0 − c̄ij + c̄00 ) + (ēij0 − ēi00 − ē0j0 + ē000 )}2
i j
XX
=m [E(c2ij ) + E(c̄2i0 ) + E(c̄20j ) + E(c̄200 ) − 2 cov(cij , c̄i0 )
i j

− 2 cov(cij , c̄0j ) + 2 cov(cij , c̄00 ) + 2 cov(c̄i0 , c̄0j ) − 2 cov(c̄i0 , c̄00 )

− 2 cov(c̄0j , c̄00 ) + E(ē2ij0 ) + E(ē2i00 ) + E(ē20j0 ) + E(ē2000 )
− 2 cov(ēij0 , ēi00 ) − 2 cov(ēij0 , ē0j0 ) + 2 cov(ēij0 , ē000 )
+ 2 cov(ēi00 , ē0j0 ) − 2 cov(ēi00 , ē000 ) − cov(ē0j0 , ē000 )]

Now  
1 X σ2
cov(cij , c̄i0 ) = cov cij , cij  = AB
q q
j

2
σAB 2
σAB
cov(cij , c̄0j ) = cov (cij , c̄00 ) =
p pq
2
σAB
cov(c̄i0 , c̄0j ) =
pq
2
σAB
cov(c̄0j , c̄00 ) =
pq

Again
q
1X σ2
cov(ēij0 , ēi00 ) = cov(ēij0 , ēij0 ) = e
q qm
k=1

σe2 σe2
cov(ēij0 , ē0j0 ) = cov(ēi00 , ē000 ) =
pm pqm

σe2 σe2
cov(ēi00 , ē000 ) = cov(ē0j0 , ē000 ) =
pqm pqm

σe2
cov(ēi00 , ē0j0 ) =
pqm

26
 Therefore,
2
(p − 1)(q − 1)σe2
 
(p − 1)(q − 1)σAB
E[SS(AB)] = m · pq + · pq
pq pq
 2
+ σe2

= (p − 1)(q − 1) mσAB
 
SS(AB) 2
⇒E = mσAB + σe2
(p − 1)(q − 1)
2
⇒E[M S(AB)] = mσAB + σe2
2 2
E(M SA) = mσAB + qmσA + σe2
2 2
E(M SB) = mσAB + pmσB + σe2
E(M SE) = σe2

2 = 0 vs H
Here we want to test H0A : σA 2
1A : σA > 0

2 2
H0B · σB = 0 vs H1B : σB >0
2
and H0AB · σAB 2
= 0 vs H1AB : σAB >0
2 + σ 2 = E(M S(AB))
Under H0A , E(M SA) = mσAB e
As we deviate away from H0A , E(M SA) ≥ E(M S(AB)) so right tailed test
M SA M SA
based on is appropriate under H0A , ∼ Fp−1,(p−1)(q−1)
M S(AB) M S(AB)
we reject H0A at level α if
M SA
> fαj(p−1);(p−1)(q−1) .
M S(AB)

Similarly, under H0B , E(M SB) = E(M S(AB)) = σe2 + mσAB

2

As we deviate away from H0B , E(M SB) ≥ E(M S(AB)). A right tailed
M SB
test based on is appropriate. Similarly for testing H0AB a right
M S(AB)
M S(AB)
tailed test based on is appropriate.
M SE

Mixed effects model:

Motivation: The yield of paddy in India is varied due to different states ad

different types of seeds. Now number of different kinds of seeds is not much
high, so data can be collected, covering all kinds of seeds. But number of
states is over thirty. Thus a sampling is performed. So the yield of paddy is
modelled using a two way mixed effects model.

27
 Two way mixed effects model: m observations per cell
Suppose there are two factors A and B. For the factor A there are ‘p’-fixed
levels and for the factor B there are randomly chosen ‘q’ levels.
Example:
Yield and crop may vary in different state and fertilisers. We have randomly
chosen 10 states and kept all the varieties of fertilisers. So yield is varied due
to the states. Which gives a fixed effect so as a whole underlined ANOVA
model is influenced by both fixed and random factor. So this model is mixed
effects model.
Suppose there are ‘m’ observations per cell. The ANOVA model is given
by,

yijk = µ + ai + bj + cij + eijk ,i = 1(1)p

j = 1(1)q
k = 1(1)m

µ = general effect
ai = additional level due to ith level of A.
bj = additional level due to jth level of B.
cij = random interaction level due to ith level of A and jth level of B.

Assumption:

p
X
(i) ai = 0
i=1
p
X p
X p
X
(ii) cij = αi bj = bj αi = 0
i=1 i=1 i=1

iid 2
(iii) bj ∼ N (0, σB )

(iv) cij ∼ N (0, σi2 ) independently.

iid
(v) eijk ∼ N (0, σe2 )

Remark: {bj } and {eij } are independently distributed.

Orthogonal splitting of total sum of squares:

28
 XXX p
X q
X
(yijk − ȳ000 )2 = qm (ȳi00 − ȳ000 )2 + pm (ȳ0j0 − ȳ000 )2
i j k i=1 j=1
XX
+m (ȳij0 − ȳi00 − ȳ0j0 + ȳ000 )2
i j
XXX
+ (yijk − ȳij0 )2
i j k
1 XX
ȳi00 = yijk
qm
j k
1 XXX
ȳ000 = yijk
mpq
i j k
1 XX
ȳ0j0 = yijk
pm
i k
1 X
ȳij0 = yijk
m
k
1 XX
ȳi00 = yijk
qm
j k
1 XX
= (µ + ai + bj + eij + eijk )
qm
j k

= µ + ai + b̄ + c̄i0 + ēi00
ȳ0j0 = µ + bj + ē0j0
ȳij0 = µ + ai + bj + cj + ēij0
ȳ000 = µ + b̄ + ē000

cij ∼ N (0, σ12 )

Define

p p
2 1 X 2 2 1 X 2
σA = ai ; σAB = σi
p−1 p−1
i=1 i=1

29
Expectation of SS:

p
X
E[SSA] = qm E(ȳi00 − ȳ000 )2
i=1
p
X
= qm E(µ + ai + b̄ + ēi0 + ēi00 − µ − b̄ − ē000 )2
i=1
Xp
= qm E(ai + ēi0 + ēi00 − ē000 )2
i=1
Xp
E(a2i ) + E(ē (i0)) + E(ēi00 − ē000 )2

= qm
i=1
+ product term vanishes due to independent.
" p p p
#
X
2
X σi2 X  2 2

= qm ai + + E(ēi00 ) + E(ē000 ) − 2E(ēi00 ē000 )
q
i=1 i=1 i=1

Now

p
!
1X
E(ēi00 , ē000 ) = cov(ēi00 , ē000 ) = cov ēi00 , ēi00
p
i=1
1 σe2
= V (ēi00 ) =
p pqm

Therefore,

" p p p  #
X 1 X X σ 2
e
qm a2i + σi2 +
q pqm
i=1 i=1 i=1
  
2 (p − 1) 2 p 2 p−1
= qm (p − 1)σA + σAB + σ
q qm e p
2 2
+ σe2
 
= (p − 1) qmσA + mσAB

30
 q
X
and E(SSB) = pm E(ȳ0j0 − ȳ000 )2
j=1
Xq
= pm E(µ + bj + ē0j0 − µ − b̄ − ē000 )2
j=1
Xq 
= pm E(bj − b̄)2 + E(ē0j0 − ē000 )2
j=1


− 2E (bj − b̄)(ē0j0 − ē000 )
| {z }
0
q
X
E(bj − b̄)2 + E(ē0j0 − ē000 )2

= pm
j=1

∴ E(bj − b̄) = E(b2j ) + E(b̄2 ) − 2E(bj b̄)

2

σ2B σ2
 
2 q−1 2
= σB + −2 B = σB
q q q
E(ē0j0 − ē000 )2 = E(ē20j0 ) + E(ē2000 ) − 2 cov(ē000 , ē0j0 )
σe2 σ2 2σe2 σ2 σ2
=+ e − = e − e
pm pqm pqm pm pqm
 
1 q−1
= σe2
pm q
q     
X q−1 2 q−1 1 2
E(SSB) = pm σB + σ
q q pm e
j=1
2
X
= pm(q − 1) +(q − 1)σe2
B
2
= (q − 1)(pmσB + σe2 )

31
 Now
p X
X q
E[SS(AB)] = m E [ȳij0 − ȳi00 − ȳ0j0 + ē000 ]2
i=1 j=1
Xp X q
=m E[µ + ai + ej + cij + ēij0 − µ − ai − b̄ − ēi0
i=1 j=1

− ēi00 − µ − bj − ē0j0 + µ + b̄ + ē000 ]

 
p X
X q  
=m E (cij − c̄i0 )2 + (ēij0 )2 + (ēij0 − ēi00 − ē0j0 + ē000 )2 
 
 | {z } 
i=1 j=1  
(p−1)(q−1) 2
pqm
σe (already done)

Therefore,

E[cij − c̄i0 ]2 = E(c2ij ) + E(c̄2i0 ) − 2 cov(cij , c̄i0 )

σi2 (1 − 1) 2
= σi2 − = σi
q q

p X
q  
2 (q − 1) (p − 1)(q − 1) 2
X
∴ E[SS(AB)] = m σi + σe
q pqm
i=1 j=1
m(q − 1) q
= Pp σi2 + (p − 1)(q − 1)σe2
q i=1
2
= m(p − 1)(q − 1)σAB + (p − 1)(q − 1)σe2
2
= (p − 1)(q − 1)(mσAB + σe2 )

So finally,
 
SSA 2 2
E(M SA) = E = pmσA + mσAB + σe2
p−1

E(M SB) = pmσb2 + σe2

2
E(M S(AB)) = mσAB + σe2
and E(M SE) = σe2

Here we want to test

2 = 0 [i.e., a = 0 ∀ i] vs H
H0A : σA 2
i 1A : σA > 0

32
 2 = 0 ag H
H0B : σB 2
1B : σB > 0
2
H0AB : σAB 2
= 0 ag H1AB : σAB >0
Under H0A
2
E[M SA] = mσAB + σe2 = E[M S(AB)]

As we deviate from H0A ,

E[M SA] ≥ E[M S(AB)]
M SA
so a right tailed test based on is appropriate under H0A ,
M S(AB)
M SA
∼ F(p−1),(p−1)(q−1)
M S(AB)
we reject H0A at level α if
M SA
> Fα j(p−1),(p−1)(q−1)
M S(AB)
under H0b , E(M SB) = E(M SE) = σe2
E(M SB) ≥ E(M SE)
M SB
A right tailed test based on is appropriat similarly for testing H0AB
M SE
M S(AB)
a right tailed test based on is appropriate.
M SE
Remark:
M SA
In general has no F -distribution under H0A . So we consider a
M S(AB)
approximate F -statistic with degrees of freedom (p − 1), (p − 1)(q − 1).
Some important questions and answers:

1. What is general linear hypothesis?

Let y1 , y2 , . . . , yn be independently distributed normal variables with,
E(yi ) = ai1 β1 + ai2 β2 + · · · + aip βp
and V (yi ) = σ 2 ∀ i = (1)n
i.e.,    
E(y1 ) a11 βq + a12 β2 + · · · + a1p βp
   
 E(y2 )   a21 β1 + a22 β2 + · · · + a2p βp 
 . =
   
 ..   .
..


   
E(yn ) an1 β1 + an2 β2 + · · · + anp βp

33
   
a11 a12 . . . a1p β1
  
 a21 a22 . . . a2p  β2 
⇒ E(y ) =  . (2)
  
. .. .. 
  ... 
 
e 
 . . .  
an1 an2 . . . anp βp
⇒ E(y ) = Xβ
e e
X is called design matrix containing known coefficients. β is the vector
unknown model parameters. e

Let us consider that parameters β are subject to ‘m’ independent linear

constraints. e
m×p p×1 m×1
= (3)
H β h
e e
Now for the linear model (1) and restriction (2), we consider a set of
hypothesis containing linear equation in βi ’s given as follows.

t×p p×1 t×1

H0 : =
L β l

where t linear functions in β are assumed to be independent. It is necessary

to assume that the row vectors
e in L are linearly dependent on row vectors
of X and H.
This set of hypothesis is called general linear hypothesis.

2. I will give you later

3. What is orthogonal splitting?

In analysis of variance main objective is to partition the total variability
of any response into disjoint parts, where each parts indicate variability
due to certain effects. Now this partitioning is achieved via orthogonal
splitting of total sum of squares. Here total sum of square refers to as
total variance and via orthogonal splitting total sum of square is splitting
into sum square due to different sources of variation. Here the term square
refers to as total variance and via orthogonal splitting total sum of square
is splitted into sum square due to different sources of variation. Here the
term “orthogonal” indicates that sum of square due to different sources
are independent of each other.

34
 In one way layout, fixed effect model, the total sum of square is partitioned
into sum of square due to the single factor and sum square due to error
as total variability is caused by the single factor and error.

4. If the F ratio is fractional, then what would be the interpretation?

A higher value of F -ratio indicates the rejection of H0 and if F ratio
becomes one the numerator of the test statistic becomes equal the valid
error. Then the null hypothesis is trivially accepted.
Whenever the F -ratio becomes fraction, then it is strongly implied that
the valid error is dominant over the presence of the factor. Hence the null
hypothesis is strongly accepted, i.e., effect of the corresponding factor is
tested not to be true.

5. What do you mean by valid error in the context of ANOVA? Consider

a vendor effects model for a two way clarified data with m observation
in each call. Find the expectation of the related mean sum of squares
and discuss the role of valid errors for testing the significance of different
effects.
Write down the definition of valid.

Refer to two way random effects ANOVA

Note that under H0A and H0B

E[M SA] = E[M S(AB)] and

E(M SB) = E[M S(AB)]

As we drift away from H0A and H0B , E(M SA) ≥ E(M S(AB)) and
E(M SB) ≥ E(M S(AB)) i.e., a right tailed test based on MM SA
S(AB) and
M SB
M S(AB) is appropriate M S(AB) services as the valid error for testing H0A
and H0B . Whereas under H0AB , E(M S(AB)) = E(M SE) and as we drift
away from H0AB E(M S(AB)) ≥ E(M SE). Them on right tailed test based
on M S(AB) is appropriate. Hence M SE serves as the valid error in this
case.

35