Introduction to Time Series Analysis. Lecture 6.

Peter Bartlett
www.stat.berkeley.edu/∼bartlett/courses/153-fall2010

Last lecture:

1. Causality
2. Invertibility

3. AR(p) models
4. ARMA(p,q) models

1
 Introduction to Time Series Analysis. Lecture 6.
Peter Bartlett
www.stat.berkeley.edu/∼bartlett/courses/153-fall2010

1. ARMA(p,q) models

2. Stationarity, causality and invertibility

3. The linear process representation of ARMA processes: ψ.

4. Autocovariance of an ARMA process.

5. Homogeneous linear difference equations.

2
 Review: Causality

A linear process {Xt } is causal (strictly, a causal function

of {Wt }) if there is a

ψ(B) = ψ0 + ψ1 B + ψ2 B 2 + · · ·
∞
X
with |ψj | < ∞
j=0

and Xt = ψ(B)Wt .

3
 Review: Invertibility

A linear process {Xt } is invertible (strictly, an invertible

function of {Wt }) if there is a

π(B) = π0 + π1 B + π2 B 2 + · · ·
∞
X
with |πj | < ∞
j=0

and Wt = π(B)Xt .

4
Review: AR(p), Autoregressive models of order p

An AR(p) process {Xt } is a stationary process that satisfies

Xt − φ1 Xt−1 − · · · − φp Xt−p = Wt ,

where {Wt } ∼ W N (0, σ 2 ).

Equivalently, φ(B)Xt = Wt ,
where φ(B) = 1 − φ1 B − · · · − φp B p .

5
Review: AR(p), Autoregressive models of order p

Theorem: A (unique) stationary solution to φ(B)Xt = Wt

exists iff the roots of φ(z) avoid the unit circle:

|z| = 1 ⇒ φ(z) = 1 − φ1 z − · · · − φp z p 6= 0.

This AR(p) process is causal iff the roots of φ(z) are outside
the unit circle:

|z| ≤ 1 ⇒ φ(z) = 1 − φ1 z − · · · − φp z p 6= 0.

6
 Reminder: Polynomials of a complex variable

Every degree p polynomial a(z) can be factorized as

a(z) = a0 + a1 z + · · · + ap z p = ap (z − z1 )(z − z2 ) · · · (z − zp ),

where z1 , . . . , zp ∈ C are the roots of a(z). If the coefficients a0 , a1 , . . . , ap

are all real, then the roots are all either real or come in complex conjugate
pairs, zi = z̄j .
Example: z + z 3 = z(1 + z 2 ) = (z − 0)(z − i)(z + i),
that is, z1 = 0, z2 = i, z3 = −i. So z1 ∈ R; z2 , z3 6∈ R; z2 = z̄3 .
Recall notation: A complex number z = a + ib has Re(z) = a, Im(z) = b,
√
z̄ = a − ib, |z| = a2 + b2 , arg(z) = tan−1 (b/a) ∈ (−π, π].

7
 Review: Calculating ψ for an AR(p): general case

φ(B)Xt = Wt , ⇔ Xt = ψ(B)Wt
so 1 = ψ(B)φ(B)
⇔ 1 = (ψ0 + ψ1 B + · · · )(1 − φ1 B − · · · − φp B p )
⇔ 1 = ψ0 , 0 = ψj (j < 0),
0 = φ(B)ψj (j > 0).

We can solve these linear difference equations in several ways:

• numerically, or
• by guessing the form of a solution and using an inductive proof, or
• by using the theory of linear difference equations.

8
 Introduction to Time Series Analysis. Lecture 6.
1. Review: Causality, invertibility, AR(p) models

2. ARMA(p,q) models
3. Stationarity, causality and invertibility

4. The linear process representation of ARMA processes: ψ.

5. Autocovariance of an ARMA process.

6. Homogeneous linear difference equations.

9
 ARMA(p,q): Autoregressive moving average models

An ARMA(p,q) process {Xt } is a stationary process that

satisfies

Xt −φ1 Xt−1 −· · ·−φp Xt−p = Wt +θ1 Wt−1 +· · ·+θq Wt−q ,

where {Wt } ∼ W N (0, σ 2 ).

• AR(p) = ARMA(p,0): θ(B) = 1.

• MA(q) = ARMA(0,q): φ(B) = 1.

10
 ARMA(p,q): Autoregressive moving average models

An ARMA(p,q) process {Xt } is a stationary process that

satisfies

Xt −φ1 Xt−1 −· · ·−φp Xt−p = Wt +θ1 Wt−1 +· · ·+θq Wt−q ,

where {Wt } ∼ W N (0, σ 2 ).

Usually, we insist that φp , θq 6= 0 and that the polynomials

φ(z) = 1 − φ1 z − · · · − φp z p , θ(z) = 1 + θ1 z + · · · + θq z q

have no common factors. This implies it is not a lower order ARMA model.

11
 ARMA(p,q): An example of parameter redundancy

Consider a white noise process Wt . We can write

Xt = Wt
⇒ Xt − Xt−1 + 0.25Xt−2 = Wt − Wt−1 + 0.25Wt−2
(1 − B + 0.25B 2 )Xt = (1 − B + 0.25B 2 )Wt

This is in the form of an ARMA(2,2) process, with

φ(B) = 1 − B + 0.25B 2 , θ(B) = 1 − B + 0.25B 2 .

But it is white noise.

12
 ARMA(p,q): An example of parameter redundancy

ARMA model: φ(B)Xt = θ(B)Wt ,

with φ(B) = 1 − B + 0.25B 2 ,
θ(B) = 1 − B + 0.25B 2

Xt = ψ(B)Wt
θ(B)
⇔ ψ(B) = = 1.
φ(B)
i.e., Xt = Wt .

13
 Introduction to Time Series Analysis. Lecture 6.
1. Review: Causality, invertibility, AR(p) models

2. ARMA(p,q) models
3. Stationarity, causality and invertibility

4. The linear process representation of ARMA processes: ψ.

5. Autocovariance of an ARMA process.

6. Homogeneous linear difference equations.

14
 Recall: Causality and Invertibility

A linear process {Xt } is causal if there is a

ψ(B) = ψ0 + ψ1 B + ψ2 B 2 + · · ·
∞
X
with |ψj | < ∞ and Xt = ψ(B)Wt .
j=0

It is invertible if there is a

π(B) = π0 + π1 B + π2 B 2 + · · ·
∞
X
with |πj | < ∞ and Wt = π(B)Xt .
j=0

15
ARMA(p,q): Stationarity, causality, and invertibility

Theorem: If φ and θ have no common factors, a (unique) sta-

tionary solution to φ(B)Xt = θ(B)Wt exists iff the roots of
φ(z) avoid the unit circle:

|z| = 1 ⇒ φ(z) = 1 − φ1 z − · · · − φp z p 6= 0.

This ARMA(p,q) process is causal iff the roots of φ(z) are out-
side the unit circle:

|z| ≤ 1 ⇒ φ(z) = 1 − φ1 z − · · · − φp z p 6= 0.

It is invertible iff the roots of θ(z) are outside the unit circle:

|z| ≤ 1 ⇒ θ(z) = 1 + θ1 z + · · · + θq z q 6= 0.

16
 ARMA(p,q): Stationarity, causality, and invertibility

Example: (1 − 1.5B)Xt = (1 + 0.2B)Wt .

 
3 2
φ(z) = 1 − 1.5z = − z− ,
2 3
1
θ(z) = 1 + 0.2z = (z + 5) .
5
1. φ and θ have no common factors, and φ’s root is at 2/3, which is not on
the unit circle, so {Xt } is an ARMA(1,1) process.
2. φ’s root (at 2/3) is inside the unit circle, so {Xt } is not causal.
3. θ’s root is at −5, which is outside the unit circle, so {Xt } is invertible.

17
 ARMA(p,q): Stationarity, causality, and invertibility

Example: (1 + 0.25B 2 )Xt = (1 + 2B)Wt .

2 1 2
1
φ(z) = 1 + 0.25z = z + 4 = (z + 2i)(z − 2i),
4  4
1
θ(z) = 1 + 2z = 2 z + .
2
1. φ and θ have no common factors, and φ’s roots are at ±2i, which is not
on the unit circle, so {Xt } is an ARMA(2,1) process.
2. φ’s roots (at ±2i) are outside the unit circle, so {Xt } is causal.
3. θ’s root (at −1/2) is inside the unit circle, so {Xt } is not invertible.

18
 Causality and Invertibility

Theorem: Let {Xt } be an ARMA process defined by

φ(B)Xt = θ(B)Wt . If all |z| = 1 have θ(z) 6= 0, then there
are polynomials φ̃ and θ̃ and a white noise sequence W̃t such
that {Xt } satisfies φ̃(B)Xt = θ̃(B)W̃t , and this is a causal,
invertible ARMA process.

So we’ll stick to causal, invertible ARMA processes.

19
 Introduction to Time Series Analysis. Lecture 6.
1. Review: Causality, invertibility, AR(p) models

2. ARMA(p,q) models
3. Stationarity, causality and invertibility

4. The linear process representation of ARMA processes: ψ.

5. Autocovariance of an ARMA process.

6. Homogeneous linear difference equations.

20
 Calculating ψ for an ARMA(p,q): matching coefficients

Example: Xt = ψ(B)Wt ⇔ (1 + 0.25B 2 )Xt = (1 + 0.2B)Wt ,

so 1 + 0.2B = (1 + 0.25B 2 )ψ(B)
⇔ 1 + 0.2B = (1 + 0.25B 2 )(ψ0 + ψ1 B + ψ2 B 2 + · · · )
⇔ 1 = ψ0 ,
0.2 = ψ1 ,
0 = ψ2 + 0.25ψ0 ,
0 = ψ3 + 0.25ψ1 ,
..
.

21
 Calculating ψ for an ARMA(p,q): example

⇔ 1 = ψ0 , 0.2 = ψ1 ,
0 = ψj + 0.25ψj−2 (j ≥ 2).

We can think of this as θj = φ(B)ψj , with θ0 = 1, θj = 0 for j < 0, j > q.

This is a first order difference equation in the ψj s.
We can use the θj s to give the initial conditions and solve it using the theory
of homogeneous difference equations.
1 1 1 1 1 1 1


ψj = 1, 5 , − 4 , − 20 , 16 , 80 , − 64 , − 320 , . . . .

22
 Calculating ψ for an ARMA(p,q): general case

φ(B)Xt = θ(B)Wt , ⇔ Xt = ψ(B)Wt

so θ(B) = ψ(B)φ(B)
⇔ 1 + θ1 B + · · · + θq B q = (ψ0 + ψ1 B + · · · )(1 − φ1 B − · · · − φp B p )
⇔ 1 = ψ0 ,
θ1 = ψ1 − φ1 ψ0 ,
θ2 = ψ2 − φ1 ψ1 − · · · − φ2 ψ0 ,
..
.

This is equivalent to θj = φ(B)ψj , with θ0 = 1, θj = 0 for j < 0, j > q.

23
 Introduction to Time Series Analysis. Lecture 6.
1. Review: Causality, invertibility, AR(p) models

2. ARMA(p,q) models
3. Stationarity, causality and invertibility

4. The linear process representation of ARMA processes: ψ.

5. Autocovariance of an ARMA process.

6. Homogeneous linear difference equations.

24
 Autocovariance functions of linear processes

Consider a (mean 0) linear process {Xt } defined by Xt = ψ(B)Wt .

γ(h) = E (Xt Xt+h )

= E (ψ0 Wt + ψ1 Wt−1 + ψ2 Wt−2 + · · · )
× (ψ0 Wt+h + ψ1 Wt+h−1 + ψ2 Wt+h−2 + · · · )
2
= σw (ψ0 ψh + ψ1 ψh+1 + ψ2 ψh+2 + · · · ) .

25
 Autocovariance functions of MA processes

Consider an MA(q) process {Xt } defined by Xt = θ(B)Wt .


 σ 2 Pq−h θ θ
w j=0 j j+h if h ≤ q,
γ(h) =
 0 if h > q.

26
 Autocovariance functions of ARMA processes

ARMA process: φ(B)Xt = θ(B)Wt .

To compute γ, we can compute ψ, and then use
2
γ(h) = σw (ψ0 ψh + ψ1 ψh+1 + ψ2 ψh+2 + · · · ) .

27
 Autocovariance functions of ARMA processes

An alternative approach:

Xt − φ1 Xt−1 − · · · − φp Xt−p
= Wt + θ1 Wt−1 + · · · + θq Wt−q ,
so E ((Xt − φ1 Xt−1 − · · · − φp Xt−p ) Xt−h )
= E ((Wt + θ1 Wt−1 + · · · + θq Wt−q ) Xt−h ) ,
that is, γ(h) − φ1 γ(h − 1) − · · · − φp γ(h − p)
= E (θh Wt−h Xt−h + · · · + θq Wt−q Xt−h )
q−h
X
2
= σw θh+j ψj . (Write θ0 = 1).
j=0

This is a linear difference equation.

28
Autocovariance functions of ARMA processes: Example

(1 + 0.25B 2 )Xt = (1 + 0.2B)Wt , ⇔ Xt = ψ(B)Wt ,

 
1 1 1 1 1 1 1
ψj = 1, , − , − , , , − , − ,... .
5 4 20 16 80 64 320
q−h
X
2
γ(h) − φ1 γ(h − 1) − φ2 γ(h − 2) = σw θh+j ψj
j=0

2


 σw (ψ0 + 0.2ψ1 ) if h = 0,
⇔ γ(h) + 0.25γ(h − 2) = 2
0.2σw ψ0 if h = 1,


0 otherwise.


29
 Autocovariance functions of ARMA processes: Example

We have the homogeneous linear difference equation

γ(h) + 0.25γ(h − 2) = 0

for h ≥ 2, with initial conditions

2
γ(0) + 0.25γ(−2) = σw (1 + 1/25)
2
γ(1) + 0.25γ(−1) = σw /5.

We can solve these linear equations to determine γ.

Or we can use the theory of linear difference equations...

30
 Introduction to Time Series Analysis. Lecture 6.
Peter Bartlett
www.stat.berkeley.edu/∼bartlett/courses/153-fall2010

1. ARMA(p,q) models

2. Stationarity, causality and invertibility

3. The linear process representation of ARMA processes: ψ.

4. Autocovariance of an ARMA process.

5. Homogeneous linear difference equations.

31
 Difference equations

Examples:

xt − 3xt−1 = 0 (first order, linear)

xt − xt−1 xt−2 = 0 (2nd order, nonlinear)
xt + 2xt−1 − x2t−3 = 0 (3rd order, nonlinear)

32
 Homogeneous linear diff eqns with constant coefficients

a0 xt + a1 xt−1 + · · · + ak xt−k = 0
k


⇔ a0 + a1 B + · · · + ak B xt = 0
⇔ a(B)xt = 0
auxiliary equation: a0 + a1 z + · · · + ak z k = 0
⇔ (z − z1 )(z − z2 ) · · · (z − zk ) = 0

where z1 , z2 , . . . , zk ∈ C are the roots of this characteristic polynomial.

Thus,

a(B)xt = 0 ⇔ (B − z1 )(B − z2 ) · · · (B − zk )xt = 0.

33
 Homogeneous linear diff eqns with constant coefficients

a(B)xt = 0 ⇔ (B − z1 )(B − z2 ) · · · (B − zk )xt = 0.

So any {xt } satisfying (B − zi )xt = 0 for some i also satisfies a(B)xt = 0.

Three cases:

1. The zi are real and distinct.

2. The zi are complex and distinct.

3. Some zi are repeated.

34
 Homogeneous linear diff eqns with constant coefficients

1. The zi are real and distinct.

a(B)xt = 0
⇔ (B − z1 )(B − z2 ) · · · (B − zk )xt = 0
⇔ xt is a linear combination of solutions to
(B − z1 )xt = 0, (B − z2 )xt = 0, . . . , (B − zk )xt = 0
⇔ xt = c1 z1−t + c2 z2−t + · · · + ck zk−t ,

for some constants c1 , . . . , ck .

35
 Homogeneous linear diff eqns with constant coefficients

1. The zi are real and distinct. e.g., z1 = 1.2, z2 = −1.3

c z−t + c z−t
1 1 2 2
1
c =1, c =0
1 2
c =0, c =1
1 2
0.8 c =−0.8, c =−0.2
1 2

0.6

0.4

0.2

−0.2

−0.4

−0.6

−0.8
0 2 4 6 8 10 12 14 16 18 20

36
 Reminder: Complex exponentials

a + ib = reiθ = r(cos θ + i sin θ),

p
where r = |a + ib| = a2 + b2
 
b
θ = tan−1 ∈ (−π, π].
a

Thus, r1 eiθ1 r2 eiθ2 = (r1 r2 )ei(θ1 +θ2 ) ,

z z̄ = |z|2 .

37
 Homogeneous linear diff eqns with constant coefficients

2. The zi are complex and distinct.

As before, a(B)xt = 0
⇔ xt = c1 z1−t + c2 z2−t + · · · + ck zk−t .

If z1 6∈ R, since a1 , . . . , ak are real, we must have the complex conjugate

root, zj = z¯1 . And for xt to be real, we must have cj = c¯1 . For example:

xt = c z1−t + c̄ z¯1 −t
= r eiθ |z1 |−t e−iωt + r e−iθ |z1 |−t eiωt
 
= r|z1 |−t ei(θ−ωt) + e−i(θ−ωt)
= 2r|z1 |−t cos(ωt − θ)

where z1 = |z1 |eiω and c = reiθ .

38
 Homogeneous linear diff eqns with constant coefficients

2. The zi are complex and distinct. e.g., z1 = 1.2 + i, z2 = 1.2 − i

c z−t + c z−t
1 1 2 2
1
c=1.0+0.0i
c=0.0+1.0i
0.8 c=−0.8−0.2i

0.6

0.4

0.2

−0.2

−0.4

−0.6

−0.8

−1
0 2 4 6 8 10 12 14 16 18 20

39
 Homogeneous linear diff eqns with constant coefficients

2. The zi are complex and distinct. e.g., z1 = 1 + 0.1i, z2 = 1 − 0.1i

c z−t + c z−t
1 1 2 2
2
c=1.0+0.0i
c=0.0+1.0i
c=−0.8−0.2i
1.5

0.5

−0.5

−1

−1.5

−2
0 10 20 30 40 50 60 70

40