Autoregressive models

ST 434/534

Donald E.K. Martin

North Carolina St. U.

August 31, 2022

Donald E.K. Martin (North Carolina St. U.) Autoregressive models August 31, 2022 1 / 29
 Outline

1 Autoregressive processes
The general AR(p) model
The AR(1) case
AR(2) processes

Donald E.K. Martin (North Carolina St. U.) Autoregressive models August 31, 2022 2 / 29
 Autoregressive processes

Though the autoregressive and moving average representations are

useful, they are not practical for model building since they contain an
infinite number of parameters (which is impossible to estimate with
the finite number of observations we have for real data sets).

Donald E.K. Martin (North Carolina St. U.) Autoregressive models August 31, 2022 3 / 29
 Defining the general AR(p) model

If, in the autoregressive representation of the process we have

π1 = ϕ1 , π2 = ϕ2 , . . . , πp = ϕp and πk = 0, k > p, then the
resulting process is said to be an autoregressive process of order
p, which is denoted as AR(p). The model is defined as

Z̃t = ϕ1 Z̃t−1 + . . . + ϕp Z̃t−p + at . (1)

In (1), the {ϕj } are constant parameters, {at } is a white noise

process, and Z̃t = Zt − µ.

The model may be written as ϕ(B)Z̃t = at , where

ϕ(B) = 1 − ϕ1 B − . . . − ϕp B p .

Donald E.K. Martin (North Carolina St. U.) Autoregressive models August 31, 2022 4 / 29
 Invertibility and stationarity conditions

Since 1 + ∞
P Pp
j=1 |πj | = 1 + j=1 |ϕj | < ∞, the AR(p) process is
always invertible.

For stationarity, the MA coefficients must be absolutely

summable. Since ψ(B) = π −1 (B), this is determined by the
coefficients of ϕ(B). The required condition is that the roots of
ϕ(B) = 0 lie outside the unit circle. (For more details, see the
discussion on page 55 of the class text.)

Donald E.K. Martin (North Carolina St. U.) Autoregressive models August 31, 2022 5 / 29
 Autocorrelation function for the stationary pth
order case
Assuming stationarity in the general AR(p) case, where

Z̃t = ϕ1 Z̃t−1 + . . . + ϕp Z̃t−p + at ,

we multiply by Z̃t−k and take expectation on both sides to obtain, for

k ≥ 1,

E [Z̃t−k Z̃t ] = ϕ1 E [Z̃t−k Z̃t−1 ] + . . . + ϕp E [Z̃t−k Z̃t−p ] + E [Z̃t−k at ] (2)

which gives γk = ϕ1 γk−1 + ϕ2 γk−2 + . . . + ϕp γk−p . Dividing through

by γ0 gives

ρk = ϕ1 ρk−1 + ϕ2 ρk−2 + . . . + ϕp ρk−p (3)

with ρ−k = ρk for k ≥ 1.

Donald E.K. Martin (North Carolina St. U.) Autoregressive models August 31, 2022 6 / 29
 Note that we have used that E [Z̃t−k at ] = 0, as Z̃t−k depends on
noise terms at times t − k, t − k − 1, . . . . This will not be the
case when k = 0.

Equation (3) may be written as ϕ(B)ρk = 0, if we allow B to

operate on k instead of t, an expression very similar to that for
the series itself, but with 0 on the right hand side instead of at .

Each real root will contributed a damped exponential term to

the autocorrelation function. Complex roots (which come in
pairs) contribute a damped sinusoidal term.

In general, the autocorrelation function of a stationary AR(p)

process will consist of a combination of damped exponentials
and damped sine waves. (See page 56 of the class text.)

Donald E.K. Martin (North Carolina St. U.) Autoregressive models August 31, 2022 7 / 29
 Yule-Walker equations
Taking k = 1, 2, . . . , p in (2), we obtain the Yule-Walker equations,

ρ1 = ϕ1 + ϕ2 ρ1 + . . . + ϕk ρk−1
ρ2 = ϕ1 ρ1 + ϕ2 + . . . + ϕk ρk−2
.. .. ..
. . ··· .
ρp = ϕ1 ρp−1 + ϕ2 ρp−2 + . . . + ϕp

These equations, along with ρ0 = 1, as well as the corresponding

equations for autocovariances, may be used to solve for those
quantities in terms of parameters.

The equations may also be used to solve for the parameters in terms
of the autocorrelations, which can be useful for parameter estimation
given estimates of autocorrelations.
Donald E.K. Martin (North Carolina St. U.) Autoregressive models August 31, 2022 8 / 29
 Variance and partial autocorrelation function
For k = 0 in (2), we have

γ0 = ϕ1 γ1 + . . . + ϕp γp + σa2 ,

or
γ0 − ϕ1 γ1 − . . . − ϕp γp = σa2 .
Since γk = γ0 ρk , we have

σa2
σZ2 = (4)
1 − ϕ1 ρ1 − . . . − ϕp ρp

The partial autocorrelations ϕkk may be computed using formulas for

ratios of determinants, as given previously. Note that from the
definition of the model that ϕkk = 0 for k > p.

Donald E.K. Martin (North Carolina St. U.) Autoregressive models August 31, 2022 9 / 29
 The AR(1) case
For the case where p = 1, Z̃t − ϕ1 Z̃t−1 = at ;

Z̃t = at + ϕ1 Z̃t−1
= at + ϕ1 (at−1 + ϕ1 Z̃t−2 )
= at + ϕ1 at−1 + ϕ21 Z̃t−2
= at + ϕ1 at−1 + ϕ21 at−2 + ϕ31 at−3 + . . .
∞
X
= ϕj1 at−j . (5)
j=0

This result may also be obtained using (1 − ϕ1 (B))Z̃t = at or

Z̃t = 1−ϕa1t (B) , and the result for the geometric series.

Donald E.K. Martin (North Carolina St. U.) Autoregressive models August 31, 2022 10 / 29
 Geometric sums and series

Let
Sr = 1 + x + x 2 + ... + x r (6)
xSr = x + x 2 + x 3 + ... + x r + x r +1 . (7)
Subtracting (7) from (6), we obtain

(1 − x)Sr = 1 − x r +1

or
1 − x r +1
.Sr =
1−x
If |x| < 1, taking limits as r → ∞ gives the result.

Donald E.K. Martin (North Carolina St. U.) Autoregressive models August 31, 2022 11 / 29
 Thus an AR(1) is an infinite order MA (this holds for general
order p).

From the infinite MA representation we obtain that we have

absolute summability of the ψ coefficients and the process is
stationary if and only if |ϕ1 | < 1.

Written in terms of the backshift operator, (1 − ϕ1 B)Z̃t = at ; so

that ϕ(B) = 1 − ϕ1 B. The root of ϕ(B) = 0 is then B = 1/ϕ1 ,
which must be greater than 1 in absolute value (outside the unit
circle) for stationarity.

Donald E.K. Martin (North Carolina St. U.) Autoregressive models August 31, 2022 12 / 29
 Note in (5) that if |ϕ1 | > 1, the coefficients blow up as the
exponent j goes to ∞.

If ϕ1 = 1 so that from (5) Z̃t = ∞

P
j=0 at−j , the process does not
have finite variance and is not wide-sense stationary. In that
case, {Z̃t } is called a random walk.

P∞ j
If ϕ1 = −1 so that Z̃t = j=0 (−1) at−j , the variance is again
not finite.

Donald E.K. Martin (North Carolina St. U.) Autoregressive models August 31, 2022 13 / 29
 Computing the infinite MA representation in
another manner

We have shown that

1 = (1 − ϕ1 B)(1 + ψ1 B + ψ2 B 2 + . . .).

The ψ weights are obtained by by equating the coefficients of the two

polynomials in B, 1 (on the left hand side) and
(1 − ϕ1 B)(1 + ψ1 B + ψ2 B 2 + . . .) on the right hand side. Multiplying
on the right hand side,

1 = 1 + (ψ1 − ϕ1 )B + (ψ2 − ϕ1 ψ1 )B 2 + (ψ3 − ϕ1 ψ2 )B 3 + . . . .

We then have ψ1 = ϕ1 , ψ2 = ϕ21 , and in general, ψk = ϕk1 , k ≥ 1.

Donald E.K. Martin (North Carolina St. U.) Autoregressive models August 31, 2022 14 / 29
 Mean, ACF
By definition, E [Z̃t ] = 0; and using (2),
"∞ ∞
#
X X j
γk = E [Z̃t Z̃t+k ] = E ϕi1 at−i ϕ1 at+k−j
i=0 j=0

∞ ∞
X X
i σa2
= σa2 ϕi1 ϕk+i
1 = ϕk1 σa2 ϕ21 = ϕk1 = ϕk1 Var [Z̃t ],
i=0 i=0
1 − ϕ21

assuming that |ϕ1 | < 1. The expression forVar [Z̃t ] is (4) for p = 1.
By the assumed stationarity, γk = γ−k , so that the autocorrelation
function, for k = 0, ±1, ±2, . . . , is then
|k|
ρk = ϕ1 .

Donald E.K. Martin (North Carolina St. U.) Autoregressive models August 31, 2022 15 / 29
 PACF
ϕ11 = ρ1 = ϕ1 and ϕkk = 0, k ≥ 2, since

1 ρ1 ρ2 . . . ρk−2 ρ1
ρ1 1 ρ1 . . . ρk−3 ρ2
.. .. .. .. .. ..
. . . . . .
ρk−1 ρk−2 ρk−3 . . . ρ 1 ρk
ϕkk = = 0.
1 ρ1 ρ2 . . . ρk−2 ρk−1
ρ1 1 ρ1 . . . ρk−3 ρk−2
.. .. .. ... .. ..
. . . . .
ρk−1 ρk−2 ρk−3 . . . ρ1 1

(The last column of the matrix in the numerator of the expression for
ϕkk is ρ1 = ϕ1 times the first column). Alternatively, we can simply
look at the form of the autoregression Z̃t = ϕ1 Z̃t−1 + at .)
Donald E.K. Martin (North Carolina St. U.) Autoregressive models August 31, 2022 16 / 29
 Alternate computation method

The autocorrelation function may also be obtained using the

Yule-Walker equations, which give

ρk = ϕ1 ρk−1 , k ≥ 1. (8)

To determine the condition for stationarity (since it was assumed in

the Yule-Walker method), we have that since |ρk | ≤ 1 for k > 0,
|ϕ1 | ≤ 1. But if |ϕ1 | = 1, ρk = 1 for all k, which is the degenerate
case (the series is constant). Thus |ϕ1 | < 1 for a proper stationary
process. Using (8) and the symmetry of the autocorrelation function,
|k|
ρk = ϕ1 .

Donald E.K. Martin (North Carolina St. U.) Autoregressive models August 31, 2022 17 / 29
 data work.test;
lastz=0.0; *started at time -101 with z=0;
do i= -100 to 1500;
a=rannor(32565);
z=0.8*lastz+a;
if i > 0 then output; *getting rid of starting values;
lastz=z;
end;
run;

proc arima data=work.test;

identify var=z;
run;
estimate p=1;
run;
quit;

Donald E.K. Martin (North Carolina St. U.) Autoregressive models August 31, 2022 18 / 29
 AR(1) ϕ1 = 0.8

Figure: 1.

Donald E.K. Martin (North Carolina St. U.) Autoregressive models August 31, 2022 19 / 29
 AR(1) ϕ1 = −0.8

Figure: 2.

Donald E.K. Martin (North Carolina St. U.) Autoregressive models August 31, 2022 20 / 29
 AR(2) processes
Here, Z̃t = ϕ1 Z̃t−1 + ϕ2 Z̃t−2 + at .

The process is invertible since it is an autoregressive process of

finite order.

We determine stationarity by determining whether the roots of

ϕ(B) = 1 − ϕ1 B − ϕ2 B 2 = 0 are outside the unit circle, which
implies that
ϕ2 + ϕ1 <1
ϕ2 − ϕ1 <1
−1 < ϕ2 < 1

(Use the quadratic formula and algebraic manipulation.)

Donald E.K. Martin (North Carolina St. U.) Autoregressive models August 31, 2022 21 / 29
 Autocorrelation function

Assuming that the process is stationary, since

Z̃t = ϕ1 Z̃t−1 + ϕ2 Z̃t−2 + at , multiplying by Z̃t−k and taking
expectation gives, for k ≥ 1,

E [Z̃t−k Z̃t ] = ϕ1 E [Z̃t−k Z̃t−1 ] + ϕ2 E [Z̃t−k Z̃t−2 ] + E [Z̃t−k at ]

or
γk = ϕ1 γk−1 + ϕ2 γk−2 , (9)
for k ≥ 1 and thus also

ρk = ϕ1 ρk−1 + ϕ2 ρk−2 . (10)

Donald E.K. Martin (North Carolina St. U.) Autoregressive models August 31, 2022 22 / 29
 Computing the autocorrelation function
For k = 1 and k = 2 we have, from (10),

ρ1 = ϕ1 + ϕ2 ρ1

and
ρ2 = ϕ1 ρ1 + ϕ2 ,
which implies that
ϕ1
ρ1 = ,
1 − ϕ2
ϕ21 ϕ2 + ϕ2 − ϕ22
ρ2 = + ϕ2 = 1 = ϕ1 ρk−1 + ϕ2 ρk−2 .
1 − ϕ2 1 − ϕ2

We can use (10) for k ≥ 2 to compute the ρk recursively.

Donald E.K. Martin (North Carolina St. U.) Autoregressive models August 31, 2022 23 / 29
 Partial autocorrelation function
ϕ1
We have ϕ11 = ρ1 = 1−ϕ2
,

1 ρ1
ρ1 ρ2 ρ2 − ρ21
ϕ22 = =
1 ρ1 1 − ρ21
ρ1 1
 2   2
ϕ1 +ϕ2 −ϕ22 ϕ1
1−ϕ2
− 1−ϕ2
=  2
ϕ1
1 − 1−ϕ 2

ϕ21 (1 − ϕ2 ) + ϕ2 (1 − ϕ2 )2 − ϕ21
=
(1 − ϕ2 )2 − ϕ21
ϕ2 [(1 − ϕ2 )2 − ϕ21 ]
= = ϕ2 .
(1 − ϕ2 )2 − ϕ21
Donald E.K. Martin (North Carolina St. U.) Autoregressive models August 31, 2022 24 / 29
 Partial autocorrelation function

1 ρ1 ρ1 1 ρ1 ϕ1 + ϕ2 ρ1
ρ1 1 ρ2 ρ1 1 ϕ1 ρ1 + ϕ2
ρ2 ρ1 ρ3 ρ2 ρ1 ϕ1 ρ2 + ϕ2 ρ1
ϕ33 = = =0
1 ρ1 ρ2 1 ρ1 ρ2
ρ1 1 ρ1 ρ1 1 ρ1
ρ2 ρ1 1 ρ2 ρ1 1

since the last column in the numerator is a linear combination of the

first two columns. Similarly, ϕkk = 0 for k ≥ 4. (This makes sense,
since the model is Z̃t = ϕ1 Z̃t−1 + ϕ2 Z̃t−2 + at .)

Donald E.K. Martin (North Carolina St. U.) Autoregressive models August 31, 2022 25 / 29
 Stationary?

Determine whether or not the following AR(2) processes are

stationary.

Z̃t = 0.6Z̃t−1 − 0.08Z̃t−2 + at ;

Z̃t = 1.5Z̃t−1 − 0.5Z̃t−2 + at ;

Z̃t = 0.2Z̃t−1 − 0.6Z̃t−2 + at ;

Z̃t = −0.8Z̃t−1 + 0.4Z̃t−2 + at ;

Donald E.K. Martin (North Carolina St. U.) Autoregressive models August 31, 2022 26 / 29
 Stationary? (solutions)
Z̃t = 0.6Z̃t−1 − 0.08Z̃t−2 + at . Here ϕ(B) = 1 − 0.6B + 0.08B 2
= (1 − 0.2B)(1 − 0.4B) = 0 if B = 5 or B = 5/2. (stationary)
Z̃t = 1.5Z̃t−1 − 0.5Z̃t−2 + at . Here ϕ(B) = 1 − 1.5B + 0.5B 2
= (1 − B)(1 − 0.5B) = 0 if B = 1 or B = 2. (not stationary;
B = 1 not outside the unit circle)
Z̃t = 0.2Z̃t−1 − 0.6Z̃t−2 + at . Here ϕ(B)= 1 − 0.2B + 0.6B 2 .
√ √
The roots are .2± 1.2
.04−2.4
= 61 ± i 1.2 2.36
. The roots are
1 2.36
complex, with squared modulus 36
+ 1.44
, which is greater than
1. (stationary)
Z̃t = −0.8Z̃t−1 + 0.4 Z̃ + at . Here
√ t−2 √
ϕ(B) = 1 + 0.8B − 0.4B 2 .
The roots are −.8±−0.8
.64+1.6 2.24
= 1 ± 0.8 . The roots are real. The
positive root is greater than 1, but the negative root is less than
1 in absolute value. (not stationary)
Donald E.K. Martin (North Carolina St. U.) Autoregressive models August 31, 2022 27 / 29
 data work.test;
lastz=0.0; lastz2=0.0;
do i= -100 to 1500;
a=rannor(4387);
z=0.2*lastz-0.6*lastz2+a;
if i > 0 then output;
lastz2=lastz;
lastz=z;
end;
run;

proc arima data=work.test;

identify var=z;
run;
estimate p=2;
run;
quit;
Donald E.K. Martin (North Carolina St. U.) Autoregressive models August 31, 2022 28 / 29
 AR(2) ϕ1 = 0.2, ϕ2 = −0.6

Donald E.K. Martin (North Carolina St. U.) Autoregressive models August 31, 2022 29 / 29