Time Series Analysis and Forecasting
Uploaded by
asifatikurTime Series Analysis and Forecasting
Uploaded by
asifatikurDSA5021: Time Series Analysis and Forecasting1
Dr. Md. Rezaul Karim
PhD(KU Leuven & UHasselt), MS(Biostatistics, UHasselt), MS(Statistics, JU)
Professor, Department of Statistics and Data Science
Jahangirnagar University (JU), Savar, Dhaka - 1342, Bangladesh
Mobile: 01912605556, Email: [email protected]
Fall - 2024
1
These course slides should not be reproduced nor used by others (without
permission).
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 1 / 271
Lecture Outline I
1 Chapter 1: Introduction
1.1 What is a Time Series?
1.2 Importance of Time Series Analysis
1.3 Time series data vs. Cross-sectional data
1.4 Pooled Data vs Longitudinal Data
1.5 Component of Time Series
1.6 Mathematical Models for Time Series
1.7 Time Series plot in Python
1.8 Time Series plot in R
1.9 R Packages for Time Series Analysis
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 2 / 271
Lecture Outline II
1.10 Time Series Decomposition in R
1.11 Time Series Forecasting
1.12 Starting in Eviews
2 Chapter 2: Trend Analysis
2.1 A General Approach to Time Series Modeling
2.2 Trends in Time Series
2.3 Graphic (or free-hand curve tting) method
2.4 Method of semi-averages
2.5 Estimating a trend by least squares
2.6 Method of Moving Average Smoothing
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 3 / 271
Lecture Outline III
2.7 Exponential Moving Average (Exponential Smoothing)
2.8 Moving Average in Python
2.9 Detrend of Time Series
3 Chapter 3: Measurement of Seasonal Variation
3.1 Seasonal Variation
3.2 Seasonal Index
3.3 Method of Simple Averages
3.4 Ratio to trend method
3.5 Ratio-to-moving-average method
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 4 / 271
Lecture Outline IV
3.6 Link Relative Method
4 Chapter 4: Stationary Processes (with Quiz # 1)
4.1 Measures of Cyclical Variation
4.2 Means, Variances, Covariances and Autocorrelation Function
4.3 Correlogram
4.4 Python & R Code: Autocorrelation Plot (Correlogram)
4.5 Types of Stationarity
4.6 White Noises
4.7 The Random Walk Model
4.8 Integrated Processes
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 5 / 271
Lecture Outline V
4.9 Partial Autocorrelation Function
4.10 Python & RCode: Partial Autocorrelation Plot
4.11 Eviews: Autocorrelation Function (ACF) and Correlogram
4.12 Statistical Signi cance of Autocorrelation Coe cients
5 Chapter 5: Models for Stationary Time Series
5.1 The AR(p) Autoregressive Model
5.2 Autocovariance Function for AR(p)
5.3 Autocorrelation Function
5.4 The MA(q) Moving average Model
5.5 The ARMA(p, q) Model
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 6 / 271
Lecture Outline VI
5.6 The Autocorrelation Function of ARMA(1)
5.7 The ARMA(p, q) Model
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 7 / 271
Syllabus
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 8 / 271
Text Book & References List
Text Book
1 Peter J. Brockwell, Richard A. Davis. (2016):Introduction to Time
Series and Forecasting, 3rd Edition, Springer, New York.
2 Jonathan D. Cryer, Kung-Sik Chan, (2010): Time Series Analysis
With Applications in R, Springer.
Reference list
1 Jason Brownlee. (2020): Introduction to Time Series Forecasting
with Python - How to Prepare Data and Develop Models to Predict
the Future, First Edition, Machine Learning Mastering.
2 Avishek Pal, PKS Prakash. (2017): Practical time series analysis-
master time series data processing, visualization, and modeling using
Python, Packt Publishing
3 Gupta, S. C., and V. K. Kapoor. (2019): Fundamentals of Applied
Statistics. Sultan Chand & Sons.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 9 / 271
Chapter 1: Introduction
Chapter 1: Introduction
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 10 / 271
Chapter 1: Introduction
1 Chapter 1: Introduction
1.1 What is a Time Series?
1.2 Importance of Time Series Analysis
1.3 Time series data vs. Cross-sectional data
1.4 Pooled Data vs Longitudinal Data
1.5 Component of Time Series
1.6 Mathematical Models for Time Series
1.7 Time Series plot in Python
1.8 Time Series plot in R
1.9 R Packages for Time Series Analysis
1.10 Time Series Decomposition in R
1.11 Time Series Forecasting
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 11 / 271
Chapter 1: Introduction
Dierent types of data
data
▸ data are individual units of information
▸ a datum describes a single quality or quantity of some object or
phenomenon.
data can be broadly classi ed into the following categories:
▸ time series data
▸ cross-sectional data
▸ panel data
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 12 / 271
Chapter 1: Introduction What is a Time Series?
What is a Time Series?
. a time series is a sequence of observations on a variable measured at
successive points in time or over successive periods of time
. it is a series of data points indexed (or listed or graphed) in time order
. most commonly, a time series is a sequence taken at successive
equally spaced points in time
. the measurements may be taken every hour, day, week, month, or
year, or at any other regular interval
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 13 / 271
Chapter 1: Introduction What is a Time Series?
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 14 / 271
Chapter 1: Introduction What is a Time Series?
Time Series Analysis
. time series analysis involves developing models that best capture or
describe an observed time series in order to understand the underlying
causes
. the primary objective of time series analysis is to develop
mathematical models that provide plausible descriptions from sample
data
. comprises methods for analyzing time series data in order to extract
meaningful statistics and other characteristics of the data
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 15 / 271
Chapter 1: Introduction What is a Time Series?
What are the Dierent Types of Time Series?
there are two types of time series, such as:
discrete time series
▸ a time series is said to be discrete when observations are taken only at
a speci c times, usually equally spaced
continuous time series
▸ a time series is said to be continuous when observations are recorded
continuously over some time interval
▸ e.g., a binary process at continuous time is a continuous time series
Discrete time series can arise in several ways:
▸ Sampled: Given a continuous time series, we could read o the values
at equal intervals of time to give a discrete time series, sometimes
called a sampled series. The sampling interval between successive
readings must be carefully chosen so as to lose little information.
▸ Aggregated: Aggregate the values over equal intervals of a continuous
time series. E.g., monthly exports and daily rainfalls.
in this course, we will more focus on discrete time series
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 16 / 271
Chapter 1: Introduction Importance of Time Series Analysis
Importance of Time Series Analysis
. pro t of experience
. safety from future
. utility studies
. budgetary analysis
. stock market analysis
. yield projections
. process and quality control
. inventory studies
. economic forecasting
. risk analysis & evaluation of charges
. census analysis
. ...
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 17 / 271
Chapter 1: Introduction Time series data vs. Cross-sectional data
Time series data vs. Cross-sectional data
a cross-sectional data is obtained
by taking observations from
a time series data is a dataset that
multiple individuals at the same
gives you more insight into the
point in time
variation of the variable you are
ex: (i) 2003 year-end book value
studying over time
per share for all New York Stock
ex: (i) monthly returns for past 5
Exchange-listed companies (ii) a
years (ii) a researcher collects
dataset of salaries across a city -
salary data across a city on a
they have all been gathered at one
month-by-month basis
point in time and thus we refer to
the data as cross-sectional
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 18 / 271
Chapter 1: Introduction Pooled Data vs Longitudinal Data
Pooled Data vs Longitudinal Data
. Pooled (or panel) data is where the two are combined together. i.e. a
salary dataset can contain observations collected at one point in time,
as well as across dierent time periods
. Longitudinal data is a set of observations on dierent characteristics
of the single observational unit collected at dierent times e.g.
observations on a set of nancial ratios for a single company over a
10-year period
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 19 / 271
Chapter 1: Introduction Component of Time Series
Component of Time Series
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 20 / 271
Chapter 1: Introduction Component of Time Series
Component of Time Series/Time Series Decomposition
Time series are usually decomposed into:
1 Trend component or Long-term Movement (Tt ): the smooth long
term direction of a time series
2 Periodic Changes or Short Term Fluctuations
1 Seasonal component (St ): patterns of change in a time series within a
year which tends to repeat each year
2 Cyclical component (Ct ): the rise and fall of a time series over periods
longer than one year
3 Random component or Irregular Movements (It ): the residuals
▸ Episodic unpredictable but identi able
▸ Residual also called chance uctuation and unidenti able
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 21 / 271
Chapter 1: Introduction Component of Time Series
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 22 / 271
Chapter 1: Introduction Component of Time Series
Which type of components are here?
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 23 / 271
Chapter 1: Introduction Mathematical Models for Time Series
Mathematical Models for Time Series
1 additive model: a time series (yt ) using an additive model can be
thought of as
yt = Tt + Ct + St + It (1)
2 multiplicative Model: a multiplicative model would be
yt = Tt × Ct × St × It (2)
3 mixed model:
yt = Tt Ct + St It ⎫
⎪
⎪
⎪
yt = Tt + Ct St It ⎬ (3)
⎪
⎪
⎪
yt = Tt + St + Ct It ⎭
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 24 / 271
Chapter 1: Introduction Mathematical Models for Time Series
the model (1) or (2) can be use to obtain a measure of one or more
of the components by elimination, viz., subtraction or division
for example, if trend component (Tt ) is known, then using
multiplication model, it can be isolated from the given time series to
give:
yt original values
St × Ct × It = =
Tt trend values
thus, for the annual data, for which the seasonal component St is not
here, we have
yt
yt = Tt × Ct × It ⇒ Ct × It =
Tt
remarks: an additive model would be used when the variations around the
trend do not vary with the level of the time series whereas a multiplicative
model would be appropriate if the trend is proportional to the level of the
time series
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 25 / 271
Chapter 1: Introduction Mathematical Models for Time Series
Is time series additive or multiplicative?
It depends on which kind of additivity and multiplicativity you are looking
for. Is it trend or seasonal?
Below steps to be followed:
1 decompose the time series using stl() or decompose() functions in
R
2 look seasonality component(if looking for seasonal
additive/multiplicative) or trend(if looking for trend
additive/multiplicative)
3 additive model is used when the variance in the graph doesn't change
over dierent values of the time series.
4 if the variance is higher when the time series is higher then it often
means we should use a multiplicative models.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 26 / 271
Chapter 1: Introduction Mathematical Models for Time Series
Preliminary Adjustments before Analysing Time Series
before beginning the actual work of analysing a time series it is necessary
to make the following adjustments in the raw data
adjustment for calendar variation
adjustment for population changes
adjustment for price changes and
adjustment for comparability
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 27 / 271
Chapter 1: Introduction Mathematical Models for Time Series
Classical seasonal decomposition
Component of Time Series Analysis:
Trend Component (Tt ): Represents the long-term movement in the
data, indicating if it is increasing, decreasing, or constant.
Seasonal Component (St ): Captures regular, repeating patterns at
speci c intervals, such as holiday sales.
Cyclic Component (Ct ): Refers to long-term uctuations in uenced
by economic cycles, not xed in length.
Irregular Component (It ): Represents random variations not
explained by the other components, including noise and anomalies.
Classical decomposition separates a time series yt into three main
components: trend, seasonal, and residual. This method uses either an
additive or multiplicative model.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 28 / 271
Chapter 1: Introduction Mathematical Models for Time Series
Note that classical decomposition does not explicitly model cyclical
variations. Any cyclical patterns may be present in the residuals.
To analyze cyclical variations, consider:
Business Cycle Analysis: Techniques for identifying and analyzing
business cycles.
Advanced Time Series Models: Models like SARIMA can account
for complex patterns, including cyclical variations.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 29 / 271
Chapter 1: Introduction Mathematical Models for Time Series
Time Series Plot
1 the rst thing to do in any data analysis task is to plot the data
2 graphs enable many features of the data to be visualised, including
patterns, unusual observations, changes over time, and relationships
between variables
Data Frequency
Annual 1
Quarterly 4
Monthly 12
Weekly 52
the frequency is the number of observations before the seasonal pattern
repeats
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 30 / 271
Chapter 1: Introduction Time Series plot in Python
Getting Python!
1 Basic Python 3.8.2 download (i.e., IDE then e.g. PyChanrm)
https://www.python.org/downloads/
2 Python 3.7 from the Anaconda DistributionSpyder, jupyter python
https://www.anaconda.com/download/
3 Online Python (the best one!)
https://colab.research.google.com/notebooks/intro.ipynb
CAUTION: the basic Python installation from https://www.python.org
will NOT be su cient for all the data-analysis!
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 31 / 271
Chapter 1: Introduction Time Series plot in Python
Upload your data set
upload your AirPassengers data-set
from google.colab import files
uploaded = files.upload()
read your dataset
import pandas as pd
df = pd.read_csv("AirPassengers.csv")
print(df.head(14))
# print(df.tail())
Convert the month column into a datetime object
df['Month'] = pd.to_datetime(df['Month'], format='%Y-%m')
print(df.head())
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 32 / 271
Chapter 1: Introduction Time Series plot in Python
the next thing we can do is convert the month column to an index.
This will allow us to more easily work with some of the packages we
will be covering later:
df.index = df['Month']
del df['Month']
print(df.head())
next, let's generate a time series plot using Seaborn and Matplotlib.
import matplotlib.pyplot as plt
import seaborn as sns
sns.lineplot(data=df)
plt.ylabel("Number of Passengers")
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 33 / 271
Chapter 1: Introduction Time Series plot in Python
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 34 / 271
Chapter 1: Introduction Time Series plot in Python
the decomposition
import statsmodels.api as sm
result = sm.tsa.seasonal_decompose(df['Passengers'],
model='multiplicative')
result.plot()
see the trend compoment
result.trend.plot()
see the seasonal variation
result.seasonal.plot()
all plots together
result.plot()
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 35 / 271
Chapter 1: Introduction Time Series plot in Python
Decomposition using AirPassengers dataset
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 36 / 271
Chapter 1: Introduction Time Series plot in Python
Decomposition in Python
In Python, the sm.tsa.seasonal_decompose function from the
statsmodels library is commonly used for decomposing time series data.
Trend Estimation:
▸ The trend component is estimated using a moving average method.
▸ The default window size for the moving average is based on the
frequency of the time series. For instance, if the data is monthly, the
default window size will typically be 12.
Seasonal Estimation:
▸ The seasonal component is computed as the average of the detrended
data for each season (e.g., each month).
▸ This calculation considers the periodicity of the data, allowing it to
capture repeating patterns eectively.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 37 / 271
Chapter 1: Introduction Time Series plot in Python
See the data set
import matplotlib.pyplot as plt
import seaborn as sns
flights = sns.load_dataset("flights")
flights.head()
Time series plot
may_flights = flights.query("month == 'May'")
sns.lineplot(data=may_flights, x="year", y="passengers")
Pivot the dataframe to a wide-form representation:
flights_wide = flights.pivot("year", "month", "passengers")
flights_wide.head()
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 38 / 271
Chapter 1: Introduction Time Series plot in Python
Flights dataset
To plot a single vector, pass it to data. If the vector is a
pandas.Series, it will be plotted against its index:
sns.lineplot(data=flights_wide["May"])
Passing the entire wide-form dataset to data plots a separate line for
each column:
sns.lineplot(data=flights_wide)
Passing the entire dataset in long-form mode will aggregate over
repeated values (each year) to show the mean and 95% con dence
interval:
sns.lineplot(data=flights, x="year", y="passengers")
# sns.lineplot(data=may_flights, x="year", y="passengers")
Assign a grouping semantic (hue, size, or style) to plot separate lines
sns.lineplot(data=flights, x="year", y="passengers",
hue="month")
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 39 / 271
Chapter 1: Introduction Time Series plot in R
Getting started R
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 40 / 271
Chapter 1: Introduction Time Series plot in R
Getting started RStudio
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 41 / 271
Chapter 1: Introduction Time Series plot in R
Import your data in R
1 prepare you data
2 save your data in an external.txt or .csv les
3 import your data into R as follows:
# If .txt tab file, use this
my_data <- read.delim(file.choose())
# Or, if .csv file, use this
my_data <- read.csv(file.choose())
Alternatively,
MyRData <- read.table("c:/Data.csv", header=TRUE,sep=",",
row.names="id")
MyRData <- read.csv("Data.csv", header = TRUE)
MyRData2 <- read.table("Data1.txt", header = TRUE)
Here, we will use the built-in R data set named iris
my_data <- iris # Store the data in the variable my_data
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 42 / 271
Chapter 1: Introduction Time Series plot in R
Preparing Time Series Data
How to prepare a Time Series in R?
# ts (inputData, frequency = 4, start = c(1959, 2)) # frequency
4 => Quarterly Data
ts (1:12, frequency = 12, start = 1990) # freq 12 => Monthly
data.
ts (inputData, start=c(2009), end=c(2014), frequency=1) #
Yearly Data
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 43 / 271
Chapter 1: Introduction R Packages for Time Series Analysis
R Packages for Time Series Analysis
. Time series decomposition: decomp(), decompose(), arima(),
stl()
. Time series forecasting:forecast
. Time Series Clustering: TSclust
. Dynamic Time Warping (DTW): dtw
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 44 / 271
Chapter 1: Introduction R Packages for Time Series Analysis
Data: AirPassengers
▸▸Data AirPassengers: monthly totals of Box Jenkins international
airline passengers, 1949 to 1960. It has 144(=12 Ö12) values
## load time series data
plot(AirPassengers)
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 45 / 271
Chapter 1: Introduction Time Series Decomposition in R
Time Series Decomposition
## time series decomposation
apts <- ts(AirPassengers, frequency = 12)
f <- decompose(apts)
plot(f$figure, type = "b") # seasonal figures
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 46 / 271
Chapter 1: Introduction Time Series Decomposition in R
Decomposition of additive time series
plot(f)
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 47 / 271
Chapter 1: Introduction Time Series Decomposition in R
Reading & Plotting Time Series Data
. the rst thing that you will want to do to analyse your time series
data will be to read it into R, and to plot the time series
. you can read data into R using the scan() function, which assumes
that your data for successive time points is in a simple text le with
one column
. ex: the age of death of 42 successive kings of England has been read
into the variable `kings'
kings <- scan("http://robjhyndman.com/tsdldata/misc/kings.dat",
skip=3)
kings
kingstimeseries <- ts(kings)
kingstimeseries
plot.ts(kingstimeseries)
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 48 / 271
Chapter 1: Introduction Time Series Decomposition in R
this time series could probably be described using an additive model,
since the random uctuations in the data are roughly constant in size
over time
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 49 / 271
Chapter 1: Introduction Time Series Decomposition in R
Reading & Pltting Time Series Data
. another example is a data set of the number of births per month in
New York city, from January 1946 to December 1959 (originally
collected by Newton)
. this data is available in the le
http://robjhyndman.com/tsdldata/data/nybirths.dat
births <-
scan("http://robjhyndman.com/tsdldata/data/nybirths.dat")
birthstimeseries <- ts(births, frequency=12, start=c(1946,1))
birthstimeseries
plot.ts(birthstimeseries)
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 50 / 271
Chapter 1: Introduction Time Series Decomposition in R
there seems to be seasonal variation in the number of births per
month: there is a peak every summer, and a trough every winter
it seems that this time series could probably be described using an
additive model, as the seasonal uctuations are roughly constant in
size over time and do not seem to depend on the level of the time
series, and the random uctuations also seem to be roughly constant
in size over time
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 51 / 271
Chapter 1: Introduction Time Series Decomposition in R
Reading & Pltting Time Series Data
. monthly sales for a souvenir shop at a beach resort town in
Queensland, Australia, for January 1987-December 1993 (original
data from Wheelwright and Hyndman, 1998)
. this data is available in the le
http://robjhyndman.com/tsdldata/data/fancy.dat
souvenir <-
scan("http://robjhyndman.com/tsdldata/data/fancy.dat")
souvenirtimeseries <- ts(souvenir, frequency=12, start=c(1987,1))
souvenirtimeseries
plot.ts(souvenirtimeseries)
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 52 / 271
Chapter 1: Introduction Time Series Decomposition in R
it appears that an additive model is not appropriate for describing this
time series, since the size of the seasonal uctuations and random
uctuations seem to increase with the level of the time series
thus, we may need to transform the time series in order to get a
transformed time series that can be described using an additive model
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 53 / 271
Chapter 1: Introduction Time Series Forecasting
Time Series Forecasting
. forecasting involves taking models t on historical data and using
them to predict future observations
. an important distinction in forecasting is that the future is completely
unavailable and must only be estimated from what has already
happened
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 54 / 271
Chapter 1: Introduction Time Series Forecasting
Methods of Time Series Analysis
1 time-domain methods
▸ trend analysis
▸ AR/MA/ARMA models
▸ ACF and partial autocorrelation function
▸ forecasting
▸ parameter estimation
▸ ARIMA models/seasonal ARIMA models
2 frequency-domain methods
▸ spectral analysis
▸ spectral density
▸ periodogram
▸ spectral estimation
▸ wavelet analysis
▸ state space models
▸ ARMAX models.
▸ forecasting, Kalman lter.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 55 / 271
Chapter 1: Introduction Time Series Forecasting
What is the best software for time series analysis?
SAS
Python
Eviews
STRATA
JMP Software from SAS
Minitab
SPSS
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 56 / 271
Chapter 1: Introduction Starting in Eviews
Starting in Eviews
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 57 / 271
Chapter 1: Introduction Starting in Eviews
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 58 / 271
Chapter 1: Introduction Starting in Eviews
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 59 / 271
Chapter 1: Introduction Starting in Eviews
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 60 / 271
Chapter 1: Introduction Starting in Eviews
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 61 / 271
Chapter 1: Introduction Starting in Eviews
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 62 / 271
Chapter 1: Introduction Starting in Eviews
Time Series plot in Eviews
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 63 / 271
Chapter 1: Introduction Starting in Eviews
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 64 / 271
Chapter 1: Introduction Starting in Eviews
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 65 / 271
Chapter 1: Introduction Starting in Eviews
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 66 / 271
Chapter 1: Introduction Starting in Eviews
Homework
Collect the number of COVID-19 cases in Bangladesh daily and draw
a time series plot. Interpret the plot. Decompose the COVID-19 cases
and interpret your ndings. Send your data (in an Excel le) and all
output with interpretation in Word le.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 67 / 271
Chapter 2: Trend Analysis
Chapter 2: Trend Analysis
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 68 / 271
Chapter 2: Trend Analysis
2 Chapter 2: Trend Analysis
2.1 A General Approach to Time Series Modeling
2.2 Trends in Time Series
2.3 Graphic (or free-hand curve tting) method
2.4 Method of semi-averages
2.5 Estimating a trend by least squares
2.6 Method of Moving Average Smoothing
2.7 Exponential Moving Average (Exponential Smoothing)
2.8 Moving Average in Python
2.9 Detrend of Time Series
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 69 / 271
Chapter 2: Trend Analysis A General Approach to Time Series Modeling
A General Approach to Time Series Modeling
1 plot the time series data and examine the main features of the graph,
checking in particular whether there is
(a). a trend,
(b). a seasonal component,
(c). any apparent sharp changes in behavior,
(d). any outlying observations
2 remove the trend and seasonal components to get stationary residuals
to achieve this goal it may sometimes be necessary to apply a
preliminary transformation to the data
3 choose a model to t the residuals, making use of various sample
statistics including the sample autocorrelation function
4 forecasting will be achieved by forecasting the residuals and then
inverting the transformations described above to arrive at forecasts of
the original series {yt }
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 70 / 271
Chapter 2: Trend Analysis A General Approach to Time Series Modeling
A General Approach to Time Series Modeling
5 An extremely useful alternative approach is to express the series in
terms of its Fourier components, which are sinusoidal waves of
dierent frequencies (cf. Example 1.1.4 in the textbook)
this approach is especially important in engineering applications
such as signal processing and structural design
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 71 / 271
Chapter 2: Trend Analysis A General Approach to Time Series Modeling
Models with Trend and Seasonality
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 72 / 271
Chapter 2: Trend Analysis A General Approach to Time Series Modeling
Nonseasonal Model with Trend
the graph suggests trying a model of the form
yt = Tt + εt
where Tt is a slowly changing function known as the trend
component and εt is a random noise component with zero mean
how to estimate Tt ?
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 73 / 271
Chapter 2: Trend Analysis A General Approach to Time Series Modeling
Models with Trend and Seasonal Components
the graph suggests trying a model of the form
yt = Tt + St + εt
where St is a function of seasonal component
how to estimate Tt and St ?
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 74 / 271
Chapter 2: Trend Analysis Trends in Time Series
Trends in Time Series
a trend is a long-term increase or decrease in the level of the time
series
types of trends
1 Deterministic Trends: these are trends that consistently increase or
decrease
2 Stochastic Trends: these are trends that increase and decrease
inconsistently
for each type of trends, we can get
1 Global Trends: these are trends that apply to the whole time series
2 Local Trends: these are trends that apply to parts or subsequences of
a time series
generally, global trends are easier to identify and address
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 75 / 271
Chapter 2: Trend Analysis Trends in Time Series
Identifying and Removing a Trend
identifying a trend
▸ you can plot time series data to see if a trend is obvious or not
▸ the di culty is that in practice, identifying a trend in a time series can
be a subjective process
▸ create line plots of your data and inspect the plots for obvious trends
▸ add linear and nonlinear trend lines to your plots and see if a trend is
obvious
removing a trend
▸ a time series with a trend is called non-stationary
▸ an identi ed trend can be modeled
▸ once modeled, it can be removed from the time series dataset
▸ this is called detrending the time series
▸ if a dataset does not have a trend or we successfully remove the trend,
the dataset is said to be trend stationary
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 76 / 271
Chapter 2: Trend Analysis Trends in Time Series
Measurement of Trend
trend can be studied and/or measured by the following methods
graphic (or free-hand curve tting) method
▸ you can see some practical applications in
https://lankabd.com/Home under Analysis Tools for stock price
analysis
method of semi-averages
estimating a trend by least squares
method of moving averages
▸ simple moving average (SMA)
▸ exponential moving average (EMA)
▸ weighted moving average
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 77 / 271
Chapter 2: Trend Analysis Graphic (or free-hand curve tting) method
Graphic (or free-hand curve tting) method
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 78 / 271
Chapter 2: Trend Analysis Graphic (or free-hand curve tting) method
Advantages and Disadvantages of the Free-Hand Curve
Fitting
advantages
this method is very simple and easy to understand
it is applicable to linear and non-linear trends
it gives us an idea about the rise and fall of the time series
for every long time series, the graph of the original data enables us to
decide on the application of more mathematical models for the
measurement of a trend
disadvantages
this method is not mathematical in nature, so dierent people may
draw a dierent trend
this method is subjective in nature, predictions obtained by this
method depends on the personal bias and judgment of the
investigator handling the data
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 79 / 271
Chapter 2: Trend Analysis Method of semi-averages
Method of semi-averages
Procedure of method of semi-averages
the data is divided into two equal parts. In case of odd number of
data, two equal parts can be made simply by omitting the middle year
the average of each part is calculated, thus we get two points
each point is plotted at the mid-point (year) of each half
join the two points by a straight line
the straight line can be extended on either side
the line is the trend line by the methods of semi-averages
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 80 / 271
Chapter 2: Trend Analysis Method of semi-averages
Example
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 81 / 271
Chapter 2: Trend Analysis Method of semi-averages
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 82 / 271
Chapter 2: Trend Analysis Method of semi-averages
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 83 / 271
Chapter 2: Trend Analysis Estimating a trend by least squares
Method of curve tting by principle of least squares
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 84 / 271
Chapter 2: Trend Analysis Estimating a trend by least squares
Example:
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 85 / 271
Chapter 2: Trend Analysis Estimating a trend by least squares
Least Square Estimation
1 let a trend line is yt = Tt + εt = β0 + β1 t + εt
2 minimizing function:
n n
Q = min ∑ ε2t = min ∑ (yt − ŷt )2
β0 ,β1 t= 1 β0 ,β1 t=1
n
= min ∑ (yt − β0 − β1 t)2
β0 ,β1 t=1
3 the estimators for β0 and β1 can be found by solving the following
normalized equation
∂Q ∂ n 2
= ∑ (yt − β0 − β1 t) = 0 (4)
∂β0 ∂β0 t=1
and
∂Q ∂ n 2
= ∑(yt − β0 − β1 t) = 0 (5)
∂β1 ∂β1 i=1
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 86 / 271
Chapter 2: Trend Analysis Estimating a trend by least squares
LS Estimator
from (4), we have
n
∑(yt − β0 − β1 t)(−1) = 0
i= 1
n n
⇒ ∑ yt − nβ0 − β1 ∑ t = 0
i= 1 i= 1
or
β0 = ȳ − β1 t̄
from (5), we have
n
∑(yt − β0 − β1 t)(−t) = 0
i= 1
n n n
⇒ ∑ tyt − β0 ∑ t − β1 ∑ t 2 = 0 (6)
i= 1 i= 1 1
i=
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 87 / 271
Chapter 2: Trend Analysis Estimating a trend by least squares
putting the value of β0 in Equation (6), we have
n n n
2
∑ tyt − (ȳ − β1 t̄) ∑ t − β1 ∑ t = 0
t= 1 t= 1 t= 1
n n n n
⇒ ∑ tyt − ȳ ∑ t + β1 t̄ ∑ t − β1 ∑ t 2
t= 1 1
t= t= 1 1
t=
n n
⇒ ∑ tyt − nȳ t̄ + nβ1 t̄ 2 − β1 ∑ t 2 = 0
t= 1 t= 1
n n
⇒ ∑ tyt − nȳ t̄ − β1 ( ∑ t 2 − nt̄ 2 ) = 0
t= 1 t= 1
n
∑ tyi − nt̄ ȳ
⇒β1 =
t=1
∑ t 2 − nt̄ 2
n
t= 1
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 88 / 271
Chapter 2: Trend Analysis Estimating a trend by least squares
hence, the least square estimator of β0 and β1 are
β̂0 = ȳ − β̂1 t̄
n
∑ tyi − nt̄ ȳ
β̂1 =
t=1
∑ t 2 − nt̄ 2
n
t= 1
the expression of β̂1 can also be written as
n
∑ (t − t̄)(yt − ȳ )
β̂1 =
t= 1
∑ (t − t̄)2
n
t= 1
because
n n n n
2 2
∑ (t − t̄) = ∑ t − nt̄
2 and ∑ (t − t̄)(yt − ȳ ) = ∑ tyi − nt̄ ȳ
t= 1 t= 1 t= 1 t=1
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 89 / 271
Chapter 2: Trend Analysis Estimating a trend by least squares
Example:
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 90 / 271
Chapter 2: Trend Analysis Estimating a trend by least squares
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 91 / 271
Chapter 2: Trend Analysis Estimating a trend by least squares
(see details in Gupta and Kapoor. (2019))
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 92 / 271
Chapter 2: Trend Analysis Estimating a trend by least squares
Estimating a trend by least squares
Let's analyze a time series of global mean annual temperature downloaded
fromhttps://climate.nasa.gov/system/internal_resources/
details/original/647_Global_Temperature_Data_File.txt. These
data are in degrees Celsius measured as an anomaly from a 1951-1980
base. This is climatology jargon for saying that the sample mean of the
temperature over the interval 1951-1980 was subtracted from all time
points.
global_temp <- read.table("Global_Temperature.txt",header=TRUE)
str(global_temp)
# 'data.frame': 141 obs. of 3 variables:
# $ Year : int 1880 1881 1882 1883 1884 1885 1886 1887
1888 1889 ...
# $ No_Smoothing: num -0.16 -0.08 -0.1 -0.16 -0.28 -0.33 -0.31
-0.36 -0.17 -0.1 ...
# $Lowess : num -0.08 -0.12 -0.16 -0.2 -0.23 -0.26 -0.27
-0.27 -0.26 -0.25 ...
plot(Annual~Year,data=global_temp,ty="l")
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 93 / 271
Chapter 2: Trend Analysis Estimating a trend by least squares
Mean global temperature anomaly, degrees C
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 94 / 271
Chapter 2: Trend Analysis Estimating a trend by least squares
Fitting a least squares model with a quadratic trend
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 95 / 271
Chapter 2: Trend Analysis Estimating a trend by least squares
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 96 / 271
Chapter 2: Trend Analysis Estimating a trend by least squares
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 97 / 271
Chapter 2: Trend Analysis Estimating a trend by least squares
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 98 / 271
Chapter 2: Trend Analysis Method of Moving Average Smoothing
Method of Moving Average Smoothing
a moving average (rolling average or running average) is a calculation
to analyze data points by creating a series of averages of dierent
subsets of the full data set
method of moving averages
▸ simple moving average (SMA)
▸ exponential moving average (EMA)
▸ weighted moving average also known as an exponentially weighted
moving average (EWMA)
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 99 / 271
Chapter 2: Trend Analysis Method of Moving Average Smoothing
Method of moving averages
the mean over the last k data-points (days in this example) is
denoted as SMAk and calculated as:
yn−k+1 + yn−k+2 ⋯ + yn 1 n
SMAk = = ∑ yi
k k i=n−k+1
the EMA for a series Y may be calculated recursively:
⎧
⎪
⎪y1 , t=0
St = ⎨
⎪
⎩αyt + (1 − α) ⋅ St−1 , t > 0
⎪
where:
▸ the coe cient α represents the degree of weighting decrease, a
constant smoothing factor between 0 and 1
▸ a higher α discounts older observations faster
▸ yt is the value at a time period t
▸ St is the value of the EMA at any time period t
this is known as simple exponential smoothing and Holt's exponential
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU)
smoothing MS in Data Science Fall - 2024 100 / 271
Chapter 2: Trend Analysis Method of Moving Average Smoothing
Simple moving average (SMA)
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 101 / 271
Chapter 2: Trend Analysis Method of Moving Average Smoothing
Simple moving average (SMA)
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 102 / 271
Chapter 2: Trend Analysis Method of Moving Average Smoothing
● you can see some practical applications in https://lankabd.com/Home
under Analysis Tools for stock price analysis
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 103 / 271
Chapter 2: Trend Analysis Exponential Moving Average (Exponential Smoothing)
Simple Exponential Smoothing
the simplest of the exponentially smoothing methods is naturally
called exponential moving aveage or simple exponential smoothing
(ES)
this method is suitable for forecasting data with no clear trend or
seasonal pattern
for example, the data in the following gure do not display any clear
trending behaviour or any seasonality
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 104 / 271
Chapter 2: Trend Analysis Exponential Moving Average (Exponential Smoothing)
the simple exponential smoothing (ES) for a series Y:
⎧
⎪
⎪y1 , t=0
ŷt+1 = ⎨
⎪
⎩αyt + (1 − α) ⋅ ŷt−1 , t > 0
⎪
where:
▸ the coe cient α represents the degree of weighting decrease, a
constant smoothing factor between 0 and 1
▸ a higher α discounts older observations faster
▸ yt is the value at a time period t
▸ ŷt is the value of the simple ES at any time period t
Forecasts are calculated using weighted averages, where the weights
decrease exponentially as observations come from further in the past
the smallest weights are associated with the oldest observations:
ŷt+1∣t = αyt + α(1 − α)yt−1 + α(1 − α)2 yt−2 + ⋯,
where 0 ≤α≤1 is the smoothing parameter
you can see some practical applications in
https://lankabd.com/Home under Analysis Tools for stock price
analysis
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 105 / 271
Chapter 2: Trend Analysis Exponential Moving Average (Exponential Smoothing)
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 106 / 271
Chapter 2: Trend Analysis Moving Average in Python
Moving Average in Python
data read
import pandas as pd
import numpy as np
# Create a numpy array of years and sales
arr = np.array([['2000', 4], ['2001',7],
['2002',4], ['2003',9],
['2004',7], ['2005',10]])
# Transpose array
arr_tp = arr.transpose()
print(arr_tp)
make a data frame
# Create a dataframe
df = pd.DataFrame({'Years': arr_tp[0], 'Sales': arr_tp[1]})
print(df)
simple moving average of size 3
# Let's calculate SMA for a window size of 3,
df['MA'] = df['Sales'].rolling(window=3).mean()
df.head()
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 107 / 271
Chapter 2: Trend Analysis Moving Average in Python
Airpassenger dataset
import matplotlib.pyplot as plt
import seaborn as sns
sns.lineplot(data=df, x="Month", y="#Passengers")
plt.ylabel("Number of Passengers")
simple moving average of size 3, 4 and 12
df['SMA_3'] = df['#Passengers'].rolling(window=3).mean()
df['SMA_4'] = df['#Passengers'].rolling(window=4).mean()
df['SMA_12'] = df['#Passengers'].rolling(window=12).mean()
df.head(10)
SMA plot with time series plot
import matplotlib.pyplot as plt
plt.figure(figsize=[15,10])
plt.grid(True)
sns.lineplot(data=df, x="Month",
y="#Passengers",label='#Passengers')
plt.ylabel("Number of Passengers")
plt.plot(df['SMA_3'],label='SMA 3 Months')
plt.plot(df['SMA_4'],label='SMA 4 Months')
plt.plot(df['SMA_12'],label='SMA 12 Months')
plt.legend(loc=2)
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 108 / 271
Chapter 2: Trend Analysis Moving Average in Python
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 109 / 271
Chapter 2: Trend Analysis Moving Average in Python
Airpassenger dataset
import matplotlib.pyplot as plt
import seaborn as sns
sns.lineplot(data=df, x="Month", y="#Passengers")
plt.ylabel("Number of Passengers")
EMA with α = 2/(span + 1)
df['#Passengers'].rolling(window=12).mean()
#df.iloc[:,0] means first column (#passenger column) with
all rows
# Specify decay in terms of span, alpha=2/(span+1), for
span>=1.
df['EMA'] = df.iloc[:,0].ewm(span=40,adjust=False).mean()
df.head(10)
SMA plot with EMA plot
import matplotlib.pyplot as plt
plt.figure(figsize=[15,10])
plt.grid(True)
sns.lineplot(data=df, x="Month",
y="#Passengers",label='#Passengers')
plt.ylabel("Number of Passengers")
plt.plot(df['SMA_3'],label='SMA 3 Months')
plt.plot(df['SMA_4'],label='SMA 4 Months')
plt.plot(df['SMA_12'],label='SMA 12 Months')
plt.plot(df['EMA'],label='EMA smoothing')
plt.legend(loc=2)
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 110 / 271
Chapter 2: Trend Analysis Moving Average in Python
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 111 / 271
Chapter 2: Trend Analysis Detrend of Time Series
Detrend of Time Series
Detrend by Dierencing
▸ perhaps the simplest method to detrend a time series is by dierencing
▸ speci cally, a new series is constructed where the value at the current
time step is calculated as the dierence between the original
observation and the observation at the previous time step
∆yt = yt − yt−1
▸ this approach works well for data with a linear trend
▸ if the trend is quadratic (the change in the trend also increases or
decreases), then a dierence of the already-dierenced dataset can be
taken, a second level of dierencing and so on
Detrend by Model Fitting
▸ the residuals from the t of the model are a detrended form of the
dataset. Polynomial curve tting and other nonlinear models can also
be used
ε̂t = yt − ŷt
where ŷt is a predicted value of yt
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 112 / 271
Chapter 2: Trend Analysis Detrend of Time Series
Manual Dierencing
write a function of dierences and forecast value
# create a differenced series
def difference(dataset, interval=1):
diff = list()
for i in range(interval, len(dataset)):
value = dataset[i] - dataset[i - interval]
diff.append(value)
return diff
# invert differenced forecast
def inverse_difference(last_ob, value):
return value + last_ob
de ne a dataset with a linear trend
# define a dataset with a linear trend
data = [i+1 for i in range(20)]
print(data)
see the results
# difference the dataset
diff = difference(data)
print(diff)
# invert the difference
inverted = [inverse_difference(data[i], diff[i]) for i in
range(len(diff))]
print(inverted)
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 113 / 271
Chapter 2: Trend Analysis Detrend of Time Series
Example: Automatic Dierencing
See for the Shampoo dataset
from pandas import read_csv
from pandas import datetime
from matplotlib import pyplot
series = read_csv('shampoo-sales.csv', index_col=0)
X = series.values
diff = difference(X)
series.plot()
pyplot.show()
diff1 = series.diff()
pyplot.plot(diff1)
pyplot.show()
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 114 / 271
Chapter 2: Trend Analysis Detrend of Time Series
Nonparametric methods (advanced) for trend estimation
In previous, we assumed that the Tt (or mean function in nonseasonal
model) had a certain known parametric form. This may not always be the
case. If we have no apriori knowledge of the features in the trend Tt , we
can estimate the trend using a nonparametric approach.
rolling windows
Sieve estimators
polynomial tting
smoothing technique
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 115 / 271
Chapter 2: Trend Analysis Detrend of Time Series
Homework-2
1 Download daily DSEX Index data. You can download it from
https://www.investing.com/indices/
dhaka-stock-exchange-broad-historical-data.
2 Draw a time series plot of the DSEX Index for the last 30 years and
make comments.
3 Find decomposition of the DSEX Index and interpret your results.
4 Draw a time series plot of the DSEX Index for 2021 and then draw a
linear and quadratic trend lines. Comments on your ndings.
5 Draw a plot of SMA(5), SMA(14), SMA(21) with original data.
6 Estimate the trend by using SMA and EMA.
7 Make the detrend lines, and comment on your results.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 116 / 271
Chapter 3: Measurement of Seasonal Variation
Chapter 3: Measurement of Seasonal Variation
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 117 / 271
Chapter 3: Measurement of Seasonal Variation
3 Chapter 3: Measurement of Seasonal Variation
3.1 Seasonal Variation
3.2 Seasonal Index
3.3 Method of Simple Averages
3.4 Ratio to trend method
3.5 Ratio-to-moving-average method
3.6 Link Relative Method
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 118 / 271
Chapter 3: Measurement of Seasonal Variation Seasonal Variation
Seasonal Variation
seasonal variations are uctuations that coincide with certain seasons
and are repeated year after year
seasonal uctuations in a time series can be contrasted with cyclical
patterns the latter occur when the data exhibits rises and falls that
are not of a xed period
such non-seasonal uctuations are usually due to economic conditions
and are often related to the "business cycle"; their period usually
extends beyond a single year, and the uctuations are usually of at
least two years
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 119 / 271
Chapter 3: Measurement of Seasonal Variation Seasonal Variation
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 120 / 271
Chapter 3: Measurement of Seasonal Variation Seasonal Variation
Motivation
there are several main reasons for studying seasonal variation:
the description of the seasonal eect provides a better understanding
of the impact this component has upon a particular series
understanding seasonal uctuations help plan for su cient goods and
materials on hand to meet varying seasonal demand
after establishing the seasonal pattern, methods can be implemented
to eliminate it from the time-series to study the eect of other
components such as cyclical and irregular variations
this elimination of the seasonal eect is referred to as deseasonalizing
or seasonal adjustment of data
to use the past patterns of the seasonal variations to contribute to
forecasting and the prediction of the future trends, such as in climate
normals
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 121 / 271
Chapter 3: Measurement of Seasonal Variation Seasonal Index
Seasonal Index
Seasonal variation is measured in terms of an index, called a seasonal
index
an index value is attached to each period of the time series within a
year this implies that if monthly data are considered there are 12
separate seasonal indices, one for each month
there are four methods commonly used for the measurement of
seasonal variations
▸ method of simple averages
▸ ratio to trend method
▸ ratio-to-moving-average method
▸ method of link relatives
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 122 / 271
Chapter 3: Measurement of Seasonal Variation Method of Simple Averages
Method of Simple Averages
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 123 / 271
Chapter 3: Measurement of Seasonal Variation Method of Simple Averages
Method of Simple Averages
(See details in Gupta and Kapoor (2019))
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 124 / 271
Chapter 3: Measurement of Seasonal Variation Method of Simple Averages
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 125 / 271
Chapter 3: Measurement of Seasonal Variation Method of Simple Averages
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 126 / 271
Chapter 3: Measurement of Seasonal Variation Method of Simple Averages
Advantages & Disadvantages
advantages
method of simple average is easy and simple to execute
disadvantages
this method is based on the additive model of the time series
this method is based on the basic assumption that the data do not
contain any trend and cyclic components
since most of the economic and business time series have trends and
as such this method though simple is not of much practical utility
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 127 / 271
Chapter 3: Measurement of Seasonal Variation Ratio to trend method
Ratio to trend method
1 compute the trend values by the principle of least squares by tting
an appropriate mathematical curve (straight line, 2nd degree
parabolic curve or exponential curve, etc.)
2 express the original data as the percentage of the trend values
3 assuming the multiplicative model, these percentages will, therefore
contain the seasonal, cyclic and irregular components
4 the cyclic and irregular components are then eliminated by averaging
the percentages for dierent months (quarters) if the data are
monthly (quarterly), thus leaving us with indices of seasonal variations
5 nally these indices, obtained in step 4, are adjusted to a total of
1200 for monthly data or 400 for quarterly data by multiplying them
throughout by a constant k given by
1200 400
k= and k=
Total of the indices Total of the indices
for monthly and quarterly data respectively
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 128 / 271
Chapter 3: Measurement of Seasonal Variation Ratio to trend method
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 129 / 271
Chapter 3: Measurement of Seasonal Variation Ratio to trend method
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 130 / 271
Chapter 3: Measurement of Seasonal Variation Ratio to trend method
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 131 / 271
Chapter 3: Measurement of Seasonal Variation Ratio to trend method
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 132 / 271
Chapter 3: Measurement of Seasonal Variation Ratio to trend method
Advantages & Disadvantages
advantages
this method is an improvement over the simple averages method and
this method assumes a multiplicative model
compared with the method of monthly averages this method is
certainly a more logical procedure for measuring seasonal variations
it has an advantage over the ratio to moving average method that in
this method we obtain ratio to trend values for each period for which
data are available where as it is not possible in ratio to moving
average method
disadvantages
the main defect of the ratio to trend method is that if there are
cyclical swings in the series, the trend whether a straight line or a
curve can never follow the actual data as closely as a 12- monthly
moving average does so a seasonal index computed by the ratio to
moving average method may be less biased than the one calculated
by the ratio to trend method
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 133 / 271
Chapter 3: Measurement of Seasonal Variation Ratio-to-moving-average method
Ratio-to-moving-average method
1 nd the centered 12 monthly (or 4 quarterly) moving averages of the
original data values in the time-series
2 express each original data value of the time-series as a percentage of
the corresponding centered moving average values obtained in
step(1), i.e., in a multiplicative time-series model, we get (Original
data values) / (Trend values)× 100 = (T× C × S × I) / (T × C) ×
100 = (S × I ) × 100 this implies that the ratioto-moving average
represents the seasonal and irregular components.
3 arrange these percentages according to months or quarter of given
years
4 nd the averages over all months or quarters of the given years.
5 if the sum of these indices is not 1200 (or 400 for quarterly gures),
multiply then by a correction factor = 1200 / (sum of monthly
indices) otherwise, the 12 monthly averages will be considered as
seasonal indices
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 134 / 271
Chapter 3: Measurement of Seasonal Variation Ratio-to-moving-average method
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 135 / 271
Chapter 3: Measurement of Seasonal Variation Ratio-to-moving-average method
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 136 / 271
Chapter 3: Measurement of Seasonal Variation Ratio-to-moving-average method
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 137 / 271
Chapter 3: Measurement of Seasonal Variation Ratio-to-moving-average method
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 138 / 271
Chapter 3: Measurement of Seasonal Variation Ratio-to-moving-average method
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 139 / 271
Chapter 3: Measurement of Seasonal Variation Ratio-to-moving-average method
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 140 / 271
Chapter 3: Measurement of Seasonal Variation Ratio-to-moving-average method
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 141 / 271
Chapter 3: Measurement of Seasonal Variation Ratio-to-moving-average method
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 142 / 271
Chapter 3: Measurement of Seasonal Variation Ratio-to-moving-average method
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 143 / 271
Chapter 3: Measurement of Seasonal Variation Ratio-to-moving-average method
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 144 / 271
Chapter 3: Measurement of Seasonal Variation Ratio-to-moving-average method
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 145 / 271
Chapter 3: Measurement of Seasonal Variation Ratio-to-moving-average method
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 146 / 271
Chapter 3: Measurement of Seasonal Variation Ratio-to-moving-average method
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 147 / 271
Chapter 3: Measurement of Seasonal Variation Ratio-to-moving-average method
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 148 / 271
Chapter 3: Measurement of Seasonal Variation Ratio-to-moving-average method
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 149 / 271
Chapter 3: Measurement of Seasonal Variation Ratio-to-moving-average method
Advantages & Disadvantages
advantages
of all the methods of measuring seasonal variations, the ratio to
moving average method is the most satisfactory, exible and widely
used method
the uctuations of indices based on ratio to moving average method
is less than based on other methods
disadvantages
this method does not completely utilize the data for example in
case of 12-monthly moving average seasonal indices cannot be
obtained for the rst 6 months and last 6 months
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 150 / 271
Chapter 3: Measurement of Seasonal Variation Ratio-to-moving-average method
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 151 / 271
Chapter 3: Measurement of Seasonal Variation Ratio-to-moving-average method
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 152 / 271
Chapter 3: Measurement of Seasonal Variation Ratio-to-moving-average method
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 153 / 271
Chapter 3: Measurement of Seasonal Variation Link Relative Method
Link Relative Method
this method is slightly more complicated than other methods
this method is also known as Pearson's method
this method consists in the following steps
link relative is the value of one season expressed as a percentage of
the value of the preceding seasons
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 154 / 271
Chapter 3: Measurement of Seasonal Variation Link Relative Method
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 155 / 271
Chapter 3: Measurement of Seasonal Variation Link Relative Method
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 156 / 271
Chapter 3: Measurement of Seasonal Variation Link Relative Method
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 157 / 271
Chapter 3: Measurement of Seasonal Variation Link Relative Method
Assignment-2
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 158 / 271
Chapter 3: Measurement of Seasonal Variation Link Relative Method
Classroom Assessment (Quiz - 1)
Chapters 1-3
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 159 / 271
Chapter 4: Stationary Processes (with Quiz # 1)
Chapter 4: Stationary Processes (with Quiz # 1)
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 160 / 271
Chapter 4: Stationary Processes (with Quiz # 1)
4 Chapter 4: Stationary Processes (with Quiz # 1)
4.1 Measures of Cyclical Variation
4.2 Means, Variances, Covariances and Autocorrelation Function
4.3 Correlogram
4.4 Python & R Code: Autocorrelation Plot (Correlogram)
4.5 Types of Stationarity
4.6 White Noises
4.7 The Random Walk Model
4.8 Integrated Processes
4.9 Partial Autocorrelation Function
4.10 Python & RCode: Partial Autocorrelation Plot
4.11 Eviews: Autocorrelation Function (ACF) and Correlogram
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 161 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Measures of Cyclical Variation
Measuring Cyclic Variation
Cyclical variation refers to long-term uctuations in data, remaining
after removing the trend and seasonal components.
These cyclical variations are linked to economic cycles and last longer
than seasonal changes.
Identifying cyclical variations is challenging due to overlapping
irregular uctuations.
A common approach to isolate the irregular component is applying a
moving average (3, [OP]5 months) to reduce noise and reveal
clearer cyclical patterns.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 162 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Measures of Cyclical Variation
Methods for Measuring Cyclic Variation
There are several methods for measuring cyclical variation. One widely
used approach is the Residual Method, which we will discuss below.
Some other methods are:
1 Reference Cycle Analysis Method: This method involves
identifying a benchmark cycle and comparing other cycles to it.
2 Direct Percentage Variation Method: This technique measures
cyclical variation as the percentage dierence from the trend.
3 Fitting of Sine Functions through Harmonic Analysis: A more
sophisticated approach, tting sinusoidal functions to the cyclical
data.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 163 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Measures of Cyclical Variation
Steps of Residual Method for Measuring Cyclic Variation
When dealing with yearly data, there are typically no seasonal
uctuations, and the data can be decomposed into the following three
components:
Trend (Tt )
Cyclical (Ct )
Irregular (It )
However, when working with monthly or quarterly data, seasonal
uctuations need to be considered in addition to the trend and cyclical
components.
1 Estimate the Trend and Seasonal Components: Begin by
estimating the trend component Tt (representing long-term
movements) and the seasonal values St (capturing regular, periodic
uctuations).
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 164 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Measures of Cyclical Variation
2 Calculate the Components: Divide the original time series values
Yt by the estimated trend Tt and seasonal values St to isolate the
cyclical component Ct and irregular component It . Mathematically,
this can be represented as:
yt
= Ct × I t .
Tt × St
Here, the term Ct × I t represents the combined cyclical and irregular
components.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 165 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Measures of Cyclical Variation
3 Eliminate the Irregular Component: The irregular component It is
di cult to separate directly, as it contains random noise. To isolate
the cyclical variation, we apply a smoothed technique, such as a
moving average over a 3 or 5-month period, to reduce the noise and
highlight the cyclical pattern.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 166 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Measures of Cyclical Variation
Example of Cyclical Variation Computation
To illustrate the Residual Method, let's take a look at the example in the
table below. The table presents theoretical data that has been processed
using the residual method, with the following steps applied:
least square method is used to compute the trend
First, the
equation for yt = 244.01 + 2.36t , where t represents the quarter.
The trend values are calculated and presented in Column (5).
The data is then deseasonalized, detrended, and nally smoothed
using a 3-period moving average to produce indices of cyclical
variations.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 167 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Measures of Cyclical Variation
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 168 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Means, Variances, Covariances and Autocorrelation Function
Key Statistical Properties of Time Series
Mean Function: The mean function of a time series
{yt ; t = 1, 2, . . . , n} describes how the average value of the series
changes over time. For a given time series {yt }, the mean function is
de ned as the expected value of the series at time t:
µt = E [yt ]
▸ For stationary series: µt = µ, constant mean.
▸ For non-stationary series: µt varies with time.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 169 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Means, Variances, Covariances and Autocorrelation Function
Key Statistical Properties of Time Series
Variance Function: The variance of a time series {yt } at time t is a
measure of the dispersion or spread of the values around the mean. It
is de ned as the expected value of the squared deviation of yt from
its mean function µt :
Var(yt ) = E [(yt − µt )2 ]
▸ For stationary series: Var(yt ) = σ 2 , constant variance.
▸ For non-stationary series: Var(yt ) = σt2 , varying variance.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 170 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Means, Variances, Covariances and Autocorrelation Function
Key Statistical Properties of Time Series
Autocovariance Function (γ(h)): The autocovariance function of a
time series {yt } measures the covariance between values of the series
at dierent times t and t + h. It is de ned as:
γ(h) = Cov(yt , yt+h ) = E [(yt − µt )(yt+h − µt+h )]
▸ Measures covariance between yt and yt+h .
▸ For stationary series: γ(h) depends only on lag h.
▸ For non-stationary series: γ(h) depends on both t and h.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 171 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Means, Variances, Covariances and Autocorrelation Function
Key Statistical Properties of Time Series
Autocorrelation Function (ρ(h)): The autocorrelation function
(ACF) of a time series {yt } measures the correlation between values
of the series at dierent times t and t + h. It is de ned as the
normalized autocovariance function:
γ(h) Cov(yt , yt+h ) E [(yt − µt )(yt+h − µt+h )]
ρ(h) = = =
γ(0) Var(yt ) E [(yt − µt )2 ]
▸ Normalized autocovariance: ρ(h) = γ(h)
γ(0)
.
▸ For stationary series: ρ(h) depends only on lag h.
▸ ρ(h) ranges from −1 (perfect negative) to 1 (perfect positive).
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 172 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Correlogram
Correlogram
A Correlogram is a plot of the autocorrelation function (ACF) at
dierent lags h.
The ACF measures how correlated the values of the time series are at
dierent time lags.
The correlogram visually displays:
▸ Autocorrelation at lag h, denoted ρ(h),
▸ Signi cant positive or negative correlations at speci c lags.
The correlogram helps identify patterns:
▸ Seasonality: Periodic peaks in the ACF at xed lags.
▸ Trend: A slow decay of the autocorrelation function as lag increases.
▸ Randomness: No signi cant correlation at any lag (white noise).
A correlogram can be used to diagnose:
▸ Whether the time series is stationary or non-stationary.
▸ The presence of seasonality or other underlying structures.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 173 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Correlogram
Ref: Gujarati, D. N., Porter, D. C., & Gunasekar, S. (2012). Basic
econometrics. Tata McGraw-Hill Education.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 174 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Python & R Code: Autocorrelation Plot (Correlogram)
Python Code: Autocorrelation Plot (Correlogram)
# Importing the libraries.
import matplotlib.pyplot as plt
import numpy as np
# read dataset
df = pd.read_csv("AirPassengers.csv")
# Plotting the Autocorrelation plot.
x = pd.plotting.autocorrelation_plot(df['#Passengers']);
# ploting the Curve
x.plot()
# Display
plt.show()
# Alternatively
# import the plotting functions for act and pacf
from statsmodels.graphics.tsaplots import plot_acf, plot_pacf
# plot_acf(df['#Passengers']);
plot_acf(df['#Passengers'], lags=40);
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 175 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Python & R Code: Autocorrelation Plot (Correlogram)
R Code: Autocorrelation Plot (Correlogram)
AirPassengers <- read.csv("AirPassengers.csv")
# Autocorrelation plot
acf(AirPassengers[2])
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 176 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Types of Stationarity
Stationary of a Time Series
stationarity is an invariant property which means that statistical
characteristics of the time series do not change over time
time series are stationary if they do not have trend or seasonal eects
Types of Stationarity
strict stationarity
▸ the time series {yt } is said to be strictly stationary if for any nite
sequence of integers t1 , t2 , ⋯, tk and shift h the distribution of
(yt1 , yt2 , ⋯, ytk ) and (yt1+h , yt2+h , ⋯, ytk+h ) are the same
▸ in your general notation for denities, this strict stationarity requirement
can be written as
fYn1 ,Yn2 ,...,YnK (y1 , y2 , . . . , yK ) = fYn1 +h ,Yn2 +h ,...,YnK +h (y1 , y2 , . . . , yK )
weakly stationary (also called second order stationarity)
▸ a time series model which is both mean stationary and covariance
stationary is called weakly stationary
strict stationarity implies weak stationarity (check this)!
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 177 / 271
Chapter 4: Stationary Processes (with Quiz # 1) White Noises
White Noise
White Noise is a time series with the following characteristics:
▸ Zero Mean: The expected value of the series is zero: E [yt ] = 0.
▸ Constant Variance: The variance of the series is constant over time:
Var(yt ) = σ 2 .
▸ No Autocorrelation: The autocorrelation function (ACF) is zero for
all lags h ≠ 0, i.e., ρ(h) = 0 for all h ≠ 0.
if a time series is white noise, it is a sequence of random numbers and
cannot be predicted
if the series of forecast errors are not white noise, it suggests
improvements could be made to the predictive model
In a correlogram, white noise would show:
▸ No signi cant peaks in the autocorrelation plot.
▸ The autocorrelation function should be close to zero for all lags.
White noise is considered the opposite of a time series with trends or
seasonality.
if the variables in the series are drawn from a Gaussian distribution,
the series is called Gaussian white noise
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 178 / 271
Chapter 4: Stationary Processes (with Quiz # 1) White Noises
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 179 / 271
Chapter 4: Stationary Processes (with Quiz # 1) White Noises
Figure: Autocorrelation function for the white noise series
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 180 / 271
Chapter 4: Stationary Processes (with Quiz # 1) White Noises
Gaussian White Noise
Gaussian White Noise is a special case of white noise where the
values are drawn from a normal distribution:
▸ Mean: E [yt ] = 0
▸ Variance: Var(yt ) = σ 2 (constant over time)
▸ Normal Distribution: The values yt are normally distributed at each
time point, i.e., yt ∼ N (0, σ 2 ).
Key properties:
▸ The values yt are independent and identically distributed (i.i.d.)
random variables.
▸ The autocorrelation function (ACF) for Gaussian white noise is zero at
all lags except for lag h = 0, i.e., ρ(h) = 0 for h ≠ 0.
In a correlogram:
▸ No signi cant correlations at any lag, as all lags show a zero
autocorrelation.
▸ The correlogram will have a at line near zero for all lags.
Gaussian white noise is commonly used as a model for random
uctuations in many scienti c elds, such as physics, economics, and
signal processing.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 181 / 271
Chapter 4: Stationary Processes (with Quiz # 1) White Noises
Binary White Noise
Binary White Noise is a type of white noise where each data point
takes one of two possible values (e.g., 0 or 1) with constant
probability.
Key characteristics:
▸ Independence: The values yt are independent and identically
distributed (i.i.d.).
▸ Constant Probability: Each value yt has the same probability p of
being either 0 or 1 (e.g., P(yt = 1) = p and P(yt = 0) = 1 − p ).
▸ Zero Autocorrelation: The autocorrelation function (ACF) is zero for
all lags h ≠ 0, i.e., ρ(h) = 0 for h ≠ 0.
In a correlogram:
▸ The autocorrelation at lag h = 0 is equal to the variance (since the
series takes only two values).
▸ The autocorrelation for h ≠ 0 is zero.
Binary white noise is often used to model random binary events in
communication systems, signal processing, and random walk models.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 182 / 271
Chapter 4: Stationary Processes (with Quiz # 1) The Random Walk Model
Random Walk Model
A Random Walk is a stochastic process where each value is the sum
of the previous value and a random disturbance:
yt = yt−1 + t
where:
▸ yt is the value at time t ,
▸ yt−1 is the value at time t − 1,
▸ t is a random shock or error term at time t .
Key Properties:
▸ The error term t is typically assumed to be i.i.d. (independent and
identically distributed) with zero mean and constant variance:
t ∼ N (0, σ 2 ).
▸ The series follows a non-stationary process, meaning that its mean and
variance change over time.
▸ No mean-reverting behavior: The process does not tend to return to
a xed value but instead can drift in any direction over time.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 183 / 271
Chapter 4: Stationary Processes (with Quiz # 1) The Random Walk Model
Random Walk is often used to model nancial time series, such as
stock prices, or any process where the value at each point is
in uenced by random shocks and previous values.
In a correlogram:
▸ The autocorrelation function (ACF) of a random walk typically shows a
slow decay and does not approach zero quickly, re ecting the
persistence of the shock.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 184 / 271
Chapter 4: Stationary Processes (with Quiz # 1) The Random Walk Model
Random Walk Model with Drift
The Random Walk with Drift is an extension of the basic random
walk where a constant drift term µ is added to the model:
yt = yt−1 + µ + t
where:
▸ yt is the value at time t ,
▸ yt−1 is the value at time t − 1,
▸ µ is the constant drift term (the trend),
▸ t is the random shock or noise at time t .
Key Properties:
▸ Drift (µ): The value of the series changes by a constant value µ over
time, introducing a linear trend.
▸ t is assumed to be i.i.d., typically with mean 0 and variance σ 2 .
▸ The series is non-stationary because the mean changes over time due
to the drift.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 185 / 271
Chapter 4: Stationary Processes (with Quiz # 1) The Random Walk Model
Interpretation of Autocorrelation:
▸ The autocorrelation function (ACF) for a random walk with drift
decays more slowly than that of a simple random walk.
▸ The process tends to "drift" in the direction of the drift term, µ,
leading to long-term trends.
Applications:
▸ Stock prices: Modeling stock prices or asset returns that generally
increase or decrease over time.
▸ Economics: Modeling GDP growth or other economic indicators that
trend in a certain direction.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 186 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Integrated Processes
Integrated Processes (I(d) Process)
An Integrated Process is a time series that becomes stationary after
dierencing d times, denoted as I(d).
A process yt is integrated of order d if it requires d dierences to
become stationary.
Mathematical Formulation:
▸ First dierencing (d = 1): ∆yt = yt − yt−1
▸ Second dierencing (d = 2): ∆2 yt = ∆yt − ∆yt−1
if a time series Yt is stationary to begin with (i.e., it does not require
any dierencing), it is said to be integrated of order zero, denoted by
Yt ∼ I (0)
Key Characteristics:
▸ The process is non-stationary until dierenced d times.
▸ For d = 1, the process behaves like a random walk, with unit roots.
▸ Autocorrelation decays slowly as d increases.
thus, we will use the terms stationary time series and time series
integrated of order zero to mean the same thing
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 187 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Integrated Processes
Example: A simple I(1) process is:
yt = yt−1 + t
where t is white noise.
Applications:
▸ Used for modeling economic data such as GDP, in ation, and stock
prices.
▸ Helps in modeling long-term trends and non-stationary data.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 188 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Partial Autocorrelation Function
Partial Autocorrelation Function (PACF)
The Partial Autocorrelation Function (PACF) measures the
correlation between yt and yt+h after removing the eects of
intermediate lags.
Mathematical De nition:
φhh = correlation(yt , yt+h ∣ yt−1 , yt−2 , . . . , yt+h−1 )
Interpretation:
▸ PACF at lag 1 is the same as the ACF at lag 1.
▸ PACF at higher lags removes the in uence of intermediate lags,
showing the direct relationship.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 189 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Partial Autocorrelation Function
PACF and AR Models:
▸ The PACF is used to determine the order of an AR model.
▸ For AR(p), the PACF cuts o after lag p , meaning the partial
autocorrelations at lags > p are zero.
Example:
▸ For an AR(1) process, the PACF will show a signi cant spike at lag 1
and be zero at higher lags.
▸ For an AR(2) process, the PACF will show spikes at lags 1 and 2, and
be zero at higher lags.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 190 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Partial Autocorrelation Function
The general formula for PACF (Y , lag = k)
in general, the partial auto-correlation of Yi with a k lagged version of
itself i.e.Y(i−k) is a correlation between the following two variables:
▸ Variable 1: the amount of variance in Yi that is not explained by the
variance in Y(i−1) , Y(i−2) . . . Y(i−k+1) , and,
▸ Variable 2: the amount of variance in Y(i−k) that is not explained by
the variance in Y(i−1) , Y(i−2) . . . Y(i−k+1)
in general formula
Cov ([Yi ∣Y(i−1) Y(i−2) ⋯Y(i−k+1) ], [Y(i−2) ∣Y(i−1) , Y(i−2) , ⋯Y(i−k+1) ])
PACF (Yi , k) =
σ[Yi ∣Y(i−1) Y(i−2) ⋯Y(i−k+1) ] × σ[Y(i−2) ∣Y(i−1) ,Y(i−2) ,⋯Y(i−k+1) ]
the use of this function was introduced as part of the BoxJenkins approach
to time series modelling, whereby plotting the partial autocorrelative
functions one could determine the appropriate lags p in an AR(p) model or
in an extended ARIMA(p, d, q) model
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 191 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Partial Autocorrelation Function
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 192 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Partial Autocorrelation Function
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 193 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Partial Autocorrelation Function
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 194 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Partial Autocorrelation Function
Rules of Using ACF and PACF
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 195 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Partial Autocorrelation Function
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 196 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Partial Autocorrelation Function
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 197 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Partial Autocorrelation Function
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 198 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Python & RCode: Partial Autocorrelation Plot
Python Code: PACF
import pandas as pd
from statsmodels.graphics.tsaplots import plot_pacf
from statsmodels.tsa.stattools import pacf
import matplotlib.pyplot as plt
df = pd.read_csv("AirPassengers.csv")
plot_pacf(df['#Passengers'], title='PACF: Number of Passengers')
print(pacf(df['#Passengers']))
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 199 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Python & RCode: Partial Autocorrelation Plot
R Code: Partial Autocorrelation Plot
AirPassengers <- read.csv("AirPassengers.csv")
pacf(AirPassengers[2])
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 200 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Eviews: Autocorrelation Function (ACF) and Correlogram
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 201 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Eviews: Autocorrelation Function (ACF) and Correlogram
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 202 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Statistical Signi cance of Autocorrelation Coe cients
Classical Test based on CLT
the sample auto-correlation function of these sequence are
approximately iid with ρ̂ ∼ N (0, n1 )
If more than 2 or 3 values of ρ̂ fall outside of the bounds or that one
value falls far outside the bounds, then we accept the null hypothesis
i.e., the population sequence is not iid otherwise reject the null
hypothesis.
the 95% of the sample auto correlations (ρ̂) should fall between
−1.96 1.96
( √ , √ )
n n
If more than 2 or 3 values of ρ̂ fall outside of the bounds or that one
value falls far outside the bounds, then we accept the null hypothesis
i.e., the population sequence is not iid otherwise reject the null
hypothesis
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 203 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Statistical Signi cance of Autocorrelation Coe cients
The Portmanteau Test
to test we can consider the test statistic as:
h
Q = n ∑ ρ̂2 (j)
j= 1
we know that,
1 √
ρ̂ ∼ N (0, ) i.e., nρ̂ ∼ N (0, 1)
n
Q is approximately distributed as the sum of sequences of the
independent N ( 0, 1 ) random variables
√
nρ̂ (j) ; j = 1, 2, ⋯, h
i.e., as Chi-squared with h degrees of freedom
we therefore reject the iid hypothesis at level α if Q > χ21−α (h), where
χ21−α (h) is the 1 − α quantile of the chi-squared distribution with
h d.f .
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 204 / 271
Chapter 4: Stationary Processes (with Quiz # 1) Statistical Signi cance of Autocorrelation Coe cients
Re nement of the Portmanteau Test
Ljung and Box (1978) modi ed Q and is de ned as
h
ρ̂2 (j)
QLB = n (n + 2) ∑
j=1 (n − j)
whose distribution is better approximates by the Chi-squared
distribution with h d.f.
another Portmanteau Test, formulated by McLeod and Li (1983), can
be used for testing whether the data are an iid sequence of normally
distributed random variables. They de ned the test statistic as:
h
ρ̂2ww (k)
Q̃ = n (n + 2) ∑
k=1 (n − k)
where, ρ̂2ww (k) is the sample auto-correlations of the squared data
the hypothesis of iid normal data is then rejected at level α if the
observed value of Q̃ is larger than the (1 − α) quantile of the
χ21−α (h) distribution
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 205 / 271
Chapter 5: Models for Stationary Time Series
Chapter 5: Models for Stationary Time Series
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 206 / 271
Chapter 5: Models for Stationary Time Series
5 Chapter 5: Models for Stationary Time Series
5.1 The AR(p) Autoregressive Model
5.2 Autocovariance Function for AR(p)
5.3 Autocorrelation Function
5.4 The MA(q) Moving average Model
5.5 The ARMA(p, q) Model
5.6 The Autocorrelation Function of ARMA(1)
5.7 The ARMA(p, q) Model
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 207 / 271
Chapter 5: Models for Stationary Time Series The AR(p) Autoregressive Model
Autoregressive Model AR(p)
An AR(p) model is an Autoregressive model of order p. It models a
time series as a linear function of its previous values.
The general form of an AR(p) model is:
yt = φ1 yt−1 + φ2 yt−2 + ⋯ + φp yt−p + t
where:
▸ yt is the value of the time series at time t ,
▸ φ1 , φ2 , . . . , φp are the parameters of the model,
▸ p is the order of the model (number of previous values used),
▸ t is the white noise term at time t (i.i.d. with mean 0 and variance
σ 2 ).
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 208 / 271
Chapter 5: Models for Stationary Time Series The AR(p) Autoregressive Model
Key Concepts of AR(p)
Autoregression: The model predicts future values based on a linear
combination of past values.
Order (p ): The parameter p de nes how many previous values are
considered in the model.
Stationarity: The series must be stationary for the AR model to be
valid. This means its statistical properties should not change over
time.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 209 / 271
Chapter 5: Models for Stationary Time Series The AR(p) Autoregressive Model
Estimation of Parameters
Parameters φ1 , φ 2 , . . . , φ p are typically estimated using methods such as:
Ordinary Least Squares (OLS) or Yule-Walker equations for simpler
models.
Maximum Likelihood Estimation (MLE) in more complex cases.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 210 / 271
Chapter 5: Models for Stationary Time Series The AR(p) Autoregressive Model
Model Diagnostics
After tting an AR(p) model, perform diagnostics:
Residual Analysis: The residuals (t ) should resemble white noise
(i.i.d. with mean 0 and constant variance).
Autocorrelation Function (ACF): The ACF of residuals should be
close to 0 at all lags if the model is well-speci ed.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 211 / 271
Chapter 5: Models for Stationary Time Series The AR(p) Autoregressive Model
Choosing the Order (p)
The correct order p of the AR model can be selected using:
ACF/PACF: The ACF and Partial ACF (PACF) plots help in
identifying the number of signi cant lags.
Information Criteria: Methods such as Akaike Information Criterion
(AIC) or Bayesian Information Criterion (BIC).
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 212 / 271
Chapter 5: Models for Stationary Time Series The AR(p) Autoregressive Model
Example: AR(3) Model
Suppose we have a time series of monthly temperatures, and we want to
predict next month's temperature based on the previous 3 months.
The AR(3) model would look like:
yt = φ1 yt−1 + φ2 yt−2 + φ3 yt−3 + t
where:
yt is the temperature at time t,
yt−1 , yt−2 , yt−3 are the temperatures at the previous months.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 213 / 271
Chapter 5: Models for Stationary Time Series The AR(p) Autoregressive Model
Estimating Parameters of AR(p) Model
The goal of estimating the parameters φ1 , φ 2 , . . . , φ p in an Autoregressive
model of order p (AR(p)) is to nd the values that best t the observed
time series data.
Methods for estimation include:
Ordinary Least Squares (OLS)
Yule-Walker Equations
Maximum Likelihood Estimation (MLE)
Method of Moments
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 214 / 271
Chapter 5: Models for Stationary Time Series The AR(p) Autoregressive Model
1. Ordinary Least Squares (OLS) Estimation
The OLS method minimizes the sum of squared residuals to estimate the
parameters φ1 , φ 2 , . . . , φ p . The AR(p) model is:
yt = φ1 yt−1 + φ2 yt−2 + ⋯ + φp yt−p + t
The OLS estimation is done by minimizing:
n p 2
∑ (yt − ∑ φi yt−i )
t=p+ 1 1
i=
In matrix form, the estimates are:
φ̂ = (X T X )−1 X T y
where X is the matrix of lagged values and y is the vector of observations.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 215 / 271
Chapter 5: Models for Stationary Time Series The AR(p) Autoregressive Model
2. Yule-Walker Equations for AR(p)
The Yule-Walker equations can be used to estimate the parameters
φ1 , φ 2 , . . . , φ p of an autoregressive model of order p (AR(p)) using
autocovariances.
The Yule-Walker equations are given by:
p
γk = ∑ φi γk−i for k = 1 , 2, . . . , p
i= 1
where:
γk is the autocovariance at lag k,
φ1 , φ 2 , . . . , φ p are the AR coe cients.
These equations can be written in matrix form:
Γφ = γ
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 216 / 271
Chapter 5: Models for Stationary Time Series The AR(p) Autoregressive Model
Matrix Form of Yule-Walker Equations
The Yule-Walker equations for AR(p) can be written in the following
matrix form:
⎛ γ0 γ1 γ2 ⋯ γp−1 ⎞
⎜ γ1 γ0 γ1 ⋯ γp−2 ⎟ ⎛φ1 ⎞ ⎛γ1 ⎞
⎜ ⎟ ⎜φ2 ⎟ ⎜γ2 ⎟
⎜ γ2 1 γ0 ⋯ γp−3 ⎟
⎜ γ ⎟⎜ ⎟=⎜ ⎟
⎜ ⋮ ⎟ ⎜⋮⎟
⎜ ⋮ ⋮ ⋮ ⋱ ⎟
⋮ ⎟⎝ ⎠ ⎝ ⎠
⎜
⎝γp−1 γp−2 γp−3 φ γp
⋯ γ0 ⎠ p
Here:
Γ is the p×p autocovariance matrix,
φ is the vector of AR coe cients (φ1 , φ2 , . . . , φp )T ,
γ is the vector of autocovariances (γ1 , γ2 , . . . , γp )T .
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 217 / 271
Chapter 5: Models for Stationary Time Series The AR(p) Autoregressive Model
Solving for φ using Matrix Inversion
To solve for the AR coe cients φ1 , φ 2 , . . . , φ p , we invert the
autocovariance matrix Γ and multiply it by the vector γ:
φ = Γ−1 γ
This gives the solution for the AR coe cients:
− 1
⎛ γ0 γ1 γ2 ⋯ γp−1 ⎞
⎛ φ1 ⎞ ⎜ γ1 γ0 γ1 ⋯ γp−2 ⎟ ⎛ γ1 ⎞
⎜ φ2 ⎟ ⎜ ⎟ ⎜ γ2 ⎟
⎜ ⎟=⎜ γ 2 γ 1 γ0 ⋯ γp−3 ⎟ ⎜ ⎟
⎜ ⋮ ⎟ ⎜⎜ ⋮
⎟ ⎜⋮⎟
⎝φp ⎠ ⎜ ⋮ ⋮ ⋱ ⋮ ⎟⎟ ⎝γp ⎠
⎝γp−1 γp−2 γp−3 ⋯ γ0 ⎠
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 218 / 271
Chapter 5: Models for Stationary Time Series The AR(p) Autoregressive Model
Example: Solving for AR(2) Coe cients
For an AR(2) model, the Yule-Walker equations are:
( 0 1) ( 1) = ( 1)
γ γ φ γ
γ1 γ0 φ2 γ2
To solve for φ1 and φ2 , we invert the 2 × 2 autocovariance matrix Γ and
multiply by the vector γ:
− 1
( 1) = ( 0 1) ( 1)
φ γ γ γ
φ2 γ1 γ0 γ2
After computing the matrix inverse, we can nd the values of φ1 and φ2 .
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 219 / 271
Chapter 5: Models for Stationary Time Series The AR(p) Autoregressive Model
3. Maximum Likelihood Estimation (MLE)
In Maximum Likelihood Estimation (MLE), the parameters are estimated
by maximizing the likelihood function of the observed data.
The likelihood function for the AR(p) model is:
n
L(φ1 , φ2 , . . . , φp ) = ∏ f (yt ∣ yt−1 , yt−2 , . . . , yt−p ; φ1 , φ2 , . . . , φp )
t=p+ 1
The log-likelihood function is usually maximized:
n
log L(φ1 , φ2 , . . . , φp ) = ∑ log f (yt ∣ yt−1 , yt−2 , . . . , yt−p ; φ1 , φ2 , . . . , φp )
t=p+ 1
In practice, we assume that the errors t are normally distributed, so f (yt )
is the normal density function. This maximization often requires numerical
optimization techniques.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 220 / 271
Chapter 5: Models for Stationary Time Series The AR(p) Autoregressive Model
4. Method of Moments
The Method of Moments is used primarily for simple models like AR(1). In
this method, the sample moments (e.g., autocovariances) are equated to
the theoretical moments.
For an AR(1) model:
yt = φ1 yt−1 + t
where:
yt is the value of the time series at time t,
φ1 is the autoregressive coe cient,
t is the error term, assumed to be white noise with mean 0 and
variance σ2.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 221 / 271
Chapter 5: Models for Stationary Time Series The AR(p) Autoregressive Model
The autocovariance at lag 0 (the variance) is:
γ0 = Var(yt )
From the AR(1) model:
yt = φ1 yt−1 + t
Assuming that yt and t are independent, the variance of yt is:
γ0 = Var(yt ) = φ21 Var(yt−1 ) + Var(t )
Since the variance of t is σ2, we have:
γ0 = φ21 γ0 + σ 2
Rearranging this:
γ0 (1 − φ21 ) = σ 2
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 222 / 271
Chapter 5: Models for Stationary Time Series The AR(p) Autoregressive Model
So, the theoretical variance γ0 is:
σ2
γ0 = 2
1 − φ1
The autocovariance at lag 1 is:
γ1 = Cov(yt , yt−1 )
Substituting the AR(1) model:
γ1 = Cov(φ1 yt−1 + t , yt−1 )
Since t is independent of yt−1 , this simpli es to:
γ1 = φ1 Var(yt−1 ) = φ1 γ0
Thus, the theoretical autocovariance at lag 1 is:
γ1 = φ1 γ0
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 223 / 271
Chapter 5: Models for Stationary Time Series The AR(p) Autoregressive Model
The sample autocovariances are computed directly from the observed data.
The sample autocovariance at lag 1 is:
n
1
γ̂1 = ∑ (yt − ȳ )(yt−1 − ȳ )
n t=2
The sample variance at lag 0 is:
n
γ̂0 =
1
∑ (yt − ȳ )
2
n t=1
According to the Method of Moments, we equate the sample
autocovariances with the theoretical values:
γ̂1 = φ1 γ̂0
Solving for φ1 , we get:
γ̂1
φ̂1 =
γ̂0
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 224 / 271
Chapter 5: Models for Stationary Time Series The AR(p) Autoregressive Model
Therefore, the estimate for φ1 is:
φ̂1 =
Cov(yt , yt− 1)
Var(yt− 1)
For higher-order AR models, the method becomes impractical, and
methods like OLS or Yule-Walker are preferred.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 225 / 271
Chapter 5: Models for Stationary Time Series The AR(p) Autoregressive Model
Summary of Estimation Methods
Here's a summary of the methods for estimating the parameters of an
AR(p) model:
OLS Estimation: Minimize the sum of squared residuals.
Yule-Walker Equations: Solve the system of equations based on
autocovariances.
Maximum Likelihood Estimation (MLE): Maximize the likelihood
function based on normal errors.
Method of Moments: Equate sample moments with theoretical
moments (mainly for AR(1)).
The choice of method depends on the model complexity, the available
data, and the computational resources.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 226 / 271
Chapter 5: Models for Stationary Time Series Autocovariance Function for AR(p)
Autocovariance Function for AR(p)
An Autoregressive process of order p (AR(p)) is given by:
yt = φ1 yt−1 + φ2 yt−2 + ⋅ ⋅ ⋅ + φp yt−p + t
where:
yt is the value of the time series at time t,
φ1 , φ 2 , . . . , φ p are the autoregressive coe cients,
t is a white noise error term with mean 0 and variance σ2.
The autocovariance function γk = Cov(yt , yt−k ) measures the covariance
between yt and yt−k .
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 227 / 271
Chapter 5: Models for Stationary Time Series Autocovariance Function for AR(p)
Autocovariance at Lag 0
For lag 0, the autocovariance is the variance of yt :
γ0 = Var(yt )
Substituting the AR(p) model into the equation for yt :
yt = φ1 yt−1 + φ2 yt−2 + ⋅ ⋅ ⋅ + φp yt−p + t
Taking the variance of both sides:
Var(yt ) = Var(φ1 yt−1 + φ2 yt−2 + ⋅ ⋅ ⋅ + φp yt−p + t )
Since t is white noise with zero mean and variance σ2, the variance
becomes:
Var(yt ) = φ21 Var(yt−1 ) + φ22 Var(yt−2 ) + ⋅ ⋅ ⋅ + φ2p Var(yt−p ) + σ 2
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 228 / 271
Chapter 5: Models for Stationary Time Series Autocovariance Function for AR(p)
Assuming stationarity, Var(yt ) = Var(yt−i ), we get:
γ0 = φ21 γ0 + φ22 γ0 + ⋅ ⋅ ⋅ + φ2p γ0 + σ 2
Factor out γ0 :
p
γ0 (1 − ∑ φ2i ) = σ 2
i= 1
Solve for γ0 :
σ2
γ0 = p 2
1−∑
i=1 φi
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 229 / 271
Chapter 5: Models for Stationary Time Series Autocovariance Function for AR(p)
Autocovariance at Lag 1
Now, let's calculate the autocovariance at lag 1:
γ1 = Cov(yt , yt−1 )
Substituting the AR(p) model for yt and yt−1 :
yt = φ1 yt−1 + φ2 yt−2 + ⋅ ⋅ ⋅ + φp yt−p + t
yt−1 = φ1 yt−2 + φ2 yt−3 + ⋅ ⋅ ⋅ + φp yt−p−1 + t−1
The covariance between yt and yt−1 is:
γ1 = Cov(φ1 yt−1 + φ2 yt−2 + ⋅ ⋅ ⋅ + φp yt−p + t , yt−1 )
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 230 / 271
Chapter 5: Models for Stationary Time Series Autocovariance Function for AR(p)
Using the properties of covariance, and assuming t is independent of yt−1 ,
we have:
γ1 = φ1 Var(yt−1 ) + φ2 Cov(yt−1 , yt−2 ) + ⋅ ⋅ ⋅ + φp Cov(yt−1 , yt−p )
From stationarity:
Var(yt− 1 ) = γ0 , Cov(yt− 1 , yt−2 ) = γ1 , and so on.
Therefore, the recursive equation for γ1 is:
γ1 = φ1 γ0 + φ2 γ1 + φ3 γ2 + ⋅ ⋅ ⋅ + φp γp−1
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 231 / 271
Chapter 5: Models for Stationary Time Series Autocovariance Function for AR(p)
General Autocovariance at Lag k
For the general autocovariance at lag k (where k ≥ 1):
γk = Cov(yt , yt−k )
This can be written as:
p
γk = ∑ φi γk−i
i= 1
This is a recursive relationship for computing autocovariances at higher
lags. Once γ0 is known, the other autocovariances can be computed using
this recurrence.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 232 / 271
Chapter 5: Models for Stationary Time Series Autocorrelation Function
Autocorrelation Function
The autocorrelation function ρk is the normalized version of the
autocovariance function:
γk
ρk =
γ0
For lag k, we have:
p
∑ φi γk−i
ρk = i=1
γ0
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 233 / 271
Chapter 5: Models for Stationary Time Series Autocorrelation Function
Special Case: AR(1)
For the special case of an AR(1) model:
yt = φ1 yt−1 + t
The autocovariance at lag 0 is:
σ2
γ0 = 2
1 − φ1
The autocovariance at lag 1 is:
γ1 = φ1 γ0
For k ≥ 2:
γk = φk1 γ0
Therefore, the autocorrelation function for the AR(1) model is:
ρk = φk1
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 234 / 271
Chapter 5: Models for Stationary Time Series Autocorrelation Function
Since ∣φ∣ < 1, the magnitude of autocorrelation function decreases
exponentially as the number of lags, k, increases.
If 0 < φ < 1, all correlations are positive
If −1 < φ < 0, the lag 1 autocorrelation is negative (φ 1 < 0) and the
signs of successive autocorrelations alternate from positive to
negative, with their magnitudes decreasing exponentially.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 235 / 271
Chapter 5: Models for Stationary Time Series Autocorrelation Function
Autocorrelation Functions for Several AR(1) Models
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 236 / 271
Chapter 5: Models for Stationary Time Series Autocorrelation Function
Simulate AR(1) Time Series in Python
import numpy as np
from matplotlib import pyplot as plt
from statsmodels.tsa.arima_process import ArmaProcess
# Plot 1: AR parameter = +0.6
plt.subplot(2, 1, 1)
ar1 = np.array([1, -0.6])
ma1 = np.array([1])
AR_object1 = ArmaProcess(ar1, ma1)
simulated_data_1 = AR_object1.generate_sample(nsample=100)
plt.plot(simulated_data_1);
# Plot 2: AR parameter = -0.6
plt.subplot(2, 1, 2)
ar2 = np.array([1, 0.6])
ma2 = np.array([1])
AR_object2 = ArmaProcess(ar2, ma2)
simulated_data_2 = AR_object2.generate_sample(nsample=100)
plt.plot(simulated_data_2);
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 237 / 271
Chapter 5: Models for Stationary Time Series Autocorrelation Function
Compare the ACF for Several AR Time Series
from statsmodels.graphics.tsaplots import plot_acf
# Plot 1: AR parameter = +0.6
plot_acf(simulated_data_1, alpha=1, lags=20);
# Plot 2: AR parameter = -0.6
plot_acf(simulated_data_2, alpha=1, lags=20);
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 238 / 271
Chapter 5: Models for Stationary Time Series Autocorrelation Function
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 239 / 271
Chapter 5: Models for Stationary Time Series Autocorrelation Function
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 240 / 271
Chapter 5: Models for Stationary Time Series Autocorrelation Function
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 241 / 271
Chapter 5: Models for Stationary Time Series The MA(q) Moving average Model
The MA(q) Moving average Model
A Moving Average (MA) model of order q, denoted as MA(q ), is a
time series model where the current value of the series is expressed as
a linear combination of past error terms.
The general form of the MA(q ) model is:
yt = µ + t + θ1 t−1 + θ2 t−2 + ⋯ + θq t−q
Where:
▸ yt is the observed value at time t ,
▸ µ is the mean of the time series (often zero),
▸ t is the error term at time t ,
▸ θ1 , θ2 , . . . , θq are the parameters (coe cients) of the model,
▸ q is the order of the moving average (number of past error terms).
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 242 / 271
Chapter 5: Models for Stationary Time Series The MA(q) Moving average Model
Key Features of the MA(q) Model
Stationarity: MA models are typically applied to stationary time series
data (constant mean and variance over time).
Short Memory: The MA model only depends on a xed number of
past error terms (up to lag q ).
White Noise Assumption: The error terms t are assumed to be i.i.d.
with mean zero and constant variance.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 243 / 271
Chapter 5: Models for Stationary Time Series The MA(q) Moving average Model
Example of MA(2) Model
Consider an MA(2) model, where the value at time t depends on the
current error term and the two previous error terms:
yt = µ + t + θ1 t−1 + θ2 t−2
Here, the current value yt is in uenced by the:
Current error term t ,
First lagged error term t−1 ,
Second lagged error term t−2 .
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 244 / 271
Chapter 5: Models for Stationary Time Series The MA(q) Moving average Model
Use in Time Series Forecasting
The MA(q) model is used for time series data where:
Short-term dependencies or noise patterns are observed,
Forecasting future values based on past errors is desired.
The MA model is often combined with AutoRegressive (AR) models to
form the ARMA(p, q) model, where p represents the order of the AR part,
and q represents the order of the MA part.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 245 / 271
Chapter 5: Models for Stationary Time Series The MA(q) Moving average Model
Simulating a moving average model
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 246 / 271
Chapter 5: Models for Stationary Time Series The MA(q) Moving average Model
Autocovarance and Autocorrelation Function of MA(1)
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 247 / 271
Chapter 5: Models for Stationary Time Series The MA(q) Moving average Model
Autocorrelation Function of MA(1)
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 248 / 271
Chapter 5: Models for Stationary Time Series The MA(q) Moving average Model
Autocorrelation Function of MA(1)
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 249 / 271
Chapter 5: Models for Stationary Time Series The MA(q) Moving average Model
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 250 / 271
Chapter 5: Models for Stationary Time Series The MA(q) Moving average Model
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 251 / 271
Chapter 5: Models for Stationary Time Series The MA(q) Moving average Model
Usually, ACF de nes error serial correlation well and is thus used to
detect MA(q); however, as the order increases and other time series
components such as seasonality,trend, or stationary come into picture,
this makes interpretation harder
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 252 / 271
Chapter 5: Models for Stationary Time Series The ARMA(p, q) Model
The ARMA(p, q) Model
The ARMA(p, q) model combines two key components:
AutoRegressive (AR) part: Models the dependence of the current
value on its past values.
Moving Average (MA) part: Models the dependence of the current
value on past error terms.
It is commonly used in time series forecasting when data is stationary. The
general form of the ARMA(p, q) model is:
yt = µ + φ1 yt−1 + φ2 yt−2 + ⋯ + φp yt−p + t + θ1 t−1 + θ2 t−2 + ⋯ + θq t−q
The error terms t are generally assumed to be independent, identically
distributed variables sampled from a normal distribution with zero mean.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 253 / 271
Chapter 5: Models for Stationary Time Series The ARMA(p, q) Model
Where:
yt is the observed value at time t,
µ is the mean of the series (often zero),
t is the error term at time t,
φ1 , φ 2 , . . . , φ p are the parameters of the AR part,
θ1 , θ2 , . . . , θ q are the parameters of the MA part,
p is the order of the AR part,
q is the order of the MA part.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 254 / 271
Chapter 5: Models for Stationary Time Series The ARMA(p, q) Model
Stationarity and White Noise
For an ARMA model to be valid, the following conditions should hold:
Stationarity: The time series must be stationary, meaning its mean,
variance, and autocovariance do not change over time.
White Noise: The error terms t must be uncorrelated, with mean
zero and constant variance (i.i.d.).
The model is typically tted to stationary data for reliable forecasting.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 255 / 271
Chapter 5: Models for Stationary Time Series The ARMA(p, q) Model
Example of ARMA(1, 1) Model
An ARMA(1, 1) model has the following form:
yt = µ + φ1 yt−1 + t + θ1 t−1
In this case, the current value yt depends on:
The previous value yt−1 (AR part),
The current error term t ,
The previous error term t−1 (MA part).
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 256 / 271
Chapter 5: Models for Stationary Time Series The ARMA(p, q) Model
Model Estimation and Use
To estimate the parameters φi and θi in the ARMA(p, q) model:
Use methods like Maximum Likelihood Estimation (MLE) or Least
Squares.
Autocorrelation Function (ACF) and Partial Autocorrelation Function
(PACF) plots are helpful for determining the appropriate values of p
and q.
ARMA models are widely used for time series forecasting, especially in
nancial markets, economics, and signal processing.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 257 / 271
Chapter 5: Models for Stationary Time Series The Autocorrelation Function of ARMA(1)
Autocovariance and Autocorrelation Function of ARMA(1)
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 258 / 271
Chapter 5: Models for Stationary Time Series The Autocorrelation Function of ARMA(1)
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 259 / 271
Chapter 5: Models for Stationary Time Series The Autocorrelation Function of ARMA(1)
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 260 / 271
Chapter 5: Models for Stationary Time Series The ARMA(p, q) Model
The ARMA(p, q) Model
In an ARMA(p, q) model:
p is the order of the autoregressive (AR) part.
q is the order of the moving average (MA) part.
The challenge is to determine the optimal values of p and q. We can use
several methods:
ACF and PACF plots.
Information criteria (AIC, BIC).
Cross-validation.
Model diagnostics.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 261 / 271
Chapter 5: Models for Stationary Time Series The ARMA(p, q) Model
ACF and PACF Plots
The Autocorrelation Function (ACF) and Partial Autocorrelation
Function (PACF) plots are key tools to determine p and q.
▸ ACF helps determine the order of the MA part (q ).
▸ PACF helps determine the order of the AR part (p ).
ACF characteristics for MA(q):
▸ The ACF should cut o (i.e., drop to zero) after lag q .
PACF characteristics for AR(p):
▸ The PACF should cut o after lag p .
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 262 / 271
Chapter 5: Models for Stationary Time Series The ARMA(p, q) Model
Example of ACF and PACF Interpretation
Suppose you have the following ACF and PACF plots:
The ACF cuts o after lag 2. Therefore, we choose q = 2.
The PACF cuts o after lag 1. Therefore, we choose p = 1.
This suggests an ARMA(1, 2) model.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 263 / 271
Chapter 5: Models for Stationary Time Series The ARMA(p, q) Model
AIC and BIC Information Criteria
Akaike Information Criterion (AIC) and Bayesian Information Criterion
(BIC) help to compare models with dierent p and q values.
AIC:
AIC = 2k − 2 ln(L)
where k is the number of parameters and L is the likelihood.
BIC:
BIC = ln(n)k − 2 ln(L)
where n is the number of observations.
The model with the lowest AIC or BIC is preferred.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 264 / 271
Chapter 5: Models for Stationary Time Series The ARMA(p, q) Model
Cross-Validation
Cross-validation involves splitting the data into training and test sets to
evaluate the model's forecasting performance.
Fit ARMA models with dierent p and q to the training set.
Assess the forecasting accuracy on the test set (e.g., using Mean
Squared Error).
Select the model with the best performance.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 265 / 271
Chapter 5: Models for Stationary Time Series The ARMA(p, q) Model
Model Diagnostics
After selecting p and q, check the residuals to ensure the model ts well:
ACF of residuals: The residuals should resemble white noise, meaning
there should be no signi cant autocorrelation.
Ljung-Box Test: This test checks whether the residuals are
independent. If the test suggests signi cant autocorrelation, consider
adjusting p and q.
The residuals should be approximately normally distributed and
uncorrelated.
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 266 / 271
Chapter 5: Models for Stationary Time Series The ARMA(p, q) Model
Python Code for Cross-Validation for Time-Series
from sklearn.model_selection import TimeSeriesSplit
from sklearn.metrics import mean_squared_error
import statsmodels.api as sm
tscv = TimeSeriesSplit(n_splits = 4)
rmse = []
for train_index, test_index in tscv.split(cross_validation):
cv_train, cv_test = cross_validation.iloc[train_index],
cross_validation.iloc[test_index]
arma = sm.tsa.ARMA(cv_train, (2,2)).fit(disp=False)
predictions = arma.predict(cv_test.index.values[0],
cv_test.index.values[-1])
true_values = cv_test.values
rmse.append(sqrt(mean_squared_error(true_values, predictions)))
print("RMSE: {}".format(np.mean(rmse)))
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 267 / 271
Chapter 5: Models for Stationary Time Series The ARMA(p, q) Model
Python Code for ACF and PACF
# Load modules
from __future__ import print_functionimport os
import pandas as pdimport numpy as np
from matplotlib import pyplot as plt
# Load Dataset
ibm_df =
pd.read_csv('datasets/ibm-common-stock-closing-prices.csv')ibm_d
#Rename the second column
ibm_df.rename(columns={'IBM common stock closing prices':
'Close_Price'},inplace=True)ibm_df.head()
from statsmodels.graphics.tsaplots import plot_acf,
plot_pacfibm_df['residual']=ibm_df['Close_Price']-ibm_df['Close_
plot_acf(ibm_df.residual, lags=50)plot_pacf(ibm_df.residual,
lags=50)
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 268 / 271
Chapter 5: Models for Stationary Time Series The ARMA(p, q) Model
Python Code for AIC
# Optimize ARMA parametersaicVal=[]
for ari in range(1, 3):
for maj in range(0,3):
arma_obj = smtsa.ARMA(ibm_df.Close_Price.tolist(),
order=(ari,maj)).fit(maxlag=30, method='mle',
trend='nc')aicVal.append([ari, maj, arma_obj.aic])
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 269 / 271
Chapter 5: Models for Stationary Time Series The ARMA(p, q) Model
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 270 / 271
Chapter 5: Models for Stationary Time Series The ARMA(p, q) Model
Dr. Md. Rezaul Karim (Professor, Dept. of Statistics and Data Science, JU) MS in Data Science Fall - 2024 271 / 271
You might also like
- Generalised Linear Models and Bayesian StatisticsNo ratings yetGeneralised Linear Models and Bayesian Statistics35 pages
- Wiley - Student Solutions Manual To Accompany Introduction To Time Series Analysis and Forecasting - 978-0-470-43574-80% (1)Wiley - Student Solutions Manual To Accompany Introduction To Time Series Analysis and Forecasting - 978-0-470-43574-83 pages
- Time Series Non-Linear ARDL Model/ Asymmetric ARDL Cointegration by MEO School of ResearchNo ratings yetTime Series Non-Linear ARDL Model/ Asymmetric ARDL Cointegration by MEO School of Research24 pages
- Open Mapping Theorem (Complex Analysis) PDFNo ratings yetOpen Mapping Theorem (Complex Analysis) PDF2 pages
- (MADHU MANGAL PAUL) Numerical Analysis For Scienti100% (1)(MADHU MANGAL PAUL) Numerical Analysis For Scienti666 pages
- Advanced Statistical Inference Lecture NotesNo ratings yetAdvanced Statistical Inference Lecture Notes22 pages
- A Koutsoyiannis - Theory of Econometrics - An Introductory Exposition of Econometric Methods (1977, Macmillan) - Libgen - Li (1) - 1-35 - CompressedNo ratings yetA Koutsoyiannis - Theory of Econometrics - An Introductory Exposition of Econometric Methods (1977, Macmillan) - Libgen - Li (1) - 1-35 - Compressed35 pages
- Book - Introduction To Mathematical Economics - Walter BossertNo ratings yetBook - Introduction To Mathematical Economics - Walter Bossert169 pages
- 2918Instant download Matrix Differential Calculus with Applications in Statistics and Econometrics 3rd Edition Jan R. Magnus pdf all chapter100% (12)2918Instant download Matrix Differential Calculus with Applications in Statistics and Econometrics 3rd Edition Jan R. Magnus pdf all chapter67 pages
- 202004160626023624rajiv Saksena Advance Statistical InferenceNo ratings yet202004160626023624rajiv Saksena Advance Statistical Inference31 pages
- Oksendal Stochastic Differential Equations PDF FreeNo ratings yetOksendal Stochastic Differential Equations PDF Free385 pages
- (Ian W. Knowles, Roger T. Lewis, International Con (BookFi)No ratings yet(Ian W. Knowles, Roger T. Lewis, International Con (BookFi)401 pages
- Algebra of Matrices & Determinants: Complete Study Guide & Notes OnNo ratings yetAlgebra of Matrices & Determinants: Complete Study Guide & Notes On8 pages
- Panel Data Methods For Microeconometrics Using Stata: A. Colin Cameron Univ. of California - Davis100% (1)Panel Data Methods For Microeconometrics Using Stata: A. Colin Cameron Univ. of California - Davis55 pages
- 3.2 - Interpolation and Lagrange Polynomials 1. Polynomial Interpolation: Problem: Given NNo ratings yet3.2 - Interpolation and Lagrange Polynomials 1. Polynomial Interpolation: Problem: Given N13 pages
- MATH 223: Calculus II: Dr. Joseph K. AnsongNo ratings yetMATH 223: Calculus II: Dr. Joseph K. Ansong28 pages
- Quadratic Forms & Applications To Geometry - Reducing Quadric To Canonical Standard Form - Schulstad 1967No ratings yetQuadratic Forms & Applications To Geometry - Reducing Quadric To Canonical Standard Form - Schulstad 196734 pages
- Structural VAR and Applications: Jean-Paul RenneNo ratings yetStructural VAR and Applications: Jean-Paul Renne55 pages
- Instant Access to Real Analysis and Foundations 2nd Edition Steven G. Krantz ebook Full Chapters100% (1)Instant Access to Real Analysis and Foundations 2nd Edition Steven G. Krantz ebook Full Chapters76 pages
- Measure Theory and Lebesgue Integration: Appendix DNo ratings yetMeasure Theory and Lebesgue Integration: Appendix D14 pages
- (Mathematics and Its Applications) Malempati M. Rao, Randall J. Swift - Probability Theory With Applications - Springer (2006)No ratings yet(Mathematics and Its Applications) Malempati M. Rao, Randall J. Swift - Probability Theory With Applications - Springer (2006)536 pages
- Where can buy Logistic Regression Using SAS Theory and Application Second Edition Paul D. Allison ebook with cheap price100% (1)Where can buy Logistic Regression Using SAS Theory and Application Second Edition Paul D. Allison ebook with cheap price61 pages
- Unitary and Orthogonal Operators and Their Matrices-Pt2100% (1)Unitary and Orthogonal Operators and Their Matrices-Pt222 pages
- Vector Autoregressions: How To Choose The Order of A VARNo ratings yetVector Autoregressions: How To Choose The Order of A VAR8 pages
- Time Series Analysis and Spectral AnalysisNo ratings yetTime Series Analysis and Spectral Analysis11 pages
- 01 ASAP TimeSeriesForcasting Day1 2 IntroductionNo ratings yet01 ASAP TimeSeriesForcasting Day1 2 Introduction62 pages
- Time Series Chapman Hall CRC Texts in Statistical Science 1st Edition Robert Shumway downloadNo ratings yetTime Series Chapman Hall CRC Texts in Statistical Science 1st Edition Robert Shumway download79 pages
- Tle10 He Cookery q4 Mod4 Storingmeat v4No ratings yetTle10 He Cookery q4 Mod4 Storingmeat v418 pages
- Aluminum 5182-H34: Metal Nonferrous Metal Aluminum Alloy 5000 Series Aluminum AlloyNo ratings yetAluminum 5182-H34: Metal Nonferrous Metal Aluminum Alloy 5000 Series Aluminum Alloy2 pages
- Overcoming Shoulder Impingement Syndrome, 2003100% (1)Overcoming Shoulder Impingement Syndrome, 200316 pages
- S.4 Chem Revision Questions 2020 Revision & Past PapersNo ratings yetS.4 Chem Revision Questions 2020 Revision & Past Papers5 pages
- Work Approval (PTW) : Initiator'S ChecklistNo ratings yetWork Approval (PTW) : Initiator'S Checklist2 pages
- (eBook PDF) The Sciences: An Integrated Approach, 8th Editioninstant download100% (5)(eBook PDF) The Sciences: An Integrated Approach, 8th Editioninstant download52 pages
- Grade - 4 - Social Studies - End of Term 3 - 23-24No ratings yetGrade - 4 - Social Studies - End of Term 3 - 23-247 pages
- Machine Learning Approach For Flood Risks PredictionNo ratings yetMachine Learning Approach For Flood Risks Prediction8 pages
- Project: Dog Legged Stair Design Constant: OK Under ReinforcedNo ratings yetProject: Dog Legged Stair Design Constant: OK Under Reinforced4 pages
- Simulation of The Aeronautical Radio Channel For Telemetry ApplicationsNo ratings yetSimulation of The Aeronautical Radio Channel For Telemetry Applications10 pages
