FEM 2063 - Data Analytics

CHAPTER 2
AT THE END OF THIS CHAPTER STUDENTS SHOULD BE
ABLE TO UNDERSTAND

SIMPLE LINEAR REGRESSION

1
OVERVIEW
➢2.1 Background
➢2.2 Correlation
➢2.3 Regression
➢2.4 Least Squares Method
➢2.5 Simple Linear Regression
(SLR)
➢2.6 ANOVA
➢2.7 Model Evaluation
➢2.8 Applications/Examples

2
 2.1 BACKGROUND
WHAT IS LINEAR REGRESSION (LR)?
Linear regression is a linear model, a model that
assumes a linear relationship between the input variables
(x) and the single output variable (y)
LR is relation between variables where changes in some
variables may “explain” the changes in other variables
LR model estimates the nature of the relationship between
independent and dependent variables.

Examples:
• Does change class size affect marks of students?
• Does cholesterol level depend on age, sex, or amount of
exercise?
3
2.1 What is Linear Regression (LR)?
Investigating the dependence of one variable (dependent
variable), on one or more variables (independent variable)
using a straight line.

Strong relationships Weak relationships

Y Y

X X

Y Y

4
X X
2.1 What is LR model used for?
Linear regression models are used to show or predict
the relationship between two variables or factors. The
factor that is being predicted is called the dependent
variable.
Example of Linear Regression
Regression analysis is used in stats to find trends in data.
For example, you might guess that there's a connection
between how much you eat and how much you
weight; regression analysis can help you quantify that.

5
2.1 LR model - how does it work?
Linear Regression is the process of finding a line that best
fits the data points available on the plot, so that we can use
it to predict output values for inputs that are not present
in the data set we have, with the belief that those outputs
would fall on the line.

6
 2.1 What Regression is Used for?
1.Predictive Analytics: 2. Operation Efficiency:
forecasting future opportunities Regression models can also
and risks is the most prominent be used to optimize
application of regression analysis business processes.
in business
4. Correcting Errors:
3. Supporting Decisions: Regression is not only great for
Businesses today are overloaded lending empirical support to
with data on finances, operations management decisions but also for
and customer purchases. identifying errors in judgment.
Executives are now leaning on
data analytics to make informed 5. New Insights:
business decisions that have Over time businesses have
statistical significance, thus gathered a large volume of
eliminating the intuition and gut unorganized data that has the
feel. potential to yield valuable insights.

7
2.1 Example of LR in Forecasting

© 2019 Petroliam Nasional Berhad (PETRONAS) | 8

 2.1 Types of Regression

9
2.1 Types of Regression

10
 2.1 Types of Regression

(EDUCATION) Y (Income)

(EDUCATION)
(SEX)
(EXPERIENCE) Y (Income)
(AGE)

11
 2.2 Correlation
 Correlation is a statistical technique that can show whether
and how strongly pairs of variables are related.
 The range of possible values is from -1 to +1
 The correlation is high if observations lie close to a straight
line (ie values close to +1 or -1) and low if observations are
widely scattered (correlation value close to 0)

( xi − x )( yi − y )
r=
(( xi − x ) 2 )(( yi − y ) 2 )

12
 2.2 Correlation

13
2.2 Correlation Vs Linear Regression

14
 2.3 Simple Linear Regression
• The quantitative analysis use the information to predict its
future behavior.
• Current information is usually in the form of a set of data.
• When the data form a set of pairs of numbers, we may
interpret them as representing the observed values of an
independent (predictor) variable x and a dependent
(response) variable y.
• The goal is to find a functional relation between the response
variable y and the predictor variable x,
𝑦 = 𝑓(𝑥)
SELECTION of independent variable(s)
- choose the most important predictor variable(s).
SCOPE of model
- we may need to restrict the coverage of model to some interval
15
or
region of values of the independent variable(s) depend on the
needs/requirements.
2.3 Regression - Population & Sample

16
 2.3 Regression - Regression Model
General regression model:
𝑌 = 𝛽0 + 𝛽1 𝑋 + 𝜀
0, and 1 are unknown parameters, x is a known parameter
Deviations  are independent, n(o, 2)
The values of the regression parameters 0, and 1 are
not known. We estimate them from data.

1 indicates the change in the mean response per unit

increase in X.

17
 2.3 Regression - Regression Line
• If the scatter plot of the sample data suggests a linear
relationship between two variables i.e.
𝑦 = 𝛽መ0 + 𝛽መ1 𝑥
the relationship can be summarized by a straight-line plot.
• Least squares method give the “best” estimated line for
our set of sample data.
• The least squares method is a statistical procedure to find
the best fit for a set of data points by minimizing the sum
of the offsets or residuals of points from the plotted curve.
Least squares method is used to predict the behavior of
dependent variables.
18
 2.4 Least Squares Method
• Line of ‘best fit’ means differences between actual y values &
predicted y values is at minimum, but positive differences
offset negative ones, so, square the errors!

LS methods minimizes the Sum of the Squared

Differences (errors) (SSE)

19
2.4 Least Squares Method

20
 2.4 Assumptions in SLR
◼ Linear relationship: The relationship between X and the
mean of Y is linear
◼ Same variance: The errors should have the same variance.
◼ Independent observations: Observations are independent
of each other.
◼ Normal distribution of error terms: The residuals 𝜀𝑖 are
normally distributed

21
 2.5 ANOVA
• ANOVA (analysis of variance) is the term for statistical
analyses of the different sources of variation.
• Partitioning of sums of squares and degrees of freedom
associated with the response variable.
2.5 ANOVA Table
2.5 ANOVA – SST, SSE & SSR

24
2.5 ANOVA – SST, SSE & SSR

© 2019 Petroliam Nasional Berhad (PETRONAS) | 25

 2.5 ANOVA – SST, SSE & SSR
The measure of total variation, denoted by SST, is the sum of
the squared deviations:
lj 2
𝑆𝑆𝑇 = ෍(𝑦𝑖 − 𝑦)

If SST = 0, all observations are the same (no variability).

The greater is SST, the greater is the variation among the y
values.
In regression model, the measure of variation is that of the y
observations variability around the fitted line:

𝑦𝑖 − 𝑦ො𝑖
 2.5 ANOVA – SST, SSE & SSR
▪ Sum Square Total (SST ):
- Measure how much variance is in the dependent variable.
- Made up of the SSE and SSR
𝐧 𝐧 𝐧

lj 𝟐 = ෍(𝐲𝐢 − 𝐲ො𝐢 )𝟐 + ෍( 𝐲ො𝐢 − 𝐲)

𝐒𝐒𝐓 = ෍(𝐲𝐢 − 𝐲) lj 𝟐
𝐢=𝟏 𝐢=𝟏 𝐢=𝟏

SST = SSE + SSR

degree of
𝑛−1 = 𝑛−2 + 1
freedom:
27
2.5 ANOVA – SST, SSE & SSR

28
2.6 Model Evaluation
SLR model evaluation is using software output

(i) standard error of estimate (s)

(ii) coefficient of determination (R2)
(iii) hypothesis test
a)the t-test of the slope
b)the F-test of the slope

29
2.6 Model Evaluation
(i) Standard error of estimate (s)
➢ Compute Standard Error of Estimate
𝐧

𝐒𝐒𝐄 = ෍(𝐲𝐢 − 𝑦ො𝐢 )𝟐

𝐢=𝟏

➢ The smaller SSE the more successful is the Linear

Regression Model in explaining y.

30
2.6 Model Evaluation
(ii) Coefficient of Determination
➢ Coefficient of determination
𝑆𝑆𝑅 𝑆𝑆𝐸
𝑅2 = =1− R2 = 1 - (SSE/SST)
𝑆𝑆𝑇 𝑆𝑆𝑇

➢proportion of variability in the observed dependent variable

that is explained by the linear regression model.
➢The coefficient of determination measures the strength of
that linear relationship, denoted by R2
➢The greater R2 the more successful is the Linear Model
➢The R2 value close to 1 indicates the best fit and close to 0
indicates the poor fit.

31
2.6 R-SQUARED

© 2019 Petroliam Nasional Berhad (PETRONAS) | 32

 2.6 Model Evaluation
(iii) The Hypothesis Test
Hypothesis testing: Decision-making procedure about
the null hypothesis

◼The Null Hypothesis (H0):

The hypothesis that cannot be viewed as false unless
sufficient evidence on the contrary is obtained.

◼The Alternative Hypothesis (H1):

The hypothesis against which the null hypothesis is
tested and is viewed true when H0 is declared false.

© 2019 Petroliam Nasional Berhad (PETRONAS) | 33

 2.6 Model Evaluation
(iii) The Hypothesis Test

Hypothesis test
• A process that uses sample statistics to test a claim about the value
of a population parameter.
• Example: An automobile manufacturer advertises that its new hybrid
car has a mean mileage of 50 miles per gallon. To test this claim, a
sample would be taken. If the sample mean differs enough from the
© 2019 Petroliam Nasional Berhad (PETRONAS) | 34
advertised mean, you can decide the advertisement is wrong.
 2.6 Model Evaluation
(iii) The Hypothesis Test
• One sided (tailed) H 0 :   0 or  = 0
lower-tail test H1 :    0

• One sided (tailed) H 0 :   0 or  = 0

upper-tail test H1 :    0

H 0 :  = 0
• Two-sided (tailed) test H1 :    0

Note: μ0 is the value given/assumed for the parameter μ.

© 2019 Petroliam Nasional Berhad (PETRONAS) | 35
 2.6 Model Evaluation
(iii) The Hypothesis Test

One sided
(tailed) upper-
tail test

One sided
(tailed) lower-
tail test

Two-sided
(tailed) test

© 2019 Petroliam Nasional Berhad (PETRONAS) | 36

 2.6 Model Evaluation
(iii) The Hypothesis Test
➢Equivalence of F-Test and t -Test:
For Given  Level, the F-Test of 1 = 0 Versus 1  0 is
Equivalent algebraically to the Two-sided t-test.

➢ Thus, at a given level, we can use Either the t-test or the F-

test for testing 1 = 0 versus 1  0.

➢ The t-test is More Flexible since it can be used for one sided
test as well.
2.6 Model Evaluation
(iii) The hypothesis test
– t-test
➢ t-test is used to check on adequate
relationship between x and y

➢ Test the hypothesis

H0 : 𝛽1 = 0 (No relationship between x and y)
H1: 𝛽1 ≠ 0 (There is relationship between x and y)
𝛽መ1 − 𝛽1 𝛽መ1 − 𝛽1
➢ Test Statistic: T – distribution: 𝑇= =
𝜎ො 2 𝑠𝑒(𝑏)
𝑠𝑠𝑥𝑥
➢ Critical Region: |T | > tα/2, n-2 .

38
2.6 Model Evaluation
(iii) The hypothesis test
– F-test
• In order to be able to construct a
statistical decision rule, we need to know
the distribution of our test statistic F.
𝑀𝑆𝑅
𝐹=
𝑀𝑆𝐸
• when h0 is true, our test statistic, F,
follows the F-distribution with 1, and n-2
degrees of freedom.
𝐹(𝛼; 1, 𝑛 − 2)
2.6 Model Evaluation
(iii) The hypothesis test – F-test
• This time we will use the F-test, the null and alternative
hypothesis are:
𝐻0 : 𝛽1 = 0
𝐻𝑎 : 𝛽1 ≠ 0
Construction of decision rule:
At  = 5% level, Reject H0 if 𝐹 > 𝐹(𝛼; 1, 𝑛 − 2)

Large values of F support Ha and values of F near 1

support H0.
 2.6 Model Evaluation - (iv) P - Value
In statistics, the p-value is the probability of obtaining
results at least as extreme as the observed results of a
statistical hypothesis test, assuming that the null hypothesis
is correct.
A smaller p-value means
that there is stronger
evidence in favor of the
alternative hypothesis.
When the p–value is
small we shall reject the
null hypothesis H0 .

© 2019 Petroliam Nasional Berhad (PETRONAS) | 41

Excel steps and outputs
 2.7 Example 1
The manager of a car plant wishes to investigate how the
plant’s electricity usage depends upon the plant production.
The data is given below estimate the linear regression
equation
Production (x) ($M) 4.51 3.58 4.31 5.06 5.64 4.99 5.29 5.83 4.7 5.61 4.9 4.2

Electricity Usage 2.48 2.26 2.47 2.77 2.99 3.05 3.18 3.46 3.03 3.26 2.67 2.5
(y)(kWh) 3

i. Estimate the linear regression equation

ii. Find the standard error of estimate of this regression.
iii. Determine the coefficient of determination of this regression.
iv. Test for significance of regression at 5% significance level.

43
 Set up the hypothesis
H0 : β = 0 ( there is no relationship between x and y)
There is no relationship between Production and
Electric Usage

H1: β ≠ 0 (the straight-line model is adequate)

There is relationship between Production and
Electric Usage

44
 Excel Results
Regression Statistics
Multiple R 0.895605603
R Square 0.802109396
Adjusted R Square 0.782320336
Standard Error 0.172947969
Observations 12

ANOVA
df SS MS F Significance F
Regression 1 1.212381668 1.21238 40.53297031 8.1759E-05
Residual 10 0.299109998 0.02991
Total 11 1.511491667

Coefficients Standard Error t Stat P-value Lower 95% Upper 95% Lower 95.0% Upper 95.0%
Intercept 0.409048191 0.385990515 1.05974 0.314189743 -0.450992271 1.269088653 -0.45099227 1.269088653
X Variable 1 0.498830121 0.078351706 6.36655 8.1759E-05 0.324251642 0.673408601 0.32425164 0.673408601
 Excel Results : Regression Line
Production Line Fit Plot
4
y = 0.4988x + 0.409
3.5 R² = 0.8021

2.5
electricity

2 electricity
Predicted electricity
1.5 Linear (electricity )

0.5

0
0 1 2 production
3 4 5 6 7
 2.7 Example 1 - Summary
Estimated Regression Line 𝑦ො = 0.4091 + 0.4988𝑥
Electricity usage = 0.4091 + 0.4988*Production
Standard Error of Estimate = 0.173
Coefficient of Determination R2 = 0.802

𝛼 = 0.05; 𝑡𝛼/2,𝑛−2 = 𝑡0.025,10 = 2.228

𝑇 = 6.37, Critical Region: |T | > tα/2, n-2 .

Since 6.37 > 2.228, reject H0 , thus, the distribution of
Electricity usage does depend on level of production
47
 2.7 Example 1 - Summary
• Using F-test. the null and alternative hypothesis are:
𝐻0 : 𝛽1 = 0
𝐻𝑎 : 𝛽1 ≠ 0

•  = 0.05. since n=12, we require F(0.05; 1,10).

from table F(0.05; 1,10) = 4.96.
decision rule is to reject h0 since:
𝐹 = 40.53 > 4.96

In conclusion, there is also a linear association between the

distribution of electricity usage and level of production
 2.7 Example 1 - Interpretation
• Production coefficient (𝛽 1 =0.498): Each unit increase in
Production($million) adds 0.498 to Electricity usage.
• 𝛽 1> 0: (positive relationship): Electricity usage increases
with the increase in Production.
• Intercept coefficient (𝛽 0 = 0.409): The Electricity usage when
Production equal to zero.
• R Square = 0.802: indicates that the model explains 80% of the
total variability in the electricity usage around its mean. (good fit)
• P-value < 0.05: The regression is significant. The change in
production impacts the electricity usage.

© 2019 Petroliam Nasional Berhad (PETRONAS) | 49

Internal
 2.8 Example 2 - Application of SLR
to Reservoir Quality Index (RQI)
Example: Given data on Permeability and Reservoir Quality
Index, RQI, investigate the dependence of RQI (Y) on
Permeability (X).

Set up the hypothesis :

H0 : β = 0 ( there is no relationship between x and y)
There is no relationship between RQI and Permeability

H1: β ≠ 0 (the straight-line model is adequate)

There is relationship between RQI and Permeability

© 2019 Petroliam Nasional Berhad (PETRONAS) | 50

Internal
 Excel Results – Example 2
Regression Statistics
Multiple R 0.680322
R Square 0.462837
Adjusted R
Square 0.461716
Standard Error 0.40947
Observations 481

ANOVA
df SS MS F Significance F
Regression 1 69.19926 69.19926 412.7226 1.22E-66
Residual 479 80.31167 0.167665
Total 480 149.5109

Coefficients Standard Error t Stat P-value

Intercept 0.309739 0.019769 15.66798 5.73E-45
Permeability
(md) 0.00171 8.42E-05 20.31558 1.22E-66

© 2019 Petroliam Nasional Berhad (PETRONAS) | 51

Internal
 Excel Results – Example 2
Permeability(md) Line Fit Plot
5.00
4.50 y = 0.3097 + 0.0017x
4.00 R² = 0.4628
3.50
3.00
RQI

2.50 RQI
2.00 Predicted RQI
1.50 Linear (RQI)

1.00
0.50
0.00
0.0 500.0 1000.0 1500.0 2000.0 2500.0
Permeability(md)
© 2019 Petroliam Nasional Berhad (PETRONAS) | 52

Internal
 2.8 Example 2 - Interpretation of
the results
• Permeability(md) coefficient (𝛽 1 =0.0017): Each unit
increase in Permeability adds 0.0017 to RQI value.
• 𝛽 1> 0: (positive relationship): RQI increases with the
increase in Permeability.
• Intercept coefficient (𝛽 0 = 0.309): The value of RQI when
Permeability equal to zero.
• R Square = 0.462837: indicates that the model explains 46% of
the total variability in the RQI values around its mean.
• P-value < 0.05: The regression is significant

© 2019 Petroliam Nasional Berhad (PETRONAS) | 53

Internal
 2.8 Example 2 (a) - Application of
SLR to Reservoir Quality Index (RQI)
Example: Given data on Permeability and Reservoir Quality
Index, RQI, investigate the dependence of RQI (Y) on
Permeability (X). For this example, outliers have been
detected and removed.
Set up the hypothesis :
H0 : β = 0 ( there is no relationship between x and y)
There is no relationship between RQI and Permeability

H1: β ≠ 0 (the straight-line model is adequate)

There is relationship between RQI and Permeability

© 2019 Petroliam Nasional Berhad (PETRONAS) | 54

Internal
 2.8 Example 2 (a) - Application of
SLR to Reservoir Quality Index (RQI)
Outlier detection arguments
Q1 0.125452
Q3 0.536934
IQR 0.411482
L Bound -0.49177
U Bound 1.154156

41 outliers were
detected and
treated

© 2019 Petroliam Nasional Berhad (PETRONAS) | 55

Internal
 Excel Results – Example 2 (a)
Observed the RQI
values on Y-axis
RQI vs Permeability (Outliers Removed )
1.80
1.60
1.40
1.20
1.00
0.80
RQI

0.60
0.40 RQI
0.20 Linear (RQI)
0.00
0.0 100.0 200.0 300.0 400.0

Permeability

© 2019 Petroliam Nasional Berhad (PETRONAS) | 56

Internal
 Excel Results – Example 2 (a)
Regression Statistics
Multiple R 0.851746
R Square 0.725471
Adjusted R What can you conclude here????
Square 0.724844
Standard Error 0.134745
Observations 440

ANOVA
df SS MS F Significance F
Regression 1 21.01512 21.01511817 1157.460884 5.0238E-125
Residual 438 7.952426 0.018156223
Total 439 28.96754

Coefficients Standard Error t Stat P-value

Intercept 0.186763 0.007286 25.63358468 3.4304E-89
Permeability
(md) 0.003609 0.000106 34.02147681 5.0238E-125

© 2019 Petroliam Nasional Berhad (PETRONAS) | 57

Internal
 Excel Results – Example 2 (a)
Regression Statistics
Multiple R 0.851746
R Square 0.725471
Adjusted R
Square 0.724844
Standard Error 0.134745
Observations 440

ANOVA
df SS MS F Significance F
Regression 1 21.01512 21.01511817 1157.460884 5.0238E-125
Residual 438 7.952426 0.018156223
Total 439 28.96754

Coefficients Standard Error t Stat P-value

Intercept 0.186763 0.007286 25.63358468 3.4304E-89
Permeability
(md) 0.003609 0.000106 34.02147681 5.0238E-125

© 2019 Petroliam Nasional Berhad (PETRONAS) | 58

Internal
 2.8 Example 2 (a) - Interpretation
of the results
• Permeability(md) coefficient (𝛽 1 =0.003609): Each unit
increase in Permeability adds 0.00361 to RQI.
• 𝛽 1> 0: (positive relationship): RQI increases with the
increase in Permeability.
• Intercept coefficient (𝛽 0 = 0.186763): The value of RQI when
Permeability equal to zero.
• R Square = 0.725471: indicates that the model explains 73% of
the total variability in the RQI values around its mean. This value
has improved significantly with the removal of outliers.
• P-value < 0.05: The regression is significant

© 2019 Petroliam Nasional Berhad (PETRONAS) | 59

Internal
60