CE 207: Applied Mathematics for Engineers
Lecture# 10
Linear Regression
(Ref: Chapter 9 of Sheldon M. Ross)
Dr. Sheikh Mokhlesur Rahman
Associate Professor, Dept. of CE
Contact: smrahman@[Link]
2
Simple Linear Regression
January 2023 Semester - SMR CE 207_Linear Regression
3
Simple Linear Regression
➢The simple linear regression considers a single
regressor or predictor x and a dependent or
response variable Y.
➢The expected value of Y at each level of x is a
random variable:
E(Y|x) = 0 + 1x
➢ We assume that each observation, Y, can be
described by the model
Y = 0 + 1x +
January 2023 Semester - SMR CE 207_Linear Regression
4
Least Squares Estimates
Objective of estimating the parameters β0 and β1 is to minimize the
sum of the squares of the vertical deviations.
Known as Least Square method
the n observations in the sample 𝑦𝑖 = 𝛽0 + 𝛽1 𝑥𝑖 + 𝜀
the sum of the squares of the deviations
𝑛 𝑛
𝐿 = 𝜀𝑖2 = (𝑦𝑖 − 𝛽0 − 𝛽1 𝑥𝑖 )2
𝑖=1 𝑖=1
January 2023 Semester - SMR CE 207_Linear Regression
5
Least Squares Estimates
least squares normal equations
January 2023 Semester - SMR CE 207_Linear Regression
6
Least Squares Estimates
The least-squares estimates of the intercept and slope in the simple
linear regression model are
ˆ = y−
ˆ x
n n
yi xi
n
i =1
yi xi − i =1
n
ˆ =
i =1
2
n
xi
n
i
x 2
− i =1
i =1 n
where 𝑦lj = (1/𝑛) ∑𝑛𝑖=1 𝑦𝑖 and 𝑥lj = (1/𝑛) ∑𝑛𝑖=1 𝑥𝑖 .
January 2023 Semester - SMR CE 207_Linear Regression
7
Simple Linear Regression
The fitted or estimated regression line is therefore
ˆ +
yˆ = ˆ x
Note that each pair of observations satisfies the relationship
yi = ˆ + ˆ xi + ei , i = 1, 2, , n
Where 𝑒𝑖 = 𝑦𝑖 − 𝑦ො𝑖 is called the residual. The residual describes the
error in the fit of the model to the ith observation yi.
January 2023 Semester - SMR CE 207_Linear Regression
8
Simple Linear Regression
Notation
2
n
xi
n n
S xx = ( xi − x )2 = xi2 − i =1
i =1 i =1 n
𝑛 𝑛
∑𝑛𝑖=1 𝑥𝑖 ∑𝑛𝑖=1 𝑦𝑖
𝑆𝑥𝑦 = 𝑦𝑖 − 𝑦lj 𝑥𝑖 − 𝑥lj = 𝑥𝑖 𝑦𝑖 −
𝑛
𝑖=1 𝑖=1
So, β1 = 𝑆𝑥𝑦
𝑆𝑥𝑥
January 2023 Semester - SMR CE 207_Linear Regression
9
Variance of the Error, 2
Estimating 2
Residual error/ residual: 𝑒𝑖 = 𝑦𝑖 − 𝑦ො𝑖
The error sum of squares is
𝑛 𝑛
𝑆𝑆𝐸 = 𝑒𝑖2 = 𝑦𝑖 − 𝑦ො𝑖 2
𝑖=1 𝑖=1
2
𝑆𝑆𝐸
𝜎ො =
𝑛−2
Degrees of freedom n – 2.
January 2023 Semester - SMR CE 207_Linear Regression
10
Variance of the Error, 2
Estimating 2
𝑛 𝑛
2
𝑆𝑆𝐸 = 𝑦𝑖 − 𝑦ො𝑖 2
= 𝑦𝑖 − (β 0 + β 1 𝑥𝑖 )
𝑖=1 𝑖=1
𝑆𝑆𝐸 = ∑𝑛𝑖=1 𝑦𝑖 2 − β 0 ∑𝑛𝑖=1 𝑦𝑖 − β 1 ∑𝑛𝑖=1 𝑥𝑖 𝑦𝑖
Simplifying, 𝑆𝑆𝐸 = 𝑆𝑆𝑇 − β 1 𝑆𝑥𝑦
where, 𝑆𝑆𝑇 = ∑𝑛𝑖=1 𝑦𝑖 − 𝑦ത 2
= ∑𝑛𝑖=1 𝑦𝑖 2 − 𝑛𝑦ത 2
January 2023 Semester - SMR CE 207_Linear Regression
Properties of the Least Squares 11
Estimators
• Slope Properties
ˆ )= 2
E (1 1
ˆ )=
V (1
S xx
• Intercept Properties
1 x 2
E (ˆ 0 ) = 0 and V (ˆ 0 ) = 2 +
n S xx
January 2023 Semester - SMR CE 207_Linear Regression
12
Estimated Standard Error
Standard error of the slope
ෝ2
σ
𝑠𝑒(β 1 ) =
𝑆𝑥𝑥
Standard error of the intercept
1 𝑥lj 2
𝑠𝑒(β 0 ) = ෝ2
σ +
𝑛 𝑆𝑥𝑥
January 2023 Semester - SMR CE 207_Linear Regression
Hypothesis Tests in Simple Linear 13
Regression
Test of Slope
H0: 1 = 1,0
H1: 1 1,0
An appropriate test statistic would be
β 1 − β1,0
𝑇0 =
ෝ2
σ
𝑆𝑥𝑥
β 1 − β 1,0
𝑇0 =
𝑠𝑒(β 1 )
We would reject the null hypothesis if |t0| > ta/2,n - 2
January 2023 Semester - SMR CE 207_Linear Regression
Hypothesis Tests in Simple Linear 14
Regression
Test of Intercept
H0: 0 = 0,0
H1: 0 0,0
An appropriate test statistic would be
β 0 − β0,0 β 0 − β0,0
𝑇0 = =
1 𝑥lj 2 𝑠𝑒(β 0 )
ෝ2
σ +
𝑛 𝑆𝑥𝑥
We would reject the null hypothesis if |t0| > ta/2,n - 2
January 2023 Semester - SMR CE 207_Linear Regression
15
Significance of Regression
Are the slopes equal to 0?
Are the slopes equal to 0?
January 2023 Semester - SMR CE 207_Linear Regression
16
Significance of Regression
An important special case of the hypotheses is
H0: 1 = 0
H1: 1 0
These hypotheses relate to the significance of
regression.
Failure to reject H0 is equivalent to concluding that
there is no linear relationship between x and Y.
January 2023 Semester - SMR CE 207_Linear Regression
17
Significance of Regression
January 2023 Semester - SMR CE 207_Linear Regression
18
Example Problem 1
Problem: To find the relationship between fuel
consumption and speed of the car. The miles per
gallon attained at various speeds was determined.
Find the relationship between fuel consumption
and speed.
Solution:
𝑛=7 𝑥𝑖 = 420 𝑦𝑖 = 156.4
𝑥lj = 60 𝑦lj = 22.34
𝑥𝑖2 = 25900
𝑥𝑖 𝑦𝑖 = 9265
January 2023 Semester - SMR CE 207_Linear Regression
19
Example Problem 1
Problem: To find the relationship between fuel consumption and
speed of the car. The miles per gallon attained at various speeds was
determined. Find the relationship between fuel consumption and speed.
∑ 𝑥𝑖 ∑ 𝑦𝑖
𝑆𝑥𝑦 = 𝑥𝑖 𝑦𝑖 − = −119
𝑛
∑ 𝑥𝑖 2
𝑆𝑥𝑥 = 𝑥𝑖2 − = 700
𝑛
𝑆𝑥𝑦 119
β 1 = =− = −0.17
𝑆𝑥𝑥 700
β 0 = 𝑦lj − β 1 𝑥lj = 22.34 − −0.17 ∗ 60 = 32.54
𝑦ො = 32.54 − 0.17𝑥
January 2023 Semester - SMR CE 207_Linear Regression
20
Example Problem 2
Problem: Test the hypothesis that there is no regression of the speed
on fuel consumption (Does the speed of the car affect the fuel
consumption?).
Solution:
𝜀𝑖2 = 1.527 β 1
𝑇= = −8.14
𝑠𝑒(β1 )
𝜎 2 = 0.305
ෝ2
σ β 0
𝑠𝑒 β 1 = = 0.0208 𝑇= = 25.61
𝑆𝑥𝑥 𝑠𝑒(β 0 )
1 𝑥lj 2
𝑠𝑒 β 0 = ෝ2
σ + = 1.27
𝑛 𝑆𝑥𝑥
January 2023 Semester - SMR CE 207_Linear Regression
21
Table for t-Distribution
January 2023 Semester - SMR CE 207_Linear Regression
22
Regression Output from Software
Solution:
Regression Statistics
Multiple R 0.964
R Square 0.930
Adjusted R Square 0.916
Standard Error 0.553
Observations 7
ANOVA
df SS MS F p-value
Regression 1 20.23 20.23 66.23 0.00045476
Residual 5 1.53 0.31
Total 6 21.76
Coefficients Standard Error t Stat P-value Lower 95% Upper 95% Lower 95.0% Upper 95.0%
Intercept 32.54 1.27 25.61 1.6943E-06 29.28 35.81 29.28 35.81
Speed -0.17 0.02 -8.14 0.00045476 -0.22 -0.12 -0.22 -0.12
January 2023 Semester - SMR CE 207_Linear Regression
Adequacy of the Regression Model: 23
Residual Analysis
• Fitting a regression model requires several
assumptions.
▪ Errors are uncorrelated random variables with mean
zero;
▪ Errors have constant variance; and,
▪ Errors be normally distributed.
• Analysis of the residuals (𝑒𝑖 = 𝑦𝑖 − 𝑦ො𝑖 ) is frequently
helpful in checking the assumption that the errors are
approximately normally distributed with constant
variance, and in determining whether additional terms
in the model would be useful.
January 2023 Semester - SMR CE 207_Linear Regression
Adequacy of the Regression Model: 24
Residual Analysis
Patterns for residual plots.
(a) Satisfactory.
(b) Funnel.
(c) Double bow.
(d) Nonlinear.
January 2023 Semester - SMR CE 207_Linear Regression
Adequacy of the Regression Model: 25
Coefficient of Determination (R2)
• Ratio of sum of squares
2
𝑆𝑆𝑅 𝑆𝑆𝐸 0 R2 1
𝑅 = =1 −
𝑆𝑆𝑇 𝑆𝑆𝑇
• Called the coefficient of determination and is often
used to judge the adequacy of a regression model.
• We often refer (loosely) to R2 as the amount of
variability in the data explained or accounted for by
the regression model.
𝑛 𝑛 𝑛
𝑦𝑖 − 𝑦lj 2 = 𝑦ො𝑖 − 𝑦lj 2 + 𝑦𝑖 − 𝑦ො𝑖 2
𝑖=1 𝑖=1 𝑖=1
𝑆𝑆𝑇 = 𝑆𝑆𝑅 + 𝑆𝑆𝐸
January 2023 Semester - SMR CE 207_Linear Regression
26
Regression on Transformed Variable
January 2023 Semester - SMR CE 207_Linear Regression
27
Multiple Linear Regression
➢Many applications of regression analysis involve
situations in which there are more than one
independent/regressor variable.
➢A regression model that contains more than one
independent variable is called a multiple regression
model.
𝑌 = 𝛽0 + 𝛽1 𝑋1 + 𝛽2 𝑋2 + ⋯ + 𝛽𝑚 𝑋𝑚 + 𝜀
➢Suppose that the effective life of a cutting tool depends
on the cutting speed and the tool angle. A possible
multiple regression model could be
𝑌 = 𝛽0 + 𝛽1 𝑋1 + 𝛽2 𝑋2 + 𝜀
here, Y: tool life, X1: cutting speed and X2: tool angle
January 2023 Semester - SMR CE 207_Linear Regression
28
Practice Problems
• Chapter 9:
5, 11, 12, 18
January 2023 Semester - SMR CE 207_Linear Regression
