Numerical Methods for Civil Engineers

Lecture 7 Curve Fitting

< Linear Regression

< Polynomial Regression
< Multiple Linear Regression

Mongkol JIRAVACHARADET
SURANAREE INSTITUTE OF ENGINEERING
UNIVERSITY OF TECHNOLOGY SCHOOL OF CIVIL ENGINEERING

LINEAR REGRESSION
We want to find the curve that will fit the data.

Candidate lines for curve fit

y = α x +β

No exact solution but many approximated solutions

 Error Between Model and Observation

y Observation: [ xi yi ]

Model: y = α x + β

Error: ei = yi – α xi – β

x
Criteria for a “Best” Fit
Find the BEST line which minimize the sum of error for all data

BEST line with error minimized ?

The problem is how to minimize the error.

We can use the error

defined as:

ei = y i − yˆ
Where
ŷ = α x i + β However, the errors can
cancel one another and
still be wrong.
ERROR Definition

To avoid ± signs cancellation, the

error may be defined as:

ei = y i − yˆ
But, the error minimization is going to have problems.
The solution is the minimization of the sum of squares.

S = ∑ (ei )
2

This will give a least square solution.

Least-Square Fit of a Straight Line

Minimize sum of the square of the errors

n n
S r = ∑ e = ∑ (y i − β − αx i )
2 2
i
i =1 i =1

Differentiate with respect to each coefficient:

∂S r
= −2 ∑( y i − β − αx i )
∂β
∂S r
= −2 ∑[( y i − β − αx i ) x i ]
∂α
Setting derivatives = 0 :

0 = ∑ y i − ∑ β − ∑ αx i

0 = ∑ y i x i − ∑ βx i − ∑ αx i2

From Σβ = n β , express equations as set of 2 unknowns ( β , α )

nβ + α ∑ x i = ∑ y i

β ∑ x i + α ∑ x i2 = ∑ y i x i

Solve equations simultaneously:

1
∑ xi yi − ∑ xi ∑ yi S xy
α= n α=
1 S xx
∑ xi2 − ( ∑ xi )
2

β = y −α x
where y and x are the mean of y and x
1
Define: S xy = Σxi yi − Σxi Σyi
n Approximated y for any x is
1
S xx = Σxi2 − ( Σxi ) ŷ = α x + β
2

n
1
S yy = Σyi2 − ( i)
Σ
2
y
n
Example: Fit a straight line to x and y values

xi yi xi2 xi yi yi2
n =7
1 0.5 1 0.5 0.25
2 2.5 4 5.0 6.25 28
3 2.0 9 6.0 4 x = =4
4 4.0 16 16.0 16 7
5 3.5 25 17.5 12.25
6 6.0 36 36.0 36 24
7 5.5 49 38.5 30.25 y = = 3 . 4286
7
Σ 28 24 140 119.5 105

(119.5) − (28)(24) / 7
α= = 0.8393 Least-square fit:
(140) − (28) 2 / 7
y = 0 . 8393 x + 0 . 0714
β = 3.4286 − 0.8393(4) = 0.0714

How good is our fit?

Sum of the square of the errors:

S r = ∑ e = ∑ ( yi − β − α xi ) = ( S xx S yy − S xy2 ) / S xx
n n
2 2
i
i =1 i =1

n
St = ∑ ( yi − y ) = S yy
2
Sum of the square around the mean:
i =1

Sr
Standard errors of the estimation: sy / x =
n−2

St
Standard deviation: sy =
n−2
 Linear regression
sy > sy/x
sy sy/x
y

St − S r S xy2
Coefficient of determination r2 = =
St S xx S yy

r2 คืออัตราสวนการแปรเปลี่ยนคา y ที่เกิดจากการเปลี่ยนคา x

For perfect fit Sr = 0 and r = r2 = 1

Example: error analysis of the linear fit y = 3.4286

α = 0.8393
( yi − y ) ( yi - β - α xi)2
2
xi yi β = 0.0714
1 0.5 8.5765 0.1687
22.7143
2 2.5 0.8622 0.5626 sy =
3 2.0 2.0408 0.3473 7−2
4 4.0 0.3265 0.3265 = 2.131
5 3.5 0.0051 0.5896
2.9911
6 6.0 6.6122 0.7972 sy / x =
7 5.5 4.2908 0.1993 7−2
= 0.773
Σ 28 24 22.7143 2.9911
St Sr
Since sy/x < sy , linear regression has merit.
22.7143 − 2.9911
r= = 0.868 = 0.932
22.7143
Linear model explains 86.8% of original uncertainty.
 OR Example: error analysis of the linear fit

xi yi x i2 xi yi y i2
S xx = 140 − 282 / 7 = 28
1 0.5 1 0.5 0.25
2 2.5 4 5.0 6.25 S yy = 105 − 242 / 7 = 22.7
3 2.0 9 6.0 4
4 4.0 16 16.0 16
5 3.5 25 17.5 12.25 S xy = 119.5 − 28 × 24 / 7 = 23.5
6 6.0 36 36.0 36
7 5.5 49 38.5 30.25 Sr = (28 × 22.7 − 23.52 ) / 28
Σ 28 24 140 119.5 105 = 2.977

Since sy/x < sy , linear regression has merit. 22.7

sy = = 2.131
7−2
23.52 2.977
r =
2
= 0.869 sy / x = = 0.772
28 × 22.7 7−2

Linear model explains 86.9% of original uncertainty.

Confidence Interval (CI)

A confidence interval is an interval in which a measurement or trial
falls corresponding to a given probability.

yˆ i ± ∆
yˆ i
y y

x xi x
For CI 95%, you can be 95% confident that the two curved
confidence bands enclose the true best-fit linear regression line,
leaving a 5% chance that the true line is outside those boundaries.
A 100 (1 - α) % confidence interval for yi is given by
Confidence interval 95% → α = 0.05

1 ( xi − x ) 2
yˆ i ± tα / 2 s y / x +
n S xx

Example: to estimate y when x is 3.4 using 95% confidence interval:

xi yi yˆ = α x + β = 0.8363(3.4) + 0.0714 = 2.9148

95% Confidence → α = 0.05 → tα/2 = t0.025(df = n-2 = 5) = 2.571

1 0.5
2 2.5
1 (3.4 − 4)2
3 2.0 Interval: 2.9148 ± (2.571) (0.772) +
4 4.0 7 28
5 3.5
6 6.0 2.9148 ± 0.7832
7 5.5

T-Distribution
Probability density function of
t 0.025 t 0.025 the t distribution:
t 0.005 t 0.005
(1 + x 2 /ν ) − (ν +1) / 2
f ( x) =
t
B(0.5, 0.5ν ) ν
95% where B is the beta function and
99% ν is a positive integer
shape parameter.

The formula for the beta function is

1
B(α , β ) = ∫ t α −1 (1 − t ) β −1 dt
0
The following is the plot of the t probability density function for 4 different
values of the shape parameter.

ν = df
Degree of freedom

In fact, the t distribution with ν equal to 1 is a Cauchy distribution.

The t distribution approaches a normal distribution as ν becomes large.
The approximation is quite good for values of ν > 30.

Critical Values of t
Confidence Interval
80% 90% 95% 98% 99% 99.8%
df 0.10 0.05 0.025 0.01 0.005 0.001

1 3.078 6.314 12.706 31.821 63.657 318.313

2 1.886 2.920 4.303 6.965 9.925 22.327
3 1.638 2.353 3.182 4.541 5.841 10.215
4 1.533 2.132 2.776 3.747 4.604 7.173
5 1.476 2.015 2.571 3.365 4.032 5.893

6 1.440 1.943 2.447 3.143 3.707 5.208

7 1.415 1.895 2.365 2.998 3.499 4.782
8 1.397 1.860 2.306 2.896 3.355 4.499
9 1.383 1.833 2.262 2.821 3.250 4.296
10 1.372 1.812 2.228 2.764 3.169 4.143

11 1.363 1.796 2.201 2.718 3.106 4.024

12 1.356 1.782 2.179 2.681 3.055 3.929
13 1.350 1.771 2.160 2.650 3.012 3.852
14 1.345 1.761 2.145 2.624 2.977 3.787
15 1.341 1.753 2.131 2.602 2.947 3.733

16 1.337 1.746 2.120 2.583 2.921 3.686

17 1.333 1.740 2.110 2.567 2.898 3.646
18 1.330 1.734 2.101 2.552 2.878 3.610
19 1.328 1.729 2.093 2.539 2.861 3.579
20 1.325 1.725 2.086 2.528 2.845 3.552
 Confidence Interval
80% 90% 95% 98% 99% 99.8%
df 0.10 0.05 0.025 0.01 0.005 0.001

21 1.323 1.721 2.080 2.518 2.831 3.527

22 1.321 1.717 2.074 2.508 2.819 3.505
23 1.319 1.714 2.069 2.500 2.807 3.485
24 1.318 1.711 2.064 2.492 2.797 3.467
25 1.316 1.708 2.060 2.485 2.787 3.450

26 1.315 1.706 2.056 2.479 2.779 3.435

27 1.314 1.703 2.052 2.473 2.771 3.421
28 1.313 1.701 2.048 2.467 2.763 3.408
29 1.311 1.699 2.045 2.462 2.756 3.396
30 1.310 1.697 2.042 2.457 2.750 3.385

31 1.309 1.696 2.040 2.453 2.744 3.375

32 1.309 1.694 2.037 2.449 2.738 3.365
33 1.308 1.692 2.035 2.445 2.733 3.356
34 1.307 1.691 2.032 2.441 2.728 3.348
35 1.306 1.690 2.030 2.438 2.724 3.340

36 1.306 1.688 2.028 2.434 2.719 3.333

37 1.305 1.687 2.026 2.431 2.715 3.326
38 1.304 1.686 2.024 2.429 2.712 3.319
39 1.304 1.685 2.023 2.426 2.708 3.313
40 1.303 1.684 2.021 2.423 2.704 3.307

Confidence Interval
80% 90% 95% 98% 99% 99.8%
df 0.10 0.05 0.025 0.01 0.005 0.001

41 1.303 1.683 2.020 2.421 2.701 3.301

42 1.302 1.682 2.018 2.418 2.698 3.296
43 1.302 1.681 2.017 2.416 2.695 3.291
44 1.301 1.680 2.015 2.414 2.692 3.286
45 1.301 1.679 2.014 2.412 2.690 3.281

46 1.300 1.679 2.013 2.410 2.687 3.277

47 1.300 1.678 2.012 2.408 2.685 3.273
48 1.299 1.677 2.011 2.407 2.682 3.269
49 1.299 1.677 2.010 2.405 2.680 3.265
50 1.299 1.676 2.009 2.403 2.678 3.261

51 1.298 1.675 2.008 2.402 2.676 3.258

52 1.298 1.675 2.007 2.400 2.674 3.255
53 1.298 1.674 2.006 2.399 2.672 3.251
54 1.297 1.674 2.005 2.397 2.670 3.248
55 1.297 1.673 2.004 2.396 2.668 3.245

56 1.297 1.673 2.003 2.395 2.667 3.242

57 1.297 1.672 2.002 2.394 2.665 3.239
58 1.296 1.672 2.002 2.392 2.663 3.237
59 1.296 1.671 2.001 2.391 2.662 3.234
60 1.296 1.671 2.000 2.390 2.660 3.232
 Confidence Interval
80% 90% 95% 98% 99% 99.8%
df 0.10 0.05 0.025 0.01 0.005 0.001

61 1.296 1.670 2.000 2.389 2.659 3.229

62 1.295 1.670 1.999 2.388 2.657 3.227
63 1.295 1.669 1.998 2.387 2.656 3.225
64 1.295 1.669 1.998 2.386 2.655 3.223
65 1.295 1.669 1.997 2.385 2.654 3.220

66 1.295 1.668 1.997 2.384 2.652 3.218

67 1.294 1.668 1.996 2.383 2.651 3.216
68 1.294 1.668 1.995 2.382 2.650 3.214
69 1.294 1.667 1.995 2.382 2.649 3.213
70 1.294 1.667 1.994 2.381 2.648 3.211

71 1.294 1.667 1.994 2.380 2.647 3.209

72 1.293 1.666 1.993 2.379 2.646 3.207
73 1.293 1.666 1.993 2.379 2.645 3.206
74 1.293 1.666 1.993 2.378 2.644 3.204
75 1.293 1.665 1.992 2.377 2.643 3.202

76 1.293 1.665 1.992 2.376 2.642 3.201

77 1.293 1.665 1.991 2.376 2.641 3.199
78 1.292 1.665 1.991 2.375 2.640 3.198
79 1.292 1.664 1.990 2.374 2.640 3.197
80 1.292 1.664 1.990 2.374 2.639 3.195

Confidence Interval
80% 90% 95% 98% 99% 99.8%
df 0.10 0.05 0.025 0.01 0.005 0.001

81 1.292 1.664 1.990 2.373 2.638 3.194

82 1.292 1.664 1.989 2.373 2.637 3.193
83 1.292 1.663 1.989 2.372 2.636 3.191
84 1.292 1.663 1.989 2.372 2.636 3.190
85 1.292 1.663 1.988 2.371 2.635 3.189

86 1.291 1.663 1.988 2.370 2.634 3.188

87 1.291 1.663 1.988 2.370 2.634 3.187
88 1.291 1.662 1.987 2.369 2.633 3.185
89 1.291 1.662 1.987 2.369 2.632 3.184
90 1.291 1.662 1.987 2.368 2.632 3.183

91 1.291 1.662 1.986 2.368 2.631 3.182

92 1.291 1.662 1.986 2.368 2.630 3.181
93 1.291 1.661 1.986 2.367 2.630 3.180
94 1.291 1.661 1.986 2.367 2.629 3.179
95 1.291 1.661 1.985 2.366 2.629 3.178

96 1.290 1.661 1.985 2.366 2.628 3.177

97 1.290 1.661 1.985 2.365 2.627 3.176
98 1.290 1.661 1.984 2.365 2.627 3.175
99 1.290 1.660 1.984 2.365 2.626 3.175
100 1.290 1.660 1.984 2.364 2.626 3.174

∞ 1.282 1.645 1.960 2.326 2.576 3.090

Polynomial Regression
Second-order polynomial:

y = a0 + a1x + a2 x2

Sum of the squares of the residuals:

S r = ∑( y i − a 0 − a 1 x i − a 2 x i2 ) 2

Take derivative with respect to each coefficients:

∂S r
= −2 ∑( y i − a 0 − a 1 x i − a 2 x i2 )
∂a 0

∂S r
= −2 ∑ x i ( y i − a 0 − a 1 x i − a 2 x i2 )
∂a 1

∂S r
= −2 ∑ x i2 ( y i − a 0 − a 1 x i − a 2 x i2 )
∂a 2

Normal equations:

( )
n a 0 + (∑ x i )a 1 + ∑ x i2 a 2 = ∑ y i

(∑ x i )a 0 + (∑ x i2 )a + (∑ x )a
1 i
3
2 = ∑ x iy i

(∑ x )a + (∑ x
i
2
0 i
3
)a + (∑ x )a
1 i
4
2 = ∑ x i2 y i

For mth-order polynomial: y = a0 + a1x + a2 x2 + . . . + amxm

We have to solve m+1 simultaneous linear equations.

MATLAB polyfit Function
For second-order polynomial, we can define
 x12 x1 1  y1 
 2  y   c1 
, Y =  2  , C =  c2 
x x2 1
A= 2
M M M M   
 2     c3 
 xm xm 1  ym 
and show that C = ( A ' A) −1 A ' Y or C=A -1Y
Fit norm Fit QR
>> C = polyfit(x, y, n)
>> [C, S] = polyfit(x, y, n)

x = independent variable C = coeff. of polynomial in

y = dependent variable descending power
n = degree of polynomial S = data structure for polyval
function

Example: Fit a second-order polynomial to the data.

( yi − y )
2
xi yi ( yi - a0 - a1xi - a2xi2)2
0 2.1 544.44 0.14332
1 7.7 314.47 1.00286
2 13.6 140.03 1.08158
3 27.2 3.12 0.80491
4 40.9 239.22 0.61951
5 61.1 1272.11 0.09439

Σ 15 152.6 2513.39 3.74657

From the given data: m=2 ∑ xi = 15 ∑ xi4 = 979

n=6 ∑ yi = 152.6 ∑ xi yi = 585.6

x = 2.5 ∑ xi2 = 55 ∑ xi2 yi = 585.6

y = 25.433 ∑ xi3 = 225

Simultaneous linear equations
6 15 25  a0   152.6 
15    
 55 225   a1  =  585.6 
55 225 979  a2  2488.8

Solving these equation gives a0 = 2.47857, a1 = 2.35929, and

a2 = 1.86071.
Least-squares quadratic equation:

y = 2.47857 + 2.35929x + 1.86071x2

Coefficient of determination:

2513.39 − 3.74657
r= = 0.99851 = 0.99925
2513.39

Solving by MATLAB polyfit Function

>> x = [0 1 2 3 4 5];
>> y = [2.1 7.7 13.6 27.2 40.9 61.1];
>> c = polyfit(x, y, 2)
>> [c, s] = polyfit(x, y, 2)
>> st = sum((y - mean(y)).^2)
>> sr = sum((y - polyval(c, x)).^2)
>> r = sqrt((st - sr) / st)
MATLAB polyval Function
Evaluate polynomial at the points defined by the input vector

>> y = polyval(c, x)
where x = Input vector
y = Value of polynomial evaluated at x
c = vector of coefficient in descending order

Y = c(1)*xn + c(2)*x(n-1) + ... + c(n)*x + c(n+1)

Example: y = 1.86071x2 + 2.35929x + 2.47857

>> c = [1.86071 2.35929 2.47857]

Polynomial Interpolation
70
60
50
40
y

30
20
10
0
0 1 2 3 4 5
x

>> y2 = polyval(c,x)
>> plot(x, y, ’o’, x, y2)
 Error Bounds

By passing an optional second output parameter from polyfit as an input

to polyval.

>> [c,s] = polyfit(x,y,2)

>> [y2,delta] = polyval(c,x,s)

>> plot(x,y,'o',x,y2,'g-',x,y2+2*delta,'r:',x,y2-2*delta,'r:')

Interval of ±2∆ = 95% confidence interval

Linear Regression Example:

xi yi
1 0.5
2 2.5
3 2.0
4 4.0
5 3.5
6 6.0
7 5.5

>> [c,s] = polyfit(x,y,1)

>> [y2,delta] = polyval(c,x,s)

>> plot(x,y,'o',x,y2,'g-',x,y2+2*delta,'r:',x,y2-2*delta,'r:')
Multiple Linear Regression

y = c0 + c1x1 + c2x2 + . . . + cpxp

Example case: two independent variables y = c0 + c1x1 + c2x2

Sum of squares of the residual: S r = ∑ ( yi − c0 − c1 x1i − c2 x2 i ) 2

Differentiate with respect to unknowns:

∂S r
= −2 ∑( yi − c0 − c1 x1i − c2 x2 i )
∂c0
∂S r
= −2 ∑ x1i ( yi − c0 − c1 x1i − c2 x2i )
∂c1
∂S r
= −2 ∑ x2i ( yi − c0 − c1 x1i − c2 x2i )
∂c2

Setting partial derivatives = 0 and expressing result in matrix form:

 n ∑ x1i ∑ x2i  c0   ∑ yi 

∑ x    
 1i ∑ x12i ∑ x1i x2i   c1  =  ∑ x1i yi 
 ∑ x2i ∑ x1i x2i ∑ x22i  c2  ∑ x2i yi 

Example:
x1 x2 y  6 16.5 14  c0   54 
16.5    
 76.25 48  c1  = 243.5
0 0 5
2 1 10  14 48 54  c2   100 
2.5 2 9
1 3 0
c0 = 5
4 6 3 c1 = 4
7 2 27
c2 = −3
Multivariate Fit in MATLAB
c0 + c1x11 + c2x12 + . . . + cpx1p = y1
c0 + c1x21 + c2x22 + . . . + cpx2p = y2
.
.
.
c0 + c1xm1 + c2xm2 + . . . + cpxmp = ym
Overdetermined system of equations: A c = y
 x11 x12 L x1 p 1  c0   y1 
x x22 L x2 p 1 c  y 
A= , c =   , and y =  2 
21 1

 M M O M M M M 
     
 xm1 xm 2 L xmp 1  p 
c  ym 
Fit norm >> c = (A’*A)\(A’*y)
Fit QR >> c = A\y

Example:
x1 x2 y
0 0 5
2 1 10
2.5 2 9
1 3 0
4 6 3
7 2 27

>> x1=[0 2 2.5 1 4 7]';

>> x2=[0 1 2 3 6 2]';
>> y=[5 10 9 0 3 27]';
>> A=[x1 x2 ones(size(x1))];
>> c=A\y
 Numerical Methods for Civil Engineers

Lecture 7 Curve Fitting

< Linear Regression

< Polynomial Regression
< Multiple Linear Regression

Mongkol JIRAVACHARADET
SURANAREE INSTITUTE OF ENGINEERING
UNIVERSITY OF TECHNOLOGY SCHOOL OF CIVIL ENGINEERING

LINEAR REGRESSION
We want to find the curve that will fit the data.

Candidate lines for curve fit

y = α x +β

No exact solution but many approximated solutions

 Error Between Model and Observation

y Observation: [ xi yi ]

Model: y = α x + β

Error: ei = yi – α xi – β

x
Criteria for a “Best” Fit
Find the BEST line which minimize the sum of error for all data

BEST line with error minimized ?

The problem is how to minimize the error.

We can use the error

defined as:

ei = y i − yˆ
Where
ŷ = α x i + β However, the errors can
cancel one another and
still be wrong.
ERROR Definition

To avoid ± signs cancellation, the

error may be defined as:

ei = y i − yˆ
But, the error minimization is going to have problems.
The solution is the minimization of the sum of squares.

S = ∑ (ei )
2

This will give a least square solution.

Least-Square Fit of a Straight Line

Minimize sum of the square of the errors

n n
S r = ∑ e = ∑ (y i − β − αx i )
2 2
i
i =1 i =1

Differentiate with respect to each coefficient:

∂S r
= −2 ∑( y i − β − αx i )
∂β
∂S r
= −2 ∑[( y i − β − αx i ) x i ]
∂α
Setting derivatives = 0 :

0 = ∑ y i − ∑ β − ∑ αx i

0 = ∑ y i x i − ∑ βx i − ∑ αx i2

From Σβ = n β , express equations as set of 2 unknowns ( β , α )

nβ + α ∑ x i = ∑ y i

β ∑ x i + α ∑ x i2 = ∑ y i x i

Solve equations simultaneously:

1
∑ xi yi − ∑ xi ∑ yi S xy
α= n α=
1 S xx
∑ xi2 − ( ∑ xi )
2

β = y −α x
where y and x are the mean of y and x
1
Define: S xy = Σxi yi − Σxi Σyi
n Approximated y for any x is
1
S xx = Σxi2 − ( Σxi ) ŷ = α x + β
2

n
1
S yy = Σyi2 − ( i)
Σ
2
y
n
Example: Fit a straight line to x and y values

xi yi xi2 xi yi yi2
n =7
1 0.5 1 0.5 0.25
2 2.5 4 5.0 6.25 28
3 2.0 9 6.0 4 x = =4
4 4.0 16 16.0 16 7
5 3.5 25 17.5 12.25
6 6.0 36 36.0 36 24
7 5.5 49 38.5 30.25 y = = 3 . 4286
7
Σ 28 24 140 119.5 105

(119.5) − (28)(24) / 7
α= = 0.8393 Least-square fit:
(140) − (28) 2 / 7
y = 0 . 8393 x + 0 . 0714
β = 3.4286 − 0.8393(4) = 0.0714

How good is our fit?

Sum of the square of the errors:

S r = ∑ e = ∑ ( yi − β − α xi ) = ( S xx S yy − S xy2 ) / S xx
n n
2 2
i
i =1 i =1

n
St = ∑ ( yi − y ) = S yy
2
Sum of the square around the mean:
i =1

Sr
Standard errors of the estimation: sy / x =
n−2

St
Standard deviation: sy =
n−2
 Linear regression
sy > sy/x
sy sy/x
y

St − S r S xy2
Coefficient of determination r2 = =
St S xx S yy

r2 คืออัตราสวนการแปรเปลี่ยนคา y ที่เกิดจากการเปลี่ยนคา x

For perfect fit Sr = 0 and r = r2 = 1

Example: error analysis of the linear fit y = 3.4286

α = 0.8393
( yi − y ) ( yi - β - α xi)2
2
xi yi β = 0.0714
1 0.5 8.5765 0.1687
22.7143
2 2.5 0.8622 0.5626 sy =
3 2.0 2.0408 0.3473 7−2
4 4.0 0.3265 0.3265 = 2.131
5 3.5 0.0051 0.5896
2.9911
6 6.0 6.6122 0.7972 sy / x =
7 5.5 4.2908 0.1993 7−2
= 0.773
Σ 28 24 22.7143 2.9911
St Sr
Since sy/x < sy , linear regression has merit.
22.7143 − 2.9911
r= = 0.868 = 0.932
22.7143
Linear model explains 86.8% of original uncertainty.
 OR Example: error analysis of the linear fit

xi yi x i2 xi yi y i2
S xx = 140 − 282 / 7 = 28
1 0.5 1 0.5 0.25
2 2.5 4 5.0 6.25 S yy = 105 − 242 / 7 = 22.7
3 2.0 9 6.0 4
4 4.0 16 16.0 16
5 3.5 25 17.5 12.25 S xy = 119.5 − 28 × 24 / 7 = 23.5
6 6.0 36 36.0 36
7 5.5 49 38.5 30.25 Sr = (28 × 22.7 − 23.52 ) / 28
Σ 28 24 140 119.5 105 = 2.977

Since sy/x < sy , linear regression has merit. 22.7

sy = = 2.131
7−2
23.52 2.977
r =
2
= 0.869 sy / x = = 0.772
28 × 22.7 7−2

Linear model explains 86.9% of original uncertainty.

Confidence Interval (CI)

A confidence interval is an interval in which a measurement or trial
falls corresponding to a given probability.

yˆ i ± ∆
yˆ i
y y

x xi x
For CI 95%, you can be 95% confident that the two curved
confidence bands enclose the true best-fit linear regression line,
leaving a 5% chance that the true line is outside those boundaries.
A 100 (1 - α) % confidence interval for yi is given by
Confidence interval 95% → α = 0.05

1 ( xi − x ) 2
yˆ i ± tα / 2 s y / x +
n S xx

Example: to estimate y when x is 3.4 using 95% confidence interval:

xi yi yˆ = α x + β = 0.8363(3.4) + 0.0714 = 2.9148

95% Confidence → α = 0.05 → tα/2 = t0.025(df = n-2 = 5) = 2.571

1 0.5
2 2.5
1 (3.4 − 4)2
3 2.0 Interval: 2.9148 ± (2.571) (0.772) +
4 4.0 7 28
5 3.5
6 6.0 2.9148 ± 0.7832
7 5.5

T-Distribution
Probability density function of
t 0.025 t 0.025 the t distribution:
t 0.005 t 0.005
(1 + x 2 /ν ) − (ν +1) / 2
f ( x) =
t
B(0.5, 0.5ν ) ν
95% where B is the beta function and
99% ν is a positive integer
shape parameter.

The formula for the beta function is

1
B(α , β ) = ∫ t α −1 (1 − t ) β −1 dt
0
The following is the plot of the t probability density function for 4 different
values of the shape parameter.

ν = df
Degree of freedom

In fact, the t distribution with ν equal to 1 is a Cauchy distribution.

The t distribution approaches a normal distribution as ν becomes large.
The approximation is quite good for values of ν > 30.

Critical Values of t
Confidence Interval
80% 90% 95% 98% 99% 99.8%
df 0.10 0.05 0.025 0.01 0.005 0.001

1 3.078 6.314 12.706 31.821 63.657 318.313

2 1.886 2.920 4.303 6.965 9.925 22.327
3 1.638 2.353 3.182 4.541 5.841 10.215
4 1.533 2.132 2.776 3.747 4.604 7.173
5 1.476 2.015 2.571 3.365 4.032 5.893

6 1.440 1.943 2.447 3.143 3.707 5.208

7 1.415 1.895 2.365 2.998 3.499 4.782
8 1.397 1.860 2.306 2.896 3.355 4.499
9 1.383 1.833 2.262 2.821 3.250 4.296
10 1.372 1.812 2.228 2.764 3.169 4.143

11 1.363 1.796 2.201 2.718 3.106 4.024

12 1.356 1.782 2.179 2.681 3.055 3.929
13 1.350 1.771 2.160 2.650 3.012 3.852
14 1.345 1.761 2.145 2.624 2.977 3.787
15 1.341 1.753 2.131 2.602 2.947 3.733

16 1.337 1.746 2.120 2.583 2.921 3.686

17 1.333 1.740 2.110 2.567 2.898 3.646
18 1.330 1.734 2.101 2.552 2.878 3.610
19 1.328 1.729 2.093 2.539 2.861 3.579
20 1.325 1.725 2.086 2.528 2.845 3.552
 Confidence Interval
80% 90% 95% 98% 99% 99.8%
df 0.10 0.05 0.025 0.01 0.005 0.001

21 1.323 1.721 2.080 2.518 2.831 3.527

22 1.321 1.717 2.074 2.508 2.819 3.505
23 1.319 1.714 2.069 2.500 2.807 3.485
24 1.318 1.711 2.064 2.492 2.797 3.467
25 1.316 1.708 2.060 2.485 2.787 3.450

26 1.315 1.706 2.056 2.479 2.779 3.435

27 1.314 1.703 2.052 2.473 2.771 3.421
28 1.313 1.701 2.048 2.467 2.763 3.408
29 1.311 1.699 2.045 2.462 2.756 3.396
30 1.310 1.697 2.042 2.457 2.750 3.385

31 1.309 1.696 2.040 2.453 2.744 3.375

32 1.309 1.694 2.037 2.449 2.738 3.365
33 1.308 1.692 2.035 2.445 2.733 3.356
34 1.307 1.691 2.032 2.441 2.728 3.348
35 1.306 1.690 2.030 2.438 2.724 3.340

36 1.306 1.688 2.028 2.434 2.719 3.333

37 1.305 1.687 2.026 2.431 2.715 3.326
38 1.304 1.686 2.024 2.429 2.712 3.319
39 1.304 1.685 2.023 2.426 2.708 3.313
40 1.303 1.684 2.021 2.423 2.704 3.307

Confidence Interval
80% 90% 95% 98% 99% 99.8%
df 0.10 0.05 0.025 0.01 0.005 0.001

41 1.303 1.683 2.020 2.421 2.701 3.301

42 1.302 1.682 2.018 2.418 2.698 3.296
43 1.302 1.681 2.017 2.416 2.695 3.291
44 1.301 1.680 2.015 2.414 2.692 3.286
45 1.301 1.679 2.014 2.412 2.690 3.281

46 1.300 1.679 2.013 2.410 2.687 3.277

47 1.300 1.678 2.012 2.408 2.685 3.273
48 1.299 1.677 2.011 2.407 2.682 3.269
49 1.299 1.677 2.010 2.405 2.680 3.265
50 1.299 1.676 2.009 2.403 2.678 3.261

51 1.298 1.675 2.008 2.402 2.676 3.258

52 1.298 1.675 2.007 2.400 2.674 3.255
53 1.298 1.674 2.006 2.399 2.672 3.251
54 1.297 1.674 2.005 2.397 2.670 3.248
55 1.297 1.673 2.004 2.396 2.668 3.245

56 1.297 1.673 2.003 2.395 2.667 3.242

57 1.297 1.672 2.002 2.394 2.665 3.239
58 1.296 1.672 2.002 2.392 2.663 3.237
59 1.296 1.671 2.001 2.391 2.662 3.234
60 1.296 1.671 2.000 2.390 2.660 3.232
 Confidence Interval
80% 90% 95% 98% 99% 99.8%
df 0.10 0.05 0.025 0.01 0.005 0.001

61 1.296 1.670 2.000 2.389 2.659 3.229

62 1.295 1.670 1.999 2.388 2.657 3.227
63 1.295 1.669 1.998 2.387 2.656 3.225
64 1.295 1.669 1.998 2.386 2.655 3.223
65 1.295 1.669 1.997 2.385 2.654 3.220

66 1.295 1.668 1.997 2.384 2.652 3.218

67 1.294 1.668 1.996 2.383 2.651 3.216
68 1.294 1.668 1.995 2.382 2.650 3.214
69 1.294 1.667 1.995 2.382 2.649 3.213
70 1.294 1.667 1.994 2.381 2.648 3.211

71 1.294 1.667 1.994 2.380 2.647 3.209

72 1.293 1.666 1.993 2.379 2.646 3.207
73 1.293 1.666 1.993 2.379 2.645 3.206
74 1.293 1.666 1.993 2.378 2.644 3.204
75 1.293 1.665 1.992 2.377 2.643 3.202

76 1.293 1.665 1.992 2.376 2.642 3.201

77 1.293 1.665 1.991 2.376 2.641 3.199
78 1.292 1.665 1.991 2.375 2.640 3.198
79 1.292 1.664 1.990 2.374 2.640 3.197
80 1.292 1.664 1.990 2.374 2.639 3.195

Confidence Interval
80% 90% 95% 98% 99% 99.8%
df 0.10 0.05 0.025 0.01 0.005 0.001

81 1.292 1.664 1.990 2.373 2.638 3.194

82 1.292 1.664 1.989 2.373 2.637 3.193
83 1.292 1.663 1.989 2.372 2.636 3.191
84 1.292 1.663 1.989 2.372 2.636 3.190
85 1.292 1.663 1.988 2.371 2.635 3.189

86 1.291 1.663 1.988 2.370 2.634 3.188

87 1.291 1.663 1.988 2.370 2.634 3.187
88 1.291 1.662 1.987 2.369 2.633 3.185
89 1.291 1.662 1.987 2.369 2.632 3.184
90 1.291 1.662 1.987 2.368 2.632 3.183

91 1.291 1.662 1.986 2.368 2.631 3.182

92 1.291 1.662 1.986 2.368 2.630 3.181
93 1.291 1.661 1.986 2.367 2.630 3.180
94 1.291 1.661 1.986 2.367 2.629 3.179
95 1.291 1.661 1.985 2.366 2.629 3.178

96 1.290 1.661 1.985 2.366 2.628 3.177

97 1.290 1.661 1.985 2.365 2.627 3.176
98 1.290 1.661 1.984 2.365 2.627 3.175
99 1.290 1.660 1.984 2.365 2.626 3.175
100 1.290 1.660 1.984 2.364 2.626 3.174

∞ 1.282 1.645 1.960 2.326 2.576 3.090

Polynomial Regression
Second-order polynomial:

y = a0 + a1x + a2 x2

Sum of the squares of the residuals:

S r = ∑( y i − a 0 − a 1 x i − a 2 x i2 ) 2

Take derivative with respect to each coefficients:

∂S r
= −2 ∑( y i − a 0 − a 1 x i − a 2 x i2 )
∂a 0

∂S r
= −2 ∑ x i ( y i − a 0 − a 1 x i − a 2 x i2 )
∂a 1

∂S r
= −2 ∑ x i2 ( y i − a 0 − a 1 x i − a 2 x i2 )
∂a 2

Normal equations:

( )
n a 0 + (∑ x i )a 1 + ∑ x i2 a 2 = ∑ y i

(∑ x i )a 0 + (∑ x i2 )a + (∑ x )a
1 i
3
2 = ∑ x iy i

(∑ x )a + (∑ x
i
2
0 i
3
)a + (∑ x )a
1 i
4
2 = ∑ x i2 y i

For mth-order polynomial: y = a0 + a1x + a2 x2 + . . . + amxm

We have to solve m+1 simultaneous linear equations.

MATLAB polyfit Function
For second-order polynomial, we can define
 x12 x1 1  y1 
 2  y   c1 
, Y =  2  , C =  c2 
x x2 1
A= 2
M M M M   
 2     c3 
 xm xm 1  ym 
and show that C = ( A ' A) −1 A ' Y or C=A -1Y
Fit norm Fit QR
>> C = polyfit(x, y, n)
>> [C, S] = polyfit(x, y, n)

x = independent variable C = coeff. of polynomial in

y = dependent variable descending power
n = degree of polynomial S = data structure for polyval
function

Example: Fit a second-order polynomial to the data.

( yi − y )
2
xi yi ( yi - a0 - a1xi - a2xi2)2
0 2.1 544.44 0.14332
1 7.7 314.47 1.00286
2 13.6 140.03 1.08158
3 27.2 3.12 0.80491
4 40.9 239.22 0.61951
5 61.1 1272.11 0.09439

Σ 15 152.6 2513.39 3.74657

From the given data: m=2 ∑ xi = 15 ∑ xi4 = 979

n=6 ∑ yi = 152.6 ∑ xi yi = 585.6

x = 2.5 ∑ xi2 = 55 ∑ xi2 yi = 585.6

y = 25.433 ∑ xi3 = 225

Simultaneous linear equations
6 15 25  a0   152.6 
15    
 55 225   a1  =  585.6 
55 225 979  a2  2488.8

Solving these equation gives a0 = 2.47857, a1 = 2.35929, and

a2 = 1.86071.
Least-squares quadratic equation:

y = 2.47857 + 2.35929x + 1.86071x2

Coefficient of determination:

2513.39 − 3.74657
r= = 0.99851 = 0.99925
2513.39

Solving by MATLAB polyfit Function

>> x = [0 1 2 3 4 5];
>> y = [2.1 7.7 13.6 27.2 40.9 61.1];
>> c = polyfit(x, y, 2)
>> [c, s] = polyfit(x, y, 2)
>> st = sum((y - mean(y)).^2)
>> sr = sum((y - polyval(c, x)).^2)
>> r = sqrt((st - sr) / st)
MATLAB polyval Function
Evaluate polynomial at the points defined by the input vector

>> y = polyval(c, x)
where x = Input vector
y = Value of polynomial evaluated at x
c = vector of coefficient in descending order

Y = c(1)*xn + c(2)*x(n-1) + ... + c(n)*x + c(n+1)

Example: y = 1.86071x2 + 2.35929x + 2.47857

>> c = [1.86071 2.35929 2.47857]

Polynomial Interpolation
70
60
50
40
y

30
20
10
0
0 1 2 3 4 5
x

>> y2 = polyval(c,x)
>> plot(x, y, ’o’, x, y2)
 Error Bounds

By passing an optional second output parameter from polyfit as an input

to polyval.

>> [c,s] = polyfit(x,y,2)

>> [y2,delta] = polyval(c,x,s)

>> plot(x,y,'o',x,y2,'g-',x,y2+2*delta,'r:',x,y2-2*delta,'r:')

Interval of ±2∆ = 95% confidence interval

Linear Regression Example:

xi yi
1 0.5
2 2.5
3 2.0
4 4.0
5 3.5
6 6.0
7 5.5

>> [c,s] = polyfit(x,y,1)

>> [y2,delta] = polyval(c,x,s)

>> plot(x,y,'o',x,y2,'g-',x,y2+2*delta,'r:',x,y2-2*delta,'r:')
Multiple Linear Regression

y = c0 + c1x1 + c2x2 + . . . + cpxp

Example case: two independent variables y = c0 + c1x1 + c2x2

Sum of squares of the residual: S r = ∑ ( yi − c0 − c1 x1i − c2 x2 i ) 2

Differentiate with respect to unknowns:

∂S r
= −2 ∑( yi − c0 − c1 x1i − c2 x2 i )
∂c0
∂S r
= −2 ∑ x1i ( yi − c0 − c1 x1i − c2 x2i )
∂c1
∂S r
= −2 ∑ x2i ( yi − c0 − c1 x1i − c2 x2i )
∂c2

Setting partial derivatives = 0 and expressing result in matrix form:

 n ∑ x1i ∑ x2i  c0   ∑ yi 

∑ x    
 1i ∑ x12i ∑ x1i x2i   c1  =  ∑ x1i yi 
 ∑ x2i ∑ x1i x2i ∑ x22i  c2  ∑ x2i yi 

Example:
x1 x2 y  6 16.5 14  c0   54 
16.5    
 76.25 48  c1  = 243.5
0 0 5
2 1 10  14 48 54  c2   100 
2.5 2 9
1 3 0
c0 = 5
4 6 3 c1 = 4
7 2 27
c2 = −3
Multivariate Fit in MATLAB
c0 + c1x11 + c2x12 + . . . + cpx1p = y1
c0 + c1x21 + c2x22 + . . . + cpx2p = y2
.
.
.
c0 + c1xm1 + c2xm2 + . . . + cpxmp = ym
Overdetermined system of equations: A c = y
 x11 x12 L x1 p 1  c0   y1 
x x22 L x2 p 1 c  y 
A= , c =   , and y =  2 
21 1

 M M O M M M M 
     
 xm1 xm 2 L xmp 1  p 
c  ym 
Fit norm >> c = (A’*A)\(A’*y)
Fit QR >> c = A\y

Example:
x1 x2 y
0 0 5
2 1 10
2.5 2 9
1 3 0
4 6 3
7 2 27

>> x1=[0 2 2.5 1 4 7]';

>> x2=[0 1 2 3 6 2]';
>> y=[5 10 9 0 3 27]';
>> A=[x1 x2 ones(size(x1))];
>> c=A\y