0% found this document useful (0 votes)
67 views

K7 Curve Fitting

This document discusses linear regression for curve fitting. It introduces linear regression, least squares fitting of a straight line, and the equations to calculate the slope (α) and y-intercept (β) that minimize the sum of squared errors. It provides an example calculation. It also discusses measuring the accuracy of the linear fit using the standard error (sy) and coefficient of determination (r2). Finally, it explains how to calculate a confidence interval for predicting y values.
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
67 views

K7 Curve Fitting

This document discusses linear regression for curve fitting. It introduces linear regression, least squares fitting of a straight line, and the equations to calculate the slope (α) and y-intercept (β) that minimize the sum of squared errors. It provides an example calculation. It also discusses measuring the accuracy of the linear fit using the standard error (sy) and coefficient of determination (r2). Finally, it explains how to calculate a confidence interval for predicting y values.
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 36

Numerical Methods for Civil Engineers

Lecture 7 Curve Fitting

< Linear Regression


< Polynomial Regression
< Multiple Linear Regression

Mongkol JIRAVACHARADET
SURANAREE INSTITUTE OF ENGINEERING
UNIVERSITY OF TECHNOLOGY SCHOOL OF CIVIL ENGINEERING

LINEAR REGRESSION
We want to find the curve that will fit the data.

Candidate lines for curve fit


y = α x +β

No exact solution but many approximated solutions


Error Between Model and Observation

y Observation: [ xi yi ]

Model: y = α x + β

Error: ei = yi – α xi – β

x
Criteria for a “Best” Fit
Find the BEST line which minimize the sum of error for all data

BEST line with error minimized ?

The problem is how to minimize the error.

We can use the error


defined as:

ei = y i − yˆ
Where
ŷ = α x i + β However, the errors can
cancel one another and
still be wrong.
ERROR Definition

To avoid ± signs cancellation, the


error may be defined as:

ei = y i − yˆ
But, the error minimization is going to have problems.
The solution is the minimization of the sum of squares.

S = ∑ (ei )
2

This will give a least square solution.

Least-Square Fit of a Straight Line

Minimize sum of the square of the errors


n n
S r = ∑ e = ∑ (y i − β − αx i )
2 2
i
i =1 i =1

Differentiate with respect to each coefficient:


∂S r
= −2 ∑( y i − β − αx i )
∂β
∂S r
= −2 ∑[( y i − β − αx i ) x i ]
∂α
Setting derivatives = 0 :

0 = ∑ y i − ∑ β − ∑ αx i

0 = ∑ y i x i − ∑ βx i − ∑ αx i2

From Σβ = n β , express equations as set of 2 unknowns ( β , α )

nβ + α ∑ x i = ∑ y i

β ∑ x i + α ∑ x i2 = ∑ y i x i

Solve equations simultaneously:

1
∑ xi yi − ∑ xi ∑ yi S xy
α= n α=
1 S xx
∑ xi2 − ( ∑ xi )
2

β = y −α x
where y and x are the mean of y and x
1
Define: S xy = Σxi yi − Σxi Σyi
n Approximated y for any x is
1
S xx = Σxi2 − ( Σxi ) ŷ = α x + β
2

n
1
S yy = Σyi2 − ( i)
Σ
2
y
n
Example: Fit a straight line to x and y values

xi yi xi2 xi yi yi2
n =7
1 0.5 1 0.5 0.25
2 2.5 4 5.0 6.25 28
3 2.0 9 6.0 4 x = =4
4 4.0 16 16.0 16 7
5 3.5 25 17.5 12.25
6 6.0 36 36.0 36 24
7 5.5 49 38.5 30.25 y = = 3 . 4286
7
Σ 28 24 140 119.5 105

(119.5) − (28)(24) / 7
α= = 0.8393 Least-square fit:
(140) − (28) 2 / 7
y = 0 . 8393 x + 0 . 0714
β = 3.4286 − 0.8393(4) = 0.0714

How good is our fit?

Sum of the square of the errors:

S r = ∑ e = ∑ ( yi − β − α xi ) = ( S xx S yy − S xy2 ) / S xx
n n
2 2
i
i =1 i =1

n
St = ∑ ( yi − y ) = S yy
2
Sum of the square around the mean:
i =1

Sr
Standard errors of the estimation: sy / x =
n−2

St
Standard deviation: sy =
n−2
Linear regression
sy > sy/x
sy sy/x
y

St − S r S xy2
Coefficient of determination r2 = =
St S xx S yy

r2 คืออัตราสวนการแปรเปลี่ยนคา y ที่เกิดจากการเปลี่ยนคา x

For perfect fit Sr = 0 and r = r2 = 1

Example: error analysis of the linear fit y = 3.4286


α = 0.8393
( yi − y ) ( yi - β - α xi)2
2
xi yi β = 0.0714
1 0.5 8.5765 0.1687
22.7143
2 2.5 0.8622 0.5626 sy =
3 2.0 2.0408 0.3473 7−2
4 4.0 0.3265 0.3265 = 2.131
5 3.5 0.0051 0.5896
2.9911
6 6.0 6.6122 0.7972 sy / x =
7 5.5 4.2908 0.1993 7−2
= 0.773
Σ 28 24 22.7143 2.9911
St Sr
Since sy/x < sy , linear regression has merit.
22.7143 − 2.9911
r= = 0.868 = 0.932
22.7143
Linear model explains 86.8% of original uncertainty.
OR Example: error analysis of the linear fit

xi yi x i2 xi yi y i2
S xx = 140 − 282 / 7 = 28
1 0.5 1 0.5 0.25
2 2.5 4 5.0 6.25 S yy = 105 − 242 / 7 = 22.7
3 2.0 9 6.0 4
4 4.0 16 16.0 16
5 3.5 25 17.5 12.25 S xy = 119.5 − 28 × 24 / 7 = 23.5
6 6.0 36 36.0 36
7 5.5 49 38.5 30.25 Sr = (28 × 22.7 − 23.52 ) / 28
Σ 28 24 140 119.5 105 = 2.977

Since sy/x < sy , linear regression has merit. 22.7


sy = = 2.131
7−2
23.52 2.977
r =
2
= 0.869 sy / x = = 0.772
28 × 22.7 7−2

Linear model explains 86.9% of original uncertainty.

Confidence Interval (CI)


A confidence interval is an interval in which a measurement or trial
falls corresponding to a given probability.

yˆ i ± ∆
yˆ i
y y

x xi x
For CI 95%, you can be 95% confident that the two curved
confidence bands enclose the true best-fit linear regression line,
leaving a 5% chance that the true line is outside those boundaries.
A 100 (1 - α) % confidence interval for yi is given by
Confidence interval 95% → α = 0.05

1 ( xi − x ) 2
yˆ i ± tα / 2 s y / x +
n S xx

Example: to estimate y when x is 3.4 using 95% confidence interval:

xi yi yˆ = α x + β = 0.8363(3.4) + 0.0714 = 2.9148

95% Confidence → α = 0.05 → tα/2 = t0.025(df = n-2 = 5) = 2.571


1 0.5
2 2.5
1 (3.4 − 4)2
3 2.0 Interval: 2.9148 ± (2.571) (0.772) +
4 4.0 7 28
5 3.5
6 6.0 2.9148 ± 0.7832
7 5.5

T-Distribution
Probability density function of
t 0.025 t 0.025 the t distribution:
t 0.005 t 0.005
(1 + x 2 /ν ) − (ν +1) / 2
f ( x) =
t
B(0.5, 0.5ν ) ν
95% where B is the beta function and
99% ν is a positive integer
shape parameter.

The formula for the beta function is


1
B(α , β ) = ∫ t α −1 (1 − t ) β −1 dt
0
The following is the plot of the t probability density function for 4 different
values of the shape parameter.

ν = df
Degree of freedom

In fact, the t distribution with ν equal to 1 is a Cauchy distribution.


The t distribution approaches a normal distribution as ν becomes large.
The approximation is quite good for values of ν > 30.

Critical Values of t
Confidence Interval
80% 90% 95% 98% 99% 99.8%
df 0.10 0.05 0.025 0.01 0.005 0.001

1 3.078 6.314 12.706 31.821 63.657 318.313


2 1.886 2.920 4.303 6.965 9.925 22.327
3 1.638 2.353 3.182 4.541 5.841 10.215
4 1.533 2.132 2.776 3.747 4.604 7.173
5 1.476 2.015 2.571 3.365 4.032 5.893

6 1.440 1.943 2.447 3.143 3.707 5.208


7 1.415 1.895 2.365 2.998 3.499 4.782
8 1.397 1.860 2.306 2.896 3.355 4.499
9 1.383 1.833 2.262 2.821 3.250 4.296
10 1.372 1.812 2.228 2.764 3.169 4.143

11 1.363 1.796 2.201 2.718 3.106 4.024


12 1.356 1.782 2.179 2.681 3.055 3.929
13 1.350 1.771 2.160 2.650 3.012 3.852
14 1.345 1.761 2.145 2.624 2.977 3.787
15 1.341 1.753 2.131 2.602 2.947 3.733

16 1.337 1.746 2.120 2.583 2.921 3.686


17 1.333 1.740 2.110 2.567 2.898 3.646
18 1.330 1.734 2.101 2.552 2.878 3.610
19 1.328 1.729 2.093 2.539 2.861 3.579
20 1.325 1.725 2.086 2.528 2.845 3.552
Confidence Interval
80% 90% 95% 98% 99% 99.8%
df 0.10 0.05 0.025 0.01 0.005 0.001

21 1.323 1.721 2.080 2.518 2.831 3.527


22 1.321 1.717 2.074 2.508 2.819 3.505
23 1.319 1.714 2.069 2.500 2.807 3.485
24 1.318 1.711 2.064 2.492 2.797 3.467
25 1.316 1.708 2.060 2.485 2.787 3.450

26 1.315 1.706 2.056 2.479 2.779 3.435


27 1.314 1.703 2.052 2.473 2.771 3.421
28 1.313 1.701 2.048 2.467 2.763 3.408
29 1.311 1.699 2.045 2.462 2.756 3.396
30 1.310 1.697 2.042 2.457 2.750 3.385

31 1.309 1.696 2.040 2.453 2.744 3.375


32 1.309 1.694 2.037 2.449 2.738 3.365
33 1.308 1.692 2.035 2.445 2.733 3.356
34 1.307 1.691 2.032 2.441 2.728 3.348
35 1.306 1.690 2.030 2.438 2.724 3.340

36 1.306 1.688 2.028 2.434 2.719 3.333


37 1.305 1.687 2.026 2.431 2.715 3.326
38 1.304 1.686 2.024 2.429 2.712 3.319
39 1.304 1.685 2.023 2.426 2.708 3.313
40 1.303 1.684 2.021 2.423 2.704 3.307

Confidence Interval
80% 90% 95% 98% 99% 99.8%
df 0.10 0.05 0.025 0.01 0.005 0.001

41 1.303 1.683 2.020 2.421 2.701 3.301


42 1.302 1.682 2.018 2.418 2.698 3.296
43 1.302 1.681 2.017 2.416 2.695 3.291
44 1.301 1.680 2.015 2.414 2.692 3.286
45 1.301 1.679 2.014 2.412 2.690 3.281

46 1.300 1.679 2.013 2.410 2.687 3.277


47 1.300 1.678 2.012 2.408 2.685 3.273
48 1.299 1.677 2.011 2.407 2.682 3.269
49 1.299 1.677 2.010 2.405 2.680 3.265
50 1.299 1.676 2.009 2.403 2.678 3.261

51 1.298 1.675 2.008 2.402 2.676 3.258


52 1.298 1.675 2.007 2.400 2.674 3.255
53 1.298 1.674 2.006 2.399 2.672 3.251
54 1.297 1.674 2.005 2.397 2.670 3.248
55 1.297 1.673 2.004 2.396 2.668 3.245

56 1.297 1.673 2.003 2.395 2.667 3.242


57 1.297 1.672 2.002 2.394 2.665 3.239
58 1.296 1.672 2.002 2.392 2.663 3.237
59 1.296 1.671 2.001 2.391 2.662 3.234
60 1.296 1.671 2.000 2.390 2.660 3.232
Confidence Interval
80% 90% 95% 98% 99% 99.8%
df 0.10 0.05 0.025 0.01 0.005 0.001

61 1.296 1.670 2.000 2.389 2.659 3.229


62 1.295 1.670 1.999 2.388 2.657 3.227
63 1.295 1.669 1.998 2.387 2.656 3.225
64 1.295 1.669 1.998 2.386 2.655 3.223
65 1.295 1.669 1.997 2.385 2.654 3.220

66 1.295 1.668 1.997 2.384 2.652 3.218


67 1.294 1.668 1.996 2.383 2.651 3.216
68 1.294 1.668 1.995 2.382 2.650 3.214
69 1.294 1.667 1.995 2.382 2.649 3.213
70 1.294 1.667 1.994 2.381 2.648 3.211

71 1.294 1.667 1.994 2.380 2.647 3.209


72 1.293 1.666 1.993 2.379 2.646 3.207
73 1.293 1.666 1.993 2.379 2.645 3.206
74 1.293 1.666 1.993 2.378 2.644 3.204
75 1.293 1.665 1.992 2.377 2.643 3.202

76 1.293 1.665 1.992 2.376 2.642 3.201


77 1.293 1.665 1.991 2.376 2.641 3.199
78 1.292 1.665 1.991 2.375 2.640 3.198
79 1.292 1.664 1.990 2.374 2.640 3.197
80 1.292 1.664 1.990 2.374 2.639 3.195

Confidence Interval
80% 90% 95% 98% 99% 99.8%
df 0.10 0.05 0.025 0.01 0.005 0.001

81 1.292 1.664 1.990 2.373 2.638 3.194


82 1.292 1.664 1.989 2.373 2.637 3.193
83 1.292 1.663 1.989 2.372 2.636 3.191
84 1.292 1.663 1.989 2.372 2.636 3.190
85 1.292 1.663 1.988 2.371 2.635 3.189

86 1.291 1.663 1.988 2.370 2.634 3.188


87 1.291 1.663 1.988 2.370 2.634 3.187
88 1.291 1.662 1.987 2.369 2.633 3.185
89 1.291 1.662 1.987 2.369 2.632 3.184
90 1.291 1.662 1.987 2.368 2.632 3.183

91 1.291 1.662 1.986 2.368 2.631 3.182


92 1.291 1.662 1.986 2.368 2.630 3.181
93 1.291 1.661 1.986 2.367 2.630 3.180
94 1.291 1.661 1.986 2.367 2.629 3.179
95 1.291 1.661 1.985 2.366 2.629 3.178

96 1.290 1.661 1.985 2.366 2.628 3.177


97 1.290 1.661 1.985 2.365 2.627 3.176
98 1.290 1.661 1.984 2.365 2.627 3.175
99 1.290 1.660 1.984 2.365 2.626 3.175
100 1.290 1.660 1.984 2.364 2.626 3.174

∞ 1.282 1.645 1.960 2.326 2.576 3.090


Polynomial Regression
Second-order polynomial:

y = a0 + a1x + a2 x2

Sum of the squares of the residuals:


S r = ∑( y i − a 0 − a 1 x i − a 2 x i2 ) 2

Take derivative with respect to each coefficients:


∂S r
= −2 ∑( y i − a 0 − a 1 x i − a 2 x i2 )
∂a 0

∂S r
= −2 ∑ x i ( y i − a 0 − a 1 x i − a 2 x i2 )
∂a 1

∂S r
= −2 ∑ x i2 ( y i − a 0 − a 1 x i − a 2 x i2 )
∂a 2

Normal equations:

( )
n a 0 + (∑ x i )a 1 + ∑ x i2 a 2 = ∑ y i

(∑ x i )a 0 + (∑ x i2 )a + (∑ x )a
1 i
3
2 = ∑ x iy i

(∑ x )a + (∑ x
i
2
0 i
3
)a + (∑ x )a
1 i
4
2 = ∑ x i2 y i

For mth-order polynomial: y = a0 + a1x + a2 x2 + . . . + amxm

We have to solve m+1 simultaneous linear equations.


MATLAB polyfit Function
For second-order polynomial, we can define
 x12 x1 1  y1 
 2  y   c1 
, Y =  2  , C =  c2 
x x2 1
A= 2
M M M M   
 2     c3 
 xm xm 1  ym 
and show that C = ( A ' A) −1 A ' Y or C=A -1Y
Fit norm Fit QR
>> C = polyfit(x, y, n)
>> [C, S] = polyfit(x, y, n)

x = independent variable C = coeff. of polynomial in


y = dependent variable descending power
n = degree of polynomial S = data structure for polyval
function

Example: Fit a second-order polynomial to the data.

( yi − y )
2
xi yi ( yi - a0 - a1xi - a2xi2)2
0 2.1 544.44 0.14332
1 7.7 314.47 1.00286
2 13.6 140.03 1.08158
3 27.2 3.12 0.80491
4 40.9 239.22 0.61951
5 61.1 1272.11 0.09439

Σ 15 152.6 2513.39 3.74657

From the given data: m=2 ∑ xi = 15 ∑ xi4 = 979


n=6 ∑ yi = 152.6 ∑ xi yi = 585.6

x = 2.5 ∑ xi2 = 55 ∑ xi2 yi = 585.6

y = 25.433 ∑ xi3 = 225


Simultaneous linear equations
6 15 25  a0   152.6 
15    
 55 225   a1  =  585.6 
55 225 979  a2  2488.8

Solving these equation gives a0 = 2.47857, a1 = 2.35929, and


a2 = 1.86071.
Least-squares quadratic equation:

y = 2.47857 + 2.35929x + 1.86071x2


Coefficient of determination:

2513.39 − 3.74657
r= = 0.99851 = 0.99925
2513.39

Solving by MATLAB polyfit Function

>> x = [0 1 2 3 4 5];
>> y = [2.1 7.7 13.6 27.2 40.9 61.1];
>> c = polyfit(x, y, 2)
>> [c, s] = polyfit(x, y, 2)
>> st = sum((y - mean(y)).^2)
>> sr = sum((y - polyval(c, x)).^2)
>> r = sqrt((st - sr) / st)
MATLAB polyval Function
Evaluate polynomial at the points defined by the input vector

>> y = polyval(c, x)
where x = Input vector
y = Value of polynomial evaluated at x
c = vector of coefficient in descending order

Y = c(1)*xn + c(2)*x(n-1) + ... + c(n)*x + c(n+1)

Example: y = 1.86071x2 + 2.35929x + 2.47857


>> c = [1.86071 2.35929 2.47857]

Polynomial Interpolation
70
60
50
40
y

30
20
10
0
0 1 2 3 4 5
x

>> y2 = polyval(c,x)
>> plot(x, y, ’o’, x, y2)
Error Bounds

By passing an optional second output parameter from polyfit as an input


to polyval.

>> [c,s] = polyfit(x,y,2)

>> [y2,delta] = polyval(c,x,s)

>> plot(x,y,'o',x,y2,'g-',x,y2+2*delta,'r:',x,y2-2*delta,'r:')

Interval of ±2∆ = 95% confidence interval

Linear Regression Example:

xi yi
1 0.5
2 2.5
3 2.0
4 4.0
5 3.5
6 6.0
7 5.5

>> [c,s] = polyfit(x,y,1)

>> [y2,delta] = polyval(c,x,s)

>> plot(x,y,'o',x,y2,'g-',x,y2+2*delta,'r:',x,y2-2*delta,'r:')
Multiple Linear Regression

y = c0 + c1x1 + c2x2 + . . . + cpxp

Example case: two independent variables y = c0 + c1x1 + c2x2

Sum of squares of the residual: S r = ∑ ( yi − c0 − c1 x1i − c2 x2 i ) 2

Differentiate with respect to unknowns:


∂S r
= −2 ∑( yi − c0 − c1 x1i − c2 x2 i )
∂c0
∂S r
= −2 ∑ x1i ( yi − c0 − c1 x1i − c2 x2i )
∂c1
∂S r
= −2 ∑ x2i ( yi − c0 − c1 x1i − c2 x2i )
∂c2

Setting partial derivatives = 0 and expressing result in matrix form:

 n ∑ x1i ∑ x2i  c0   ∑ yi 


∑ x    
 1i ∑ x12i ∑ x1i x2i   c1  =  ∑ x1i yi 
 ∑ x2i ∑ x1i x2i ∑ x22i  c2  ∑ x2i yi 

Example:
x1 x2 y  6 16.5 14  c0   54 
16.5    
 76.25 48  c1  = 243.5
0 0 5
2 1 10  14 48 54  c2   100 
2.5 2 9
1 3 0
c0 = 5
4 6 3 c1 = 4
7 2 27
c2 = −3
Multivariate Fit in MATLAB
c0 + c1x11 + c2x12 + . . . + cpx1p = y1
c0 + c1x21 + c2x22 + . . . + cpx2p = y2
.
.
.
c0 + c1xm1 + c2xm2 + . . . + cpxmp = ym
Overdetermined system of equations: A c = y
 x11 x12 L x1 p 1  c0   y1 
x x22 L x2 p 1 c  y 
A= , c =   , and y =  2 
21 1

 M M O M M M M 
     
 xm1 xm 2 L xmp 1  p 
c  ym 
Fit norm >> c = (A’*A)\(A’*y)
Fit QR >> c = A\y

Example:
x1 x2 y
0 0 5
2 1 10
2.5 2 9
1 3 0
4 6 3
7 2 27

>> x1=[0 2 2.5 1 4 7]';


>> x2=[0 1 2 3 6 2]';
>> y=[5 10 9 0 3 27]';
>> A=[x1 x2 ones(size(x1))];
>> c=A\y
Numerical Methods for Civil Engineers

Lecture 7 Curve Fitting

< Linear Regression


< Polynomial Regression
< Multiple Linear Regression

Mongkol JIRAVACHARADET
SURANAREE INSTITUTE OF ENGINEERING
UNIVERSITY OF TECHNOLOGY SCHOOL OF CIVIL ENGINEERING

LINEAR REGRESSION
We want to find the curve that will fit the data.

Candidate lines for curve fit


y = α x +β

No exact solution but many approximated solutions


Error Between Model and Observation

y Observation: [ xi yi ]

Model: y = α x + β

Error: ei = yi – α xi – β

x
Criteria for a “Best” Fit
Find the BEST line which minimize the sum of error for all data

BEST line with error minimized ?

The problem is how to minimize the error.

We can use the error


defined as:

ei = y i − yˆ
Where
ŷ = α x i + β However, the errors can
cancel one another and
still be wrong.
ERROR Definition

To avoid ± signs cancellation, the


error may be defined as:

ei = y i − yˆ
But, the error minimization is going to have problems.
The solution is the minimization of the sum of squares.

S = ∑ (ei )
2

This will give a least square solution.

Least-Square Fit of a Straight Line

Minimize sum of the square of the errors


n n
S r = ∑ e = ∑ (y i − β − αx i )
2 2
i
i =1 i =1

Differentiate with respect to each coefficient:


∂S r
= −2 ∑( y i − β − αx i )
∂β
∂S r
= −2 ∑[( y i − β − αx i ) x i ]
∂α
Setting derivatives = 0 :

0 = ∑ y i − ∑ β − ∑ αx i

0 = ∑ y i x i − ∑ βx i − ∑ αx i2

From Σβ = n β , express equations as set of 2 unknowns ( β , α )

nβ + α ∑ x i = ∑ y i

β ∑ x i + α ∑ x i2 = ∑ y i x i

Solve equations simultaneously:

1
∑ xi yi − ∑ xi ∑ yi S xy
α= n α=
1 S xx
∑ xi2 − ( ∑ xi )
2

β = y −α x
where y and x are the mean of y and x
1
Define: S xy = Σxi yi − Σxi Σyi
n Approximated y for any x is
1
S xx = Σxi2 − ( Σxi ) ŷ = α x + β
2

n
1
S yy = Σyi2 − ( i)
Σ
2
y
n
Example: Fit a straight line to x and y values

xi yi xi2 xi yi yi2
n =7
1 0.5 1 0.5 0.25
2 2.5 4 5.0 6.25 28
3 2.0 9 6.0 4 x = =4
4 4.0 16 16.0 16 7
5 3.5 25 17.5 12.25
6 6.0 36 36.0 36 24
7 5.5 49 38.5 30.25 y = = 3 . 4286
7
Σ 28 24 140 119.5 105

(119.5) − (28)(24) / 7
α= = 0.8393 Least-square fit:
(140) − (28) 2 / 7
y = 0 . 8393 x + 0 . 0714
β = 3.4286 − 0.8393(4) = 0.0714

How good is our fit?

Sum of the square of the errors:

S r = ∑ e = ∑ ( yi − β − α xi ) = ( S xx S yy − S xy2 ) / S xx
n n
2 2
i
i =1 i =1

n
St = ∑ ( yi − y ) = S yy
2
Sum of the square around the mean:
i =1

Sr
Standard errors of the estimation: sy / x =
n−2

St
Standard deviation: sy =
n−2
Linear regression
sy > sy/x
sy sy/x
y

St − S r S xy2
Coefficient of determination r2 = =
St S xx S yy

r2 คืออัตราสวนการแปรเปลี่ยนคา y ที่เกิดจากการเปลี่ยนคา x

For perfect fit Sr = 0 and r = r2 = 1

Example: error analysis of the linear fit y = 3.4286


α = 0.8393
( yi − y ) ( yi - β - α xi)2
2
xi yi β = 0.0714
1 0.5 8.5765 0.1687
22.7143
2 2.5 0.8622 0.5626 sy =
3 2.0 2.0408 0.3473 7−2
4 4.0 0.3265 0.3265 = 2.131
5 3.5 0.0051 0.5896
2.9911
6 6.0 6.6122 0.7972 sy / x =
7 5.5 4.2908 0.1993 7−2
= 0.773
Σ 28 24 22.7143 2.9911
St Sr
Since sy/x < sy , linear regression has merit.
22.7143 − 2.9911
r= = 0.868 = 0.932
22.7143
Linear model explains 86.8% of original uncertainty.
OR Example: error analysis of the linear fit

xi yi x i2 xi yi y i2
S xx = 140 − 282 / 7 = 28
1 0.5 1 0.5 0.25
2 2.5 4 5.0 6.25 S yy = 105 − 242 / 7 = 22.7
3 2.0 9 6.0 4
4 4.0 16 16.0 16
5 3.5 25 17.5 12.25 S xy = 119.5 − 28 × 24 / 7 = 23.5
6 6.0 36 36.0 36
7 5.5 49 38.5 30.25 Sr = (28 × 22.7 − 23.52 ) / 28
Σ 28 24 140 119.5 105 = 2.977

Since sy/x < sy , linear regression has merit. 22.7


sy = = 2.131
7−2
23.52 2.977
r =
2
= 0.869 sy / x = = 0.772
28 × 22.7 7−2

Linear model explains 86.9% of original uncertainty.

Confidence Interval (CI)


A confidence interval is an interval in which a measurement or trial
falls corresponding to a given probability.

yˆ i ± ∆
yˆ i
y y

x xi x
For CI 95%, you can be 95% confident that the two curved
confidence bands enclose the true best-fit linear regression line,
leaving a 5% chance that the true line is outside those boundaries.
A 100 (1 - α) % confidence interval for yi is given by
Confidence interval 95% → α = 0.05

1 ( xi − x ) 2
yˆ i ± tα / 2 s y / x +
n S xx

Example: to estimate y when x is 3.4 using 95% confidence interval:

xi yi yˆ = α x + β = 0.8363(3.4) + 0.0714 = 2.9148

95% Confidence → α = 0.05 → tα/2 = t0.025(df = n-2 = 5) = 2.571


1 0.5
2 2.5
1 (3.4 − 4)2
3 2.0 Interval: 2.9148 ± (2.571) (0.772) +
4 4.0 7 28
5 3.5
6 6.0 2.9148 ± 0.7832
7 5.5

T-Distribution
Probability density function of
t 0.025 t 0.025 the t distribution:
t 0.005 t 0.005
(1 + x 2 /ν ) − (ν +1) / 2
f ( x) =
t
B(0.5, 0.5ν ) ν
95% where B is the beta function and
99% ν is a positive integer
shape parameter.

The formula for the beta function is


1
B(α , β ) = ∫ t α −1 (1 − t ) β −1 dt
0
The following is the plot of the t probability density function for 4 different
values of the shape parameter.

ν = df
Degree of freedom

In fact, the t distribution with ν equal to 1 is a Cauchy distribution.


The t distribution approaches a normal distribution as ν becomes large.
The approximation is quite good for values of ν > 30.

Critical Values of t
Confidence Interval
80% 90% 95% 98% 99% 99.8%
df 0.10 0.05 0.025 0.01 0.005 0.001

1 3.078 6.314 12.706 31.821 63.657 318.313


2 1.886 2.920 4.303 6.965 9.925 22.327
3 1.638 2.353 3.182 4.541 5.841 10.215
4 1.533 2.132 2.776 3.747 4.604 7.173
5 1.476 2.015 2.571 3.365 4.032 5.893

6 1.440 1.943 2.447 3.143 3.707 5.208


7 1.415 1.895 2.365 2.998 3.499 4.782
8 1.397 1.860 2.306 2.896 3.355 4.499
9 1.383 1.833 2.262 2.821 3.250 4.296
10 1.372 1.812 2.228 2.764 3.169 4.143

11 1.363 1.796 2.201 2.718 3.106 4.024


12 1.356 1.782 2.179 2.681 3.055 3.929
13 1.350 1.771 2.160 2.650 3.012 3.852
14 1.345 1.761 2.145 2.624 2.977 3.787
15 1.341 1.753 2.131 2.602 2.947 3.733

16 1.337 1.746 2.120 2.583 2.921 3.686


17 1.333 1.740 2.110 2.567 2.898 3.646
18 1.330 1.734 2.101 2.552 2.878 3.610
19 1.328 1.729 2.093 2.539 2.861 3.579
20 1.325 1.725 2.086 2.528 2.845 3.552
Confidence Interval
80% 90% 95% 98% 99% 99.8%
df 0.10 0.05 0.025 0.01 0.005 0.001

21 1.323 1.721 2.080 2.518 2.831 3.527


22 1.321 1.717 2.074 2.508 2.819 3.505
23 1.319 1.714 2.069 2.500 2.807 3.485
24 1.318 1.711 2.064 2.492 2.797 3.467
25 1.316 1.708 2.060 2.485 2.787 3.450

26 1.315 1.706 2.056 2.479 2.779 3.435


27 1.314 1.703 2.052 2.473 2.771 3.421
28 1.313 1.701 2.048 2.467 2.763 3.408
29 1.311 1.699 2.045 2.462 2.756 3.396
30 1.310 1.697 2.042 2.457 2.750 3.385

31 1.309 1.696 2.040 2.453 2.744 3.375


32 1.309 1.694 2.037 2.449 2.738 3.365
33 1.308 1.692 2.035 2.445 2.733 3.356
34 1.307 1.691 2.032 2.441 2.728 3.348
35 1.306 1.690 2.030 2.438 2.724 3.340

36 1.306 1.688 2.028 2.434 2.719 3.333


37 1.305 1.687 2.026 2.431 2.715 3.326
38 1.304 1.686 2.024 2.429 2.712 3.319
39 1.304 1.685 2.023 2.426 2.708 3.313
40 1.303 1.684 2.021 2.423 2.704 3.307

Confidence Interval
80% 90% 95% 98% 99% 99.8%
df 0.10 0.05 0.025 0.01 0.005 0.001

41 1.303 1.683 2.020 2.421 2.701 3.301


42 1.302 1.682 2.018 2.418 2.698 3.296
43 1.302 1.681 2.017 2.416 2.695 3.291
44 1.301 1.680 2.015 2.414 2.692 3.286
45 1.301 1.679 2.014 2.412 2.690 3.281

46 1.300 1.679 2.013 2.410 2.687 3.277


47 1.300 1.678 2.012 2.408 2.685 3.273
48 1.299 1.677 2.011 2.407 2.682 3.269
49 1.299 1.677 2.010 2.405 2.680 3.265
50 1.299 1.676 2.009 2.403 2.678 3.261

51 1.298 1.675 2.008 2.402 2.676 3.258


52 1.298 1.675 2.007 2.400 2.674 3.255
53 1.298 1.674 2.006 2.399 2.672 3.251
54 1.297 1.674 2.005 2.397 2.670 3.248
55 1.297 1.673 2.004 2.396 2.668 3.245

56 1.297 1.673 2.003 2.395 2.667 3.242


57 1.297 1.672 2.002 2.394 2.665 3.239
58 1.296 1.672 2.002 2.392 2.663 3.237
59 1.296 1.671 2.001 2.391 2.662 3.234
60 1.296 1.671 2.000 2.390 2.660 3.232
Confidence Interval
80% 90% 95% 98% 99% 99.8%
df 0.10 0.05 0.025 0.01 0.005 0.001

61 1.296 1.670 2.000 2.389 2.659 3.229


62 1.295 1.670 1.999 2.388 2.657 3.227
63 1.295 1.669 1.998 2.387 2.656 3.225
64 1.295 1.669 1.998 2.386 2.655 3.223
65 1.295 1.669 1.997 2.385 2.654 3.220

66 1.295 1.668 1.997 2.384 2.652 3.218


67 1.294 1.668 1.996 2.383 2.651 3.216
68 1.294 1.668 1.995 2.382 2.650 3.214
69 1.294 1.667 1.995 2.382 2.649 3.213
70 1.294 1.667 1.994 2.381 2.648 3.211

71 1.294 1.667 1.994 2.380 2.647 3.209


72 1.293 1.666 1.993 2.379 2.646 3.207
73 1.293 1.666 1.993 2.379 2.645 3.206
74 1.293 1.666 1.993 2.378 2.644 3.204
75 1.293 1.665 1.992 2.377 2.643 3.202

76 1.293 1.665 1.992 2.376 2.642 3.201


77 1.293 1.665 1.991 2.376 2.641 3.199
78 1.292 1.665 1.991 2.375 2.640 3.198
79 1.292 1.664 1.990 2.374 2.640 3.197
80 1.292 1.664 1.990 2.374 2.639 3.195

Confidence Interval
80% 90% 95% 98% 99% 99.8%
df 0.10 0.05 0.025 0.01 0.005 0.001

81 1.292 1.664 1.990 2.373 2.638 3.194


82 1.292 1.664 1.989 2.373 2.637 3.193
83 1.292 1.663 1.989 2.372 2.636 3.191
84 1.292 1.663 1.989 2.372 2.636 3.190
85 1.292 1.663 1.988 2.371 2.635 3.189

86 1.291 1.663 1.988 2.370 2.634 3.188


87 1.291 1.663 1.988 2.370 2.634 3.187
88 1.291 1.662 1.987 2.369 2.633 3.185
89 1.291 1.662 1.987 2.369 2.632 3.184
90 1.291 1.662 1.987 2.368 2.632 3.183

91 1.291 1.662 1.986 2.368 2.631 3.182


92 1.291 1.662 1.986 2.368 2.630 3.181
93 1.291 1.661 1.986 2.367 2.630 3.180
94 1.291 1.661 1.986 2.367 2.629 3.179
95 1.291 1.661 1.985 2.366 2.629 3.178

96 1.290 1.661 1.985 2.366 2.628 3.177


97 1.290 1.661 1.985 2.365 2.627 3.176
98 1.290 1.661 1.984 2.365 2.627 3.175
99 1.290 1.660 1.984 2.365 2.626 3.175
100 1.290 1.660 1.984 2.364 2.626 3.174

∞ 1.282 1.645 1.960 2.326 2.576 3.090


Polynomial Regression
Second-order polynomial:

y = a0 + a1x + a2 x2

Sum of the squares of the residuals:


S r = ∑( y i − a 0 − a 1 x i − a 2 x i2 ) 2

Take derivative with respect to each coefficients:


∂S r
= −2 ∑( y i − a 0 − a 1 x i − a 2 x i2 )
∂a 0

∂S r
= −2 ∑ x i ( y i − a 0 − a 1 x i − a 2 x i2 )
∂a 1

∂S r
= −2 ∑ x i2 ( y i − a 0 − a 1 x i − a 2 x i2 )
∂a 2

Normal equations:

( )
n a 0 + (∑ x i )a 1 + ∑ x i2 a 2 = ∑ y i

(∑ x i )a 0 + (∑ x i2 )a + (∑ x )a
1 i
3
2 = ∑ x iy i

(∑ x )a + (∑ x
i
2
0 i
3
)a + (∑ x )a
1 i
4
2 = ∑ x i2 y i

For mth-order polynomial: y = a0 + a1x + a2 x2 + . . . + amxm

We have to solve m+1 simultaneous linear equations.


MATLAB polyfit Function
For second-order polynomial, we can define
 x12 x1 1  y1 
 2  y   c1 
, Y =  2  , C =  c2 
x x2 1
A= 2
M M M M   
 2     c3 
 xm xm 1  ym 
and show that C = ( A ' A) −1 A ' Y or C=A -1Y
Fit norm Fit QR
>> C = polyfit(x, y, n)
>> [C, S] = polyfit(x, y, n)

x = independent variable C = coeff. of polynomial in


y = dependent variable descending power
n = degree of polynomial S = data structure for polyval
function

Example: Fit a second-order polynomial to the data.

( yi − y )
2
xi yi ( yi - a0 - a1xi - a2xi2)2
0 2.1 544.44 0.14332
1 7.7 314.47 1.00286
2 13.6 140.03 1.08158
3 27.2 3.12 0.80491
4 40.9 239.22 0.61951
5 61.1 1272.11 0.09439

Σ 15 152.6 2513.39 3.74657

From the given data: m=2 ∑ xi = 15 ∑ xi4 = 979


n=6 ∑ yi = 152.6 ∑ xi yi = 585.6

x = 2.5 ∑ xi2 = 55 ∑ xi2 yi = 585.6

y = 25.433 ∑ xi3 = 225


Simultaneous linear equations
6 15 25  a0   152.6 
15    
 55 225   a1  =  585.6 
55 225 979  a2  2488.8

Solving these equation gives a0 = 2.47857, a1 = 2.35929, and


a2 = 1.86071.
Least-squares quadratic equation:

y = 2.47857 + 2.35929x + 1.86071x2


Coefficient of determination:

2513.39 − 3.74657
r= = 0.99851 = 0.99925
2513.39

Solving by MATLAB polyfit Function

>> x = [0 1 2 3 4 5];
>> y = [2.1 7.7 13.6 27.2 40.9 61.1];
>> c = polyfit(x, y, 2)
>> [c, s] = polyfit(x, y, 2)
>> st = sum((y - mean(y)).^2)
>> sr = sum((y - polyval(c, x)).^2)
>> r = sqrt((st - sr) / st)
MATLAB polyval Function
Evaluate polynomial at the points defined by the input vector

>> y = polyval(c, x)
where x = Input vector
y = Value of polynomial evaluated at x
c = vector of coefficient in descending order

Y = c(1)*xn + c(2)*x(n-1) + ... + c(n)*x + c(n+1)

Example: y = 1.86071x2 + 2.35929x + 2.47857


>> c = [1.86071 2.35929 2.47857]

Polynomial Interpolation
70
60
50
40
y

30
20
10
0
0 1 2 3 4 5
x

>> y2 = polyval(c,x)
>> plot(x, y, ’o’, x, y2)
Error Bounds

By passing an optional second output parameter from polyfit as an input


to polyval.

>> [c,s] = polyfit(x,y,2)

>> [y2,delta] = polyval(c,x,s)

>> plot(x,y,'o',x,y2,'g-',x,y2+2*delta,'r:',x,y2-2*delta,'r:')

Interval of ±2∆ = 95% confidence interval

Linear Regression Example:

xi yi
1 0.5
2 2.5
3 2.0
4 4.0
5 3.5
6 6.0
7 5.5

>> [c,s] = polyfit(x,y,1)

>> [y2,delta] = polyval(c,x,s)

>> plot(x,y,'o',x,y2,'g-',x,y2+2*delta,'r:',x,y2-2*delta,'r:')
Multiple Linear Regression

y = c0 + c1x1 + c2x2 + . . . + cpxp

Example case: two independent variables y = c0 + c1x1 + c2x2

Sum of squares of the residual: S r = ∑ ( yi − c0 − c1 x1i − c2 x2 i ) 2

Differentiate with respect to unknowns:


∂S r
= −2 ∑( yi − c0 − c1 x1i − c2 x2 i )
∂c0
∂S r
= −2 ∑ x1i ( yi − c0 − c1 x1i − c2 x2i )
∂c1
∂S r
= −2 ∑ x2i ( yi − c0 − c1 x1i − c2 x2i )
∂c2

Setting partial derivatives = 0 and expressing result in matrix form:

 n ∑ x1i ∑ x2i  c0   ∑ yi 


∑ x    
 1i ∑ x12i ∑ x1i x2i   c1  =  ∑ x1i yi 
 ∑ x2i ∑ x1i x2i ∑ x22i  c2  ∑ x2i yi 

Example:
x1 x2 y  6 16.5 14  c0   54 
16.5    
 76.25 48  c1  = 243.5
0 0 5
2 1 10  14 48 54  c2   100 
2.5 2 9
1 3 0
c0 = 5
4 6 3 c1 = 4
7 2 27
c2 = −3
Multivariate Fit in MATLAB
c0 + c1x11 + c2x12 + . . . + cpx1p = y1
c0 + c1x21 + c2x22 + . . . + cpx2p = y2
.
.
.
c0 + c1xm1 + c2xm2 + . . . + cpxmp = ym
Overdetermined system of equations: A c = y
 x11 x12 L x1 p 1  c0   y1 
x x22 L x2 p 1 c  y 
A= , c =   , and y =  2 
21 1

 M M O M M M M 
     
 xm1 xm 2 L xmp 1  p 
c  ym 
Fit norm >> c = (A’*A)\(A’*y)
Fit QR >> c = A\y

Example:
x1 x2 y
0 0 5
2 1 10
2.5 2 9
1 3 0
4 6 3
7 2 27

>> x1=[0 2 2.5 1 4 7]';


>> x2=[0 1 2 3 6 2]';
>> y=[5 10 9 0 3 27]';
>> A=[x1 x2 ones(size(x1))];
>> c=A\y

You might also like