Numerical Analysis ME309 Curve Fitting 1

Outlines:

1. Introduction

2. Linear Regression

3. Extension of Linear Regression

4. Linearization of Nonlinear Relationships

Numerical Analysis ME309 Curve Fitting Introduction 2

Given a set of data 𝑥𝑖 , 𝑦𝑖 with 1 ≤ 𝑖 ≤ n, curve fitting revolves around finding a

mathematical model that can describe the relationship 𝑦 𝑥 such that the
prediction of the mathematical model would match, as closely as possible, the
given data.
Numerical Analysis ME309 Curve Fitting Introduction 3

There are two advantages to finding an appropriate mathematical model with a

good fit:
1- The reduction of the data
 the mathematical model will often have much fewer parameters than the
actual data.

2- the characterization of the response 𝑦 𝑥

 this allows calculating 𝒚 for any value for the variable 𝒙 within the range of
applicability of the model.
Numerical Analysis ME309 Curve Fitting Introduction 4

The first step in curve fitting (also called regression) is to assume a particular
mathematical model that might fit the data.

E.g.

𝑦 𝑥 = 𝑎0 +𝑎1 𝑥 (Straight line)

𝑦 𝑥 = 𝑎0 +𝑎1 𝑥 + 𝑎2 𝑥 2 (parabolic curve)

𝑦 𝑥 = 𝑎0 + 𝑎1−𝑥 (nonlinear curve)

Numerical Analysis ME309 Curve Fitting Introduction 5

The second step in curve fitting is to find the mathematical model parameters that

would result in the “best fit”.

 By minimize the sum of the squares of the errors between the model

prediction and the data.

We will first study:

o Linear regression → using a linear mathematical model

o Nonlinear regression → using nonlinear mathematical models

Numerical Analysis ME309 Curve Fitting Introduction 6

Choosing the Model:

Ideally, the form of the mathematical model should be chosen based on the
physics of the problem.

If nothing is know about the data, you can test different model function and
choose the best

o Observe the fit with the trend of the data

o Observe the relative magnitude of the coefficients

o Check a plot of the residual (errors between model and data)

Numerical Analysis ME309 Curve Fitting Linear Regression 7

Given a set of data 𝑥𝑖 , 𝑦𝑖 with 1 ≤ 𝑖 ≤ n, a linear model fit to this set of data
has the form:
𝑦 𝑥 = 𝑎𝑥 + 𝑏
Where 𝑎 and 𝑏 are the model parameters.
Numerical Analysis ME309 Curve Fitting Linear Regression 8

The model parameters can be found by minimizing the sum of the squares of the

difference 𝑆 between the data points and the model predictions:

∆2𝑖 = 𝑦 𝑥𝑖 − 𝑦𝑖 2

→ 𝑆 = σ𝑛𝑖 = 1 𝑦 𝑥 𝑖 − 𝑦 𝑖 2

Substituting the model form:

→ 𝑆 = σ𝑛𝑖 = 1 𝑎 𝑥 + 𝑏 − 𝑦 𝑖 2

→ We want to make this deference minimum to get 𝑎 and 𝑏

Numerical Analysis ME309 Curve Fitting Linear Regression 9

To minimize 𝑆, we need to differentiate with respect to the unknown parameters

𝑎 and 𝑏 and set the two equations to zero:

𝑛 σ𝑛 𝑥 𝑦 −σ𝑛 𝑥 σ𝑛 𝑦
𝜕𝑆
𝜕𝑎
= 2 σ𝑛𝑖=1 𝑎𝑥𝑖 + 𝑏 − 𝑦𝑖 𝑥𝑖 = 0 𝑎= 𝑖=1 𝑖 𝑖 𝑖=1 𝑖 𝑖=1 𝑖
2
𝑛 2 𝑛
𝑛 σ𝑖=1 𝑥𝑖 − σ𝑖=1 𝑥𝑖

σ𝑛 𝑛
𝑖=1 𝑦𝑖 −𝑎 σ𝑖=1 𝑥𝑖
𝜕𝑆
𝜕𝑏
= 2 σ𝑛𝑖=1 𝑎𝑥𝑖 + 𝑏 − 𝑦𝑖 = 0 𝑏= 𝑛

The model 𝑦 𝑥 = 𝑎 + 𝑏𝑥 with

these coefficients provides the
test linear fit to the data
Numerical Analysis ME309 Curve Fitting Linear Regression 10

Coefficient of Determination:
The coefficient of determination (also called correlation coefficient) denoted 𝑅2 or
𝑟 2 and pronounced 𝑅 squared, is a number that provides a statistical measure of
how well the model fits the data.
The value of 𝑅2 can be calculated as follows:
2
σ𝑛
𝑖=1 𝑦𝑖 −𝑦 𝑥𝑖
𝑅2 = 1 − 2
𝑛 1 𝑛
σ𝑖=1 𝑦𝑖 − σ𝑖=1 𝑦𝑖
𝑛

ഥ
𝒚

Note that 0 ≤ 𝑅2 ≤ 1 A value closer to 1 indicates that the model is a good fit for
the data, while a value of 0 indicates that the model does not fit the data.
Numerical Analysis ME309 Curve Fitting Linear Regression 11

Coefficient of Determination:
𝑛 2
σ𝑖=1 𝑦𝑖 −𝑦 𝑥𝑖
Consider the limiting values: 𝑅2 = 1 − 1 𝑛 2
𝑛
σ𝑖=1 𝑦𝑖 − σ𝑖=1 𝑦𝑖
𝑛

2
1. σ𝑛𝑖=1𝑦𝑖 − 𝑦 𝑥𝑖 = 0 » 𝑅2 = 1
The residual sum of square is zero, indicating the regression line is a perfect fit

2 2
2. σ𝑛𝑖=1 𝑦𝑖 − 𝑦 𝑥𝑖 = σ𝑛𝑖=1 𝑦𝑖 − 1 𝑛
σ 𝑦
𝑛 𝑖=1 𝑖
» 𝑅2 = 0
The residual sum of square is equal to the total sum of square, indicating that the
regression line is no better than a simple average i.e. 𝑦 is linearly independent of
𝑥.
Numerical Analysis ME309 Curve Fitting 12

Example 1.1
Find the best linear fit to the data:
Numerical Analysis ME309 Curve Fitting 13

Example 1.1
Numerical Analysis ME309 Curve Fitting Extension of Linear Regression 14

In the previous section, the model function 𝑦 was linear 𝑥.

However, we can have a model function 𝑦 that is linear in the unknown

coefficients but non-linear in 𝑥.

In a general sense, the model function 𝑦 can be composed of 𝑚 terms with the

following form:
Numerical Analysis ME309 Curve Fitting Extension of Linear Regression 15

Note that the linear regression model can be viewed as a special case of this

general form with only two functions 𝑓1 𝑥 = 1 and 𝑓2 𝑥 = 𝑥 .

Then the model has form of: 𝑦 𝑥 = 𝑎1 +𝑎2 𝑥

Another special case of this general form is polynomial regression where the

model function has the form:

Numerical Analysis ME309 Curve Fitting Extension of Linear Regression 16

The regression procedure constitutes finding the coefficients 𝑎𝑗 that would yield

the least sum of squared differences between the data and model prediction.

Given a set of data 𝑥𝑖 , 𝑦𝑖 with 1 ≤ 𝑖 ≤ n, and if 𝑆 is the sum of the squared

differences between a general linear regression model and the data, then 𝑆 has

the form:
Numerical Analysis ME309 Curve Fitting Extension of Linear Regression 17

To find the minimizers of 𝑆, the derivatives of 𝑆 with respect to each of the

coefficients 𝑎𝑗 can be equated to zero.

Taking the derivative of 𝑆 with respect to an arbitrary coefficient 𝑎𝑘 and equating

to zero yields the general equation:

Numerical Analysis ME309 Curve Fitting Extension of Linear Regression 18

A system of 𝑚-equations of the 𝑚 unknowns 𝑎1 , 𝑎2 , … . 𝑎𝑚 can be formed as:

Numerical Analysis ME309 Curve Fitting Extension of Linear Regression 19

It can be shown that the above system always has a unique solution when the
functions 𝑓𝑖 𝑥 are non-zero and distinct.

Solving these equations yields the best fit to the data, i.e., the best coefficients 𝑎𝑖
that would minimize the sum of the squares of the differences between the model
and the data.

Note: the coefficient of determination can be obtained as described for linear

regression (slide 10).
Numerical Analysis ME309 Curve Fitting 20

Example 1.2

Consider the data

(1 , 0.5), (2 , 2.5), (3 , 2), (4 , 4), (5 , 3.5), (6 , 6), (7 , 5.5)

And consider a model of the form

𝑦 = 𝑎1 + 𝑎2 𝑥 + 𝑎3 𝑐𝑜𝑠 π𝑥

Find the coefficients 𝑎1 , 𝑎2 and 𝑎3 that would give the best fit and 𝑅2 .
Numerical Analysis ME309 Curve Fitting 21

Example 1.2

𝑦 = 𝑎1 + 𝑎2 𝑥 + 𝑎3 𝑐𝑜𝑠 π𝑥

This model is composed of a linear combination of three functions:

𝑓1 𝑥 = 1 8

𝑓2 𝑥 = 𝑥 6

𝑓3 𝑥 = cos 𝜋𝑥
4

y
3

0
0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0
X

To find the best coefficients, the following linear system of equations needs to be solved:
Numerical Analysis ME309 Curve Fitting 22

Example 1.2

In Excel:

Therefore, the linear system of equations can be written as:

7 28 −1 𝑎1 24
28 140 −4 𝑎2 = 119.5
−1 −4 7 𝑎3 1
Numerical Analysis ME309 Curve Fitting 23

Example 1.2

Solving this system yields: 𝑎1 = 0.16369, 𝑎2 = 0.83929, 𝑎3 = 0.64583.

So, the best-fit model has the form:

𝑦 𝑥 = 0.16369 + 0.83929𝑥 + 0.64583 cos 𝜋𝑥

Numerical Analysis ME309 Curve Fitting 24

Example 1.2

𝑦 𝑥 = 0.16369 + 0.83929𝑥 + 0.64583 cos 𝜋𝑥

Numerical Analysis ME309 Curve Fitting 25

Example 1.2
𝑛 2
σ𝑖=1 𝑦𝑖 −𝑦 𝑥𝑖
To find the coefficient of determination: 𝑅2 = 1 − 1 𝑛 2
𝑛
σ𝑖=1 𝑦𝑖 − σ𝑖=1 𝑦𝑖
𝑛

0.13095
𝑅2 =1− = 0.994235 𝑅2 is closed to 1, indicating a very good fit.
22.71429
Numerical Analysis ME309 Curve Fitting 26

Example 1.3

Fit a cubic polynomial to the data (1 , 1.93),(1.1 , 1.61),(1.2 , 2.27),(1.3 , 3.19),(1.4 ,

3.19),(1.5 , 3.71),(1.6 , 4.29),(1.7 , 4.95),(1.8 , 6.07),(1.9 , 7.48),(2 , 8.72),(2.1 , 9.34),(2.2 ,
11.62).
Numerical Analysis ME309 Curve Fitting 27

Example 1.3

A cubic model has form:

𝑦 = 𝑎0 + 𝑎1 𝑥 + 𝑎2 𝑥 2 + 𝑎3 𝑥 3

This is a linear combination of the functions:

𝑓0 𝑥 = 1, 𝑓1 𝑥 = 𝑥, 𝑓2 𝑥 = 𝑥 2 , 𝑓3 𝑥 = 𝑥 3

To find the coefficients, the following linear system of equations must be solved:
Numerical Analysis ME309 Curve Fitting 28

Example 1.3

In Excel:
Numerical Analysis ME309 Curve Fitting 29

Example 1.3
Numerical Analysis ME309 Curve Fitting 30

Example 1.3

Therefore, the linear system of equations can be written as:

13 20.8 35.1 61.984 𝑎0 68.37

20.8 35.1 61.984 113.607 𝑎1 = 123.638
35.1 61.984 113.607 214.502 𝑎2 231.4056
61.984 113.607 214.502 414.623 𝑎3 444.8628
Solving this system yields: 𝑎0 = −1.6247, 𝑎1 = 6.6301, 𝑎2 = −5.7273, 𝑎3 = 2.46212

Therefore, the best-fit model has the form:

𝑦 𝑥 = −1.6247 + 6.6301𝑥 − 5.7273𝑥 2 + 2.46212𝑥 3

Numerical Analysis ME309 Curve Fitting 31

Example 1.3
𝑦 𝑥 = −1.6247 + 6.6301𝑥 − 5.7273𝑥 2 + 2.46212𝑥 3
Numerical Analysis ME309 Curve Fitting 32

Example 1.3

Coefficient of Determination:

2
σ𝑛
𝑖=1 𝑦𝑖 −𝑦 𝑥𝑖
𝑅2 = 1 − 2
𝑛 1 𝑛
σ𝑖=1 𝑦𝑖 − σ𝑖=1 𝑦𝑖
𝑛

0.9506
𝑅2 =1− = 0.9921
120.0129
Numerical Analysis ME309 Curve Fitting Linearization of Nonlinear Relationships 33

In the previous two sections, the model function was formed as a linear
combination of functions 𝑓1 , 𝑓2 , … . 𝑓𝑚 .

The minimization of the sum of the squares of the differences between the model
prediction and the data produced a linear system of equations to solve for the
coefficients in the model.

In that case 𝑦 was linear in the coefficients.

In certain situations, it is possible to convert nonlinear relationships to a linear

form.
Numerical Analysis ME309 Curve Fitting Linearization of Nonlinear Relationships 34

For example, consider the following models:

𝑦𝑒𝑥𝑝 = 𝑏1 𝑒 𝑎1 𝑥
This model is nonlinear in 𝑥 and nonlinear in unknown coefficients.
However, it can be transformed into a linear model.
𝑦𝑒𝑥𝑝 = 𝑏1 𝑒 𝑎1 𝑥
Take the natural
logarithm of both sides Then, transform back
ln 𝑦𝑒𝑥𝑝 = 𝑎1 𝑥 + ln 𝑏1 the coefficients:
𝑏1 = 𝑒 𝑐2
𝑎1 = 𝑐1
This is linear in 𝑣 𝑐1 𝑢 𝑐2
coefficients 𝑐1 and 𝑐2
Numerical Analysis ME309 Curve Fitting Linearization of Nonlinear Relationships 35

The following are some examples of nonlinear functions that can be transformed
into the linear functions:
𝑣 = 𝑎0 + 𝑎1 𝑢
Numerical Analysis ME309 Curve Fitting Linearization of Nonlinear Relationships 36

For nonlinear relationships, the coefficient of determination is not a very good

measure for how well the data fit the model.

In fact, different software will give different values for 𝑅2

We will (cautiously) use the coefficient of determination for nonlinear

relationships defined as:

2
σ𝑛 𝑦𝑖 −𝑦 𝑥𝑖
𝑅2 = 1 − 𝑖=1
σ𝑛 2
𝑖=1 𝑦𝑖
Numerical Analysis ME309 Curve Fitting 37

Example 1.4

Fit an exponential model to the data: (1,1.93) ,(1.1,1.61) ,(1.2,2.27) ,(1.3,3.19)

,(1.4,3.19) ,(1.5,3.71) ,(1.6,4.29) ,(1.7,4.95) ,(1.8,6.07) ,(1.9,7.48) ,(2,8.72)
,(2.1,9.34) ,(2.2,11.62).
Numerical Analysis ME309 Curve Fitting 38

Example 1.4
The exponential model has the form:
𝑦 = 𝑏1 𝑒 𝑎1 𝑥 ln 𝑦 = 𝑎1 𝑥 + ln 𝑏1
The data needs to be converted to 𝑥𝑖 , ln𝑦𝑖 . We will 𝑣𝑖 and 𝑢𝑖 to represent the
converted data with 𝑣𝑖 = ln𝑦𝑖 and 𝑢𝑖 = 𝑥𝑖
Numerical Analysis ME309 Curve Fitting 39

Example 1.4

The linear regression procedure (Linear Regression Section) will be used to find
the best fit for the model

ln 𝑦 = 𝑎1 𝑥 + ln 𝑏1 𝑣 = 𝑐1 𝑢 + 𝑐2

𝑛 σ𝑛 𝑢 𝑣 − σ 𝑛 𝑢 σ𝑛 𝑣
𝑐1 = 𝑖=1 𝑖 𝑖 𝑖=1 𝑖 𝑖=1 𝑖 = 13(33.8013_−20.8(19.3085) = 1.5976
𝑛 σ𝑛 𝑢 2 − σ𝑛 𝑣 2 13 35.10 −(20.8)2
𝑖=1 𝑖 𝑖=1 𝑖

σ𝑛 𝑣 −𝑐 σ𝑛
1 𝑖=1 𝑢𝑖
𝑐2 = 𝑖=1 𝑖
𝑛
= 19.3085−1.5976(20.8)
13
= −1.0709
Numerical Analysis ME309 Curve Fitting 40

Example 1.4
Using Excel:
Numerical Analysis ME309 Curve Fitting 41

Example 1.4

We can then calculate the coefficients in the original model:

𝑎1 = 𝑐1 = 1.5976 𝑏1 = 𝑒 𝑐2 = 𝑒 −1.0709 = 0.3427

Therefore, the best exponential model based on the least squares of the linearized
version has the form:

𝑦 = 0.3427𝑒1.5976𝑥
Numerical Analysis ME309 Curve Fitting 42

Example 1.4

Using Excel with an “Exponential” Trendline: 𝑦 = 0.3427𝑒 1.5976𝑥

Numerical Analysis ME309 Curve Fitting 43

Example 1.4
Coefficient of Determination:
We could calculate the coefficient of determination for the linearized version of
this model, however, it would only describe how good the linearized model is.

For the nonlinear model, we will use the

coefficient of determination as described below:

𝑛 2
σ𝑖=1 𝑦𝑖 −𝑦 𝑥𝑖
𝑅2 = 1 − σ𝑛 2
𝑖=1 𝑦𝑖

0.97
𝑅2 = 1 − 479.59 = 0.998
Numerical Analysis ME309 Curve Fitting 44

Example 1.5
Fit a power model to the data: (1,1.93) ,(1.1,1.61) ,(1.2,2.27) ,(1.3,3.19) ,(1.4,3.19)
,(1.5,3.71) ,(1.6,4.29) ,(1.7,4.95) ,(1.8,6.07) ,(1.9,7.48) ,(2,8.72) ,(2.1,9.34) ,(2.2,11.62).
Numerical Analysis ME309 Curve Fitting 45

Example 1.5
The power model has the form 𝑦 = 𝑏2 𝑥 𝑎2 .
This form can be linearized as follows: ln 𝑦 = 𝑎2 ln 𝑥 + ln 𝑏2
The data needs to be converted to ln 𝑥𝑖 , ln 𝑦𝑖 . We will 𝑣𝑖 and 𝑢𝑖 to represent the
converted data with 𝑣𝑖 = ln𝑦𝑖 and 𝑢𝑖 = ln𝑥𝑖 .
Numerical Analysis ME309 Curve Fitting 46

Example 1.5

The linear regression procedure (Linear Regression Section) will be used to find
the best fit for the model.

ln 𝑦 = 𝑎1 ln 𝑥 + ln 𝑏2 𝑣 = 𝑐1 𝑢 + 𝑐2

𝑛 σ𝑛 𝑢 𝑣 −σ𝑛 𝑢 σ𝑛 𝑣
𝑐1 = 𝑖=1 𝑖 𝑖 𝑖=1 𝑖 𝑖=1 𝑖
2 = 13(10.3985_−5.7357(19.3085)
13 3.3013 −(5.7357)2
= 2.4387
𝑛 2 𝑛
𝑛 σ𝑖=1 𝑢𝑖 − σ𝑖=1 𝑣𝑖

σ𝑛 𝑣 −𝑐 σ𝑛
𝑖=1 𝑢𝑖
𝑐2 = 𝑖=1 𝑖
𝑛
1
= 19.3085−2.4387(5.7357)
13 = 0.4387
Numerical Analysis ME309 Curve Fitting 47

Example 1.5

Using Excel:
Numerical Analysis ME309 Curve Fitting 48

Example 1.5

We can then calculate the coefficients in the original model:

𝑎2 = 𝑐1 = 2.4387 𝑏2 = 𝑒 𝑐2 = 𝑒 0.4093 = 1.5058

Therefore, the best power model based on the least squares of the linearized

version has the form:

𝑦 = 1.5058𝑥 2.4387
Numerical Analysis ME309 Curve Fitting 49

Example 1.5

Using Excel with an “Power” Trendline: 𝑦 = 1.5058𝑥 2.4387

Numerical Analysis ME309 Curve Fitting 50

Example 1.5
Coefficient of Determination:
For the nonlinear model, we will use the coefficient of determination as:

2
σ𝑛 𝑦 −𝑦 𝑥
𝑅2 = 1 − 𝑖=1 𝑖
σ𝑛 𝑦
𝑖
2
𝑖=1 𝑖

3.25
𝑅2 = 1 − 479.59 = 0.9932
Numerical Analysis ME309 Homework 1 51

Problem 1:

Fit the equation:

𝑥
𝑦=
𝑎1 +𝑎2 𝑥

to the following data:

Numerical Analysis ME309 Homework 1 52

Problem 2:

Fit the equation:

1
𝑦=
𝑎1 +𝑎2 cos 𝜑

to the following data:

Numerical Analysis ME309 Homework 1 53

Problem 17.17 and 17.31

“Numerical Methods for Engineers, 7th Edition”