Numerical Methods and

Optimization
Topic 1:
Introduction to Numerical Methods
Lectures 1-4:

1
 Lecture 1
Introduction to Numerical Methods

 What are NUMERICAL METHODS?

 Why do we need them?
 Topics covered in NMO.

Reading Assignment: Pages 3-10 of textbook

2
Numerical Methods
Numerical Methods:
Algorithms that are used to obtain numerical
solutions of a mathematical problem.
Why do we need them?
1. No analytical solution exists,
2. An analytical solution is difficult to obtain
or not practical.

3
What do we need?
Basic Needs in the Numerical Methods:
 Practical:
Can be computed in a reasonable amount of time.
 Accurate:
 Good approximate to the true value,
 Information about the approximation error
(Bounds, error order,… ).

4
Outlines of the Course
 Number Representation
 Approximate solution of nonlinear Equations
 Solution of linear Equations (Direct methods)
 Solution of linear Equations (Iterative methods)
 Polynomial Interpolation
 Least Squares approximation
 Numerical Integration

5
Solution of Nonlinear Equations
 Some simple equations can be solved analytically:
x2  4x  3  0
4 4 2  4(1)(3)
Analytic solution roots 
2(1)
x  1 and x  3

 Many other equations have no analytical solution:

x 9  2 x 2  5  0

x  No analytic solution
xe 

6
Methods for Solving Nonlinear Equations

o Bisection Method

o Newton-Raphson Method

o Secant Method

7
Solution of Systems of Linear Equations

x1  x2  3
x1  2 x2  5
We can solve it as :
x1  3  x2 , 3  x2  2 x2  5
 x2  2, x1  3  2  1
What to do if we have
1000 equations in 1000 unknowns.
8
Cramer’s Rule is Not Practical
Cramer' s Rule can be used to solve the system :
3 1 1 3
5 2 1 5
x1   1, x2  2
1 1 1 1
1 2 1 2

But Cramer' s Rule is not practical for large problems.

To solve N equations with N unknowns, we need (N  1)(N  1)N!
multiplica tions.
To solve a 30 by 30 system, 2.3 1035 multiplica tions are needed.
A super computer needs more than 10 20 years to compute this.
9
Methods for Solving Systems of Linear
Equations
o Naive Gaussian Elimination

o Gaussian Elimination with Scaled

Partial Pivoting

o Algorithm for Tri-diagonal

Equations

10
Curve Fitting
 Given a set of data:

x 0 1 2
y 0.5 10.3 21.3

 Select a curve that best fits the data. One

choice is to find the curve so that the sum
of the square of the error is minimized.

11
Polynomial Interpolation
 Given a set of data:

xi 0 1 2
yi 0.5 10.3 15.3

 Find a polynomial P(x) whose graph

passes through all tabulated points.

yi  P( xi ) if xi is in the table
12
Methods for Curve Fitting
o Least Squares
o Linear Regression
o Nonlinear Least Squares Problems

o Interpolation
o Newton Polynomial Interpolation
o Lagrange Interpolation

13
Integration
 Some functions can be integrated
analytically:

3 3
1 2 9 1
1 xdx  2 x 1  2  2  4
But many functions have no analytical solutions :
a

e
 x2
dx  ?
0

14
Methods for Numerical Integration
o Upper and Lower Sums

o Trapezoid Method

o Romberg Method

o Gauss Quadrature

15
Summary
Topics Covered in the Course
 Numerical Methods:
Algorithms that are
 Solution of Nonlinear Equations
used to obtain  Solution of Linear Equations
numerical solution of a
 Curve Fitting
mathematical problem.
 We need them when  Least Squares
No analytical solution  Interpolation
exists or it is difficult  Numerical Integration
to obtain it.

16
 Lecture 2
Number Representation and Accuracy
 Number Representation
 Normalized Floating Point Representation
 Significant Digits
 Accuracy and Precision
 Rounding and Chopping

Reading Assignment: Chapter 3

17
Representing Real Numbers
 You are familiar with the decimal system:

312.45  3 10 2  1101  2 100  4 10 1  5 10 2

 Decimal System: Base = 10 , Digits (0,1,…,9)

 Standard Representations:

 3 1 2 . 4 5
sign integral fraction
part part
18
Normalized Floating Point Representation
 Normalized Floating Point Representation:

 d . f1 f 2 f 3 f 4  10 n
sign mantissa exponent

d  0,  n : signed exponent
 Scientific Notation: Exactly one non-zero digit appears
before decimal point.
 Advantage: Efficient in representing very small or very
large numbers.
19
Binary System

 Binary System: Base = 2, Digits {0,1}

 1. f1 f 2 f 3 f 4  2  n
sign mantissa signed exponent

(1.101)2  (1  1  21  0  22  1  23 )10  (1.625)10

20
Fact
 Numbers that have a finite expansion in one numbering
system may have an infinite expansion in another
numbering system:

(1.1)10  (1.000110011001100...) 2
 You can never represent 1.1 exactly in binary system.

21
IEEE 754 Floating-Point Standard
 Single Precision (32-bit representation)
 1-bit Sign + 8-bit Exponent + 23-bit Fraction

S Exponent8 Fraction23

 Double Precision (64-bit representation)

 1-bit Sign + 11-bit Exponent + 52-bit Fraction

S Exponent11 Fraction52
(continued)

22
Significant Digits

 Significant digits are those digits that can be

used with confidence.

 Single-Precision: 7 Significant Digits

1.175494… × 10-38 to 3.402823… × 1038

 Double-Precision: 15 Significant Digits

2.2250738… × 10-308 to 1.7976931… × 10308

23
Remarks

 Numbers that can be exactly represented are called

machine numbers.

 Difference between machine numbers is not uniform

 Sum of machine numbers is not necessarily a machine

number

24
Calculator Example
 Suppose you want to compute:
3.578 * 2.139
using a calculator with two-digit fractions

3.57 * 2.13 = 7.60

True answer: 7.653342

25
Significant Digits - Example
48.9

26
Accuracy and Precision

 Accuracy is related to the closeness to the true

value.

 Precision is related to the closeness to other

estimated values.

27
28
Rounding and Chopping

 Rounding: Replace the number by the nearest

machine number.

 Chopping: Throw all extra digits.

29
Rounding and Chopping

30
Error Definitions – True Error
Can be computed if the true value is known:

Absolute True Error

Et  true value  approximation
Absolute Percent Relative Error
true value  approximation
t  *100
true value

31
Error Definitions – Estimated Error
When the true value is not known:

Estimated Absolute Error

Ea  current estimate  previous estimate
Estimated Absolute Percent Relative Error
current estimate  previous estimate
a  *100
current estimate

32
Notation
We say that the estimate is correct to n
decimal digits if:
n
Error  10

We say that the estimate is correct to n

decimal digits rounded if:
1 n
Error   10
2

33
Summary
 Number Representation
Numbers that have a finite expansion in one numbering system
may have an infinite expansion in another numbering system.

 Normalized Floating Point Representation

 Efficient in representing very small or very large numbers,
 Difference between machine numbers is not uniform,
 Representation error depends on the number of bits used in
the mantissa.

34