Mathematical Foundations for

Data Science
BITS Pilani MFDS Team
Pilani Campus
 BITS Pilani
Pilani Campus

DSECL ZC416, MFDS

Lecture No.1
Agenda

• Solution of linear systems – an overview

• Gauss elimination methods

• sensitivity to changes in A
• pivoting and
• operations count

• LU decomposition methods
• Doolittle’s method
• Crout’s method

BITS Pilani, Pilani Campus

Linear System
Matrix Form of the Linear System (1).
From the definition of matrix multiplication we see that the
m equations of (1) may be written as a single vector
equation
(2) Ax = b
where the coefficient matrix A = [ajk] is the m × n matrix

are column vectors.

BITS Pilani, Pilani Campus
Matrix Form of Linear System
Matrix Form of the Linear System (1). (continued)
We assume that the coefficients ajk are not all zero, so that A is not a zero
matrix. Note that x has n components, whereas b has m components. The
matrix

is called the augmented matrix of the system (1). The dashed vertical line could
be omitted, as we shall do later. It is merely a reminder that the last column of
à did not come from matrix A but came from vector b. Thus, we augmented the
matrix A.

BITS Pilani, Pilani Campus

Gauss Elimination and Back
Substitution

Triangular form:
Triangular means that all the nonzero entries of the
corresponding coefficient matrix lie above the diagonal and
form an upside-down 90° triangle. Then we can solve the
system by back substitution.
Since a linear system is completely determined by its
augmented matrix, Gauss elimination can be done by
merely considering the matrices.
(We do this again in the next example, emphasizing the matrices by writing
them first and the equations behind them, just as a help in order not to lose
track.)

BITS Pilani, Pilani Campus

Gauss Elimination
At the end of the Gauss elimination (before the back substitution), the row
echelon form of the augmented matrix will be

• in upper triangular form

• having the first r rows non-zero

• Exactly (m – r) rows would be zero rows

• the rhs would also have the last (m-r) rows zero

• any one of the (m-r) last rows in non-zero would imply inconsistency

• complexity is O(n3), where n is the number of rows

• facilitates the back substitution

BITS Pilani, Pilani Campus

Solution

The number of nonzero rows, r, in the row-reduced

coefficient matrix R is called the rank of R and also the
rank of A. Here is the method for determining whether
Ax = b has solutions and what they are:

(a) No solution. If r is less than m (meaning that R actually

has at least one row of all 0s) and at least one of the numbers
fr+1, fr+2, … , fm is not zero, then the system Rx = f is
inconsistent: No solution is possible. Therefore the system
Ax = b is inconsistent as well.

BITS Pilani, Pilani Campus

Solution

If the system is consistent (either r = m, or r < m and all the

numbers fr+1, fr+2, … , fm are zero), then there are solutions.

(b) Unique solution. If the system is consistent and r = n,

there is exactly one solution, which can be found by back
substitution.

(c) Infinitely many solutions. To obtain any of these

solutions, choose values of xr+1, … , xn arbitrarily. Then solve
the rth equation for xr (in terms of those arbitrary values),
then the (r − 1)st equation for xr−1, and so on up the line.

BITS Pilani, Pilani Campus

Gauss Elimination
 a11 a12 a13 b1 
a b2 
Solve Ax = b  21 a22 a23
 a31 a32 a33 b3 
Consists of two phases: Forward
Forward elimination  Elimination
Back substitution
a11 a12 a13 b1 
Forward Elimination 0 '
a22 '
a23 b2' 
reduces Ax = b to an 
 0 0 ''
a33 b3'' 
upper triangular system
Tx = b’ 
b3'' b2'  a23
'
x3
Back substitution can then x3  '' x2  '
Back
solve Tx = b’ for x a33 a22 Substitution
b1  a13 x3  a12 x2
x1 
a11
BITS Pilani, Pilani Campus
Pitfalls of Gauss Elimination

Division by zero
It is possible that during both elimination and back-substitution
phases a division by zero can occur.

For example:
2x2 + 3x3 = 8 0 2 3
4x1 + 6x2 + 7x3 = -3 A=4 6 7
2x1 + x2 + 6x3 = 5 2 1 6
a11 = 0
(the pivot element)
It is possible that during both elimination and back-substitution
phases
a division by zero can occur.
Solution: Pivoting

BITS Pilani, Pilani Campus

Pitfalls of Gauss Elimination

Round-off errors
Because computers carry only a limited number of significant figures,
round-off errors will occur and they will propagate from one iteration to
the next.

This problem is especially important when large numbers of equations

(100 or more) are to be solved.

Always use double-precision numbers/arithmetic. It is slow but

needed for correctness!

It is also a good idea to substitute your results back into the original
equations and check whether a substantial error has occurred.

BITS Pilani, Pilani Campus

Ill conditioned systems

• Systems where small changes in coefficients result in large change in

solution

x1 + 2x2 = 10
1.1x1 + 2x2 = 10.4  x1 = 4.0 & x2 = 3.0

x1 + 2x2 = 10
 x1 = 8.0 & x2 = 1.0
1.05x1 + 2x2 = 10.4

BITS Pilani, Pilani Campus

BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956
Condition Number
Condition number of a non singular matrix A is defined by
By convention, cond (A) = ∞ if A is singular

BITS Pilani, Pilani Campus

BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956
BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956
BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956
Techniques for Improving the
solution
Use of more significant figures – double precision
arithmetic
Pivoting
If a pivot element is zero, normalization step leads to division by
zero. The same problem may arise, when the pivot element is close to
zero. Problem can be avoided:
Partial pivoting
• Switching the rows below so that the largest element is the pivot
element.

Complete pivoting
• Searching for the largest element in all rows and columns then
switching.
• This is rarely used because switching columns changes the order of x’s
and adds significant complexity and overhead  costly

Scaling - used to reduce the round-off errors and improve accuracy

BITS Pilani, Pilani Campus

Partial Pivoting – Example

BITS Pilani, Pilani Campus

Partial Pivoting – Example

BITS Pilani, Pilani Campus

Partial Pivoting – Example

BITS Pilani, Pilani Campus

Gauss Elimination with
Rounding
0.0004x1 + 1.402x2 = 1.406
0.4003x1 – 1.502x2 = 2.501
Original solution of the system is x1 = 10, x2 = 1
Picking the first of given equation as pivot equation , we
have to multiply this equation by m = 0.4003/0.0004 = 1001
and subtract result from the second equation , obtaining
-1405x2 = -1404  x2 = 0.9993
From first equation we get x1 = 12.5
The failure occurs because |a11| is small compared to |a12|
so that a small round off error in x2 led to a large error in x1

BITS Pilani, Pilani Campus

Gauss Elimination with
Rounding
0.0004x1 + 1.402x2 = 1.406
0.4003x1 – 1.502x2 = 2.501
Picking the second of the given equations as the pivot
equation, we have to multiply this equation by
0.0004/0.4003 = 0.0009993 and subtract the result from the
first equation obtaining
1.404x2 = 1.404
x2 = 1 and x1 = 10
Note |a21| is not very small compared to |a22| so that a
small round off error in x2 would not lead to a large error in
x1
BITS Pilani, Pilani Campus
Operation Count – Gauss
Elimination
Important factors in judging the quality of a numerical
method are
- Amount of storage
- Amount of time (= number of operations)
Consider Augmented Matrix of Ax = b, where ain+1= bi

BITS Pilani, Pilani Campus

Operation Count – Gauss
Elimination
In elimination procedure to get the rank we will make all
elements below main diagonal zero.

Total number of multiplications and additions required

to determine the rank by elimination procedure are
n 1
2.  n  k   n  k  1  O(n3 )
k 1
Total number of divisions is
n- 1

å ( n - k ) =O(n )
2

k=1
BITS Pilani, Pilani Campus
BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956
BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956
BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956
BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956
Operation Count – Gauss
Elimination
In back substitution total number of additions, multiplications
and divisions required are
 n 
      2
2. n k n O ( n )
 k 1 

If an operation takes 10-9 sec, then

BITS Pilani, Pilani Campus

BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956
BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956
LU Factorization

We write square matrix A as

A = LU

Doolittle’s Method : L is lower triangular matrix

diag(L) = 1, lii = 1 and U is
upper triangular matrix

Crout’s Method : U is upper triangular matrix with

diag(U) = 1, uii = 1 and L is lower
triangular matrix

Cholesky’s Method: U = LT

BITS Pilani, Pilani Campus

Benefits of LU Decomposition

A = LU, Thus, the system Ax = B, is

LUx = B
Let Ux = y, then
Ly = B
Algorithm :-
Step-I Solve Ly = B, to find y.
Step-II Then solve Ux = y to find x

BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956

•Methods of LU Factorization
Doolittle Method: The Factors L, U are defined as

1 0 0 0 u11 u12 u13 u14 l ij  1, for i  j

l 0 u u24
1 0 0 22 u23 l ij  0, for i  j
L  21
U 
l 31 l 32 1 0 0 0 u33 u34
    u ij  0, for i  j
l 41 l 42 l 43 1 0 0 0 u44

Crout’s Method: The Factors L, U are defined as

l11 0 0 0 1 u12 u13 u14 l ij  0, for i  j

l  0 1 u u 
l 0 0
L   21 22  U  23 24
u ij  1, for i  j
l31 l32 l33 0  0 0 1 u34
    u ij  0, for i  j
l l l l
 41 42 43 44 0 0 0 1 

BITS Pilani, Deemed to be University under Section 3 of UGC Act, 1956