Math Bootcamp AMPBA

Lecture 1
Aug 2021
Abhishek Rishabh
Today’s agenda
• About this course.

• How to get most out of this course?

• Student interaction – to set expectations and understanding

starting knowledge.

• Vectors and Matrices.

Introduction
• This course introduces you to linear algebra and matrix algebra.

• This will help you to understand the math behind most of the econometrics, machine and
deep learning models.

• Ideas/Concepts covered in these series of lectures are mathematical in nature.

• I will provide some applications/real world connections of these ideas.

How to get most out of this course?
• Ideas grasping through writing.

• Practice.

• Asking questions on LMS/email.

• Knowing that, this is a starting point. You might not be able to appreciate all the
ideas now. It’s okay, it takes time.
Student Interaction

• 20 mins session.

• What do students know? Covering extremes. (10-15 mins)

• R session. (5-10 mins)

Role of linear algebra in data science
• What do we do in data science?
• Estimation
• Prediction
• Inference
• How do we do it?
• Algorithms/techniques
• How do these techniques work?
• Some optimization
• A real example
What is a vector?
• A vector is collection of numbers.

• A vector x of order p (or dimension p) is a column of p numbers:

Vector
• We can refer x as column vector.

• The transpose of x, xT = (x1,x2,….,xp) is a row vector.

• Addition and subtraction of vectors of the same order is

performed element by element
Vector
• It is not possible to add or subtract vectors which are not
conformable, i.e., which do not have the same dimension or
order.

• Scalar multiplication of a vector is element by element:

Vector
• Two vectors x and y are equal if they are of the same order and
each pair of corresponding elements is equal, i.e., xi =yi

• A vector with all elements 0 is denoted by 0

• A vector ei with ith element 1 and all others 0 is the ith unit vector.

• A vector with all elements 1 is and is denoted by 1p

Vector
$
• Sum vector. XTlp ∑
= !"# 𝑥!

• x+y = y+x (commutativity) and (x+y)+z = x+(y+z) (associativity).

• Inner Product xTy

• Example: x=(1,2,3)T and y=(4,5,6)T. Find xTy

Vector
• Calculate xTx, where x is a p dimensional vector?
Summary
• Vector definition.

• Row v/s column vector.

• Vector dimension.

• Vector Operations.

• Types of vectors.

• Inner product.
Matrices

• An mxn matrix X is a rectangular array of scalar numbers:

• This matrix has m rows and n columns; it is an mn matrix (m by

n matrix), it is a matrix of order mn. X has dimensions m and n.
Matrices

• The numbers xij are the components (or elements) of X.

• A matrix is a square matrix if m = n, i.e.,if it has the same

number of rows and columns, i.e.,a nxn matrix is a square
matrix.

• The transpose of the mxn matrix X is the nxm matrix XT

Matrices

• A square matrix X is symmetric if XT = X. Two matrices X and Y

are equal if they are of the same order and each pair of
corresponding elements are equal, i.e.,xij =yij, for i = 1;2; … ;m
and j = 1;2; … ;n.
Matrices

• A square matrix with all elements not on the diagonal equal to 0

is a diagonal matrix, i.e.,if xij = 0 for all i!= j

• A diagonal matrix with all diagonal elements 1 (and all others

not on the diagonal 0) is denoted by In, called Identity Matrix.
Matrices

• The trace of a square matrix is the sum of all the diagonal

elements of the matrix, i.e.,trace (X) = tr(X) = tr(xij) =∑%!"# 𝑥!! .
Note that tr(In) = n.
Matrix Arithmetic
• X+Y = xij + yij

• X + Y = Y + X (commutativity) and (X + Y) + Z = X + (Y + Z)
(associativity).

• Scalar multiplication of a matrix is element by element:

Matrix Arithmetic
• X+Y = xij + yij

• X + Y = Y + X (commutativity) and (X + Y) + Z = X + (Y + Z)
(associativity).

• Scalar multiplication of a matrix is element by element:

Matrix Multiplication
• If A and B are matrices then we can multiply A by B (to get AB)
only if the number of columns of A equals the number of rows of
B.

• If C is mxn and D is pxq then the product CD can only be

defined if n = p, in which case C and D are said to be
conformable. If C and D are such that CD is not defined then
they are non-conformable.
Matrix Multiplication
• Example
Matrix Multiplication
Matrix Multiplication
• Now calculate VU.

• Is UV=VU ?
Matrix Multiplication
Matrix Multiplication
Matrix Multiplication
• Is the following true?

• (A+B)2=A2+B2+2AB

• If false, when will it be true?

Transpose and Trace of Sums and
Products
Symmetric and skew-symmetric matrices
Symmetric and skew-symmetric matrices
Symmetric and skew-symmetric matrices
• Are A and B symmetric?

• Is AB symmetric?
Orthogonal
• A pxp square matrix A is orthogonal if A’A = AA’ = Ip

• If A and B are both orthogonal and both pxp then AB is

orthogonal. Why?

• A’A=AA’=I ;; B’B=BB’=Ip;; (AB)’AB=AB(AB)’=B’A’AB=B’IpB=Ip

• An orthogonal matrix whose elements are all +1 or -1 is known

as a Hadamard matrix.
Orthogonal

• Are A and B orthogonal matrices?

Other names
• A pxp matrix is normal if AA’ = A’A
• A pxp matrix A is idempotent if A2 = A
• A matrix A != 0 is said to be nilpotent if A2 = 0
• A nxn matrix A such that A2 = In is said to be unipotent.

1 𝑥
• Example: A= , which of the above is A?
0 −1
R exercises
•
Linear Independence

• Two vectors x1, x2 are linearly independent, if ax1 + bx2 = 0

• a, b are real numbers
• Vectors x1, x2, x3…xn are linearly independent if ∑aixi = 0 implies ai = 0
• There are no non-trivial linear combinations which equals to 0
Linear Independence: Example
2 −4 1
• Are the vectors 2 , 6 , 0 linearly independent?
1 5 0

2 −4 1 0
• Independence if a1 2 + a2 6 + a3 0 = 0
1 5 0 0
If it has only a trivial solution a1 = a2 = a3 = 0

2a1-4a2+a3=0
2a1+6a2=0 (2) => a1=-3a2
a1+5a2=0 (3) => a1=-5a2
Rank of a matrix
• For a (m x n) matrix X,
• (x1, x2, x3 … xn) – column vectors of length ‘m’
• Similarly, it has ‘m’ row vectors with length ‘n’

• For a (m x n) matrix X,
• Rank of a matrix can be defined as maximum number of linearly independent
rows (or) maximum number of linearly independent columns, whichever is
lower

• Rank of the matrix X if m < n is ρ(X) ≤ m

Example 1: Rank of a matrix
• Find the Rank of the following matrix:

1 3 5
2 4 6
Example 1: Rank of a matrix (1/2)

1 3 5
X=
2 4 6

Rank of this matrix (2 x 3), ρ(X) ≤ 2

Þ a1 1 3 5 + a2 2 4 6 = 0 (Check for row vectors’ independence)

Þ a1 + 2a2 = 0 - eq(1)

Þ 3a1 + 4a2 = 0 - eq(2)

Þ 5a1 + 6a2 = 0 - eq(3)

 1 2 3
•X=4 5 6
5 7 9
Example 1: Rank of a matrix (2/2)

Þ eq(2) – 3 eq(1) gives a2 = 0

Þ Substituting a2 = 0 in eq(1) gives a1 = 0

Þ Hence, the row vectors are linearly independent.

Þ ρ(X) = 2
Example 2: Rank of a matrix
• Find the Rank of the following matrix:

4 6
6 9
Example 2: Rank of a matrix (1/2)

4 6
X=
6 9

Rank of this matrix (2 x 2), ρ(X) ≤ 2

Þ a1 4 6 + a2 6 9 = 0 (Check for row vectors’ independence)

Þ 4a1 + 6a2 = 0 - eq(1)

Þ 6a1 + 9a2 = 0 - eq(2)

Example 2: Rank of a matrix (2/2)

Þ Solving eq(1), eq(2) gives

Þ a1 = -3, a2 = 2 ( Not a trivial solution)

Þ Hence, the vectors are linearly dependent

Þ ρ(X) = 1
Example 3: Rank of a matrix
• Find the Rank of the following matrix:

1 2 3 2
4 5 6 −1
5 7 9 1
Example 3: Rank of a matrix (1/2)
A1+4a2+5a3=0
1 2 3 2 2a1+5a2+7a3=0
X = 4 5 6 −1 3a1+6a2+9a3=0
5 7 9 1 2a1-a2+a3=0
A1=1,a2=1,a3=-1
1 2 3 2
2 4 6 4
3 6 9 0

Rank of this matrix (3 x 4), ρ(X) ≤ 3

Þ But, 1 2 3 2 + 4 5 6 −1 = 5 7 9 1

Þ These rows are linearly dependent

Example 3: Rank of a matrix (1/2)
1 2 3 2
X = 4 5 6 −1
5 7 9 1

Þ Also, R1 and R2 are independent, i.e. cannot be expressed as

a real multiple of one another

Þ Hence, ρ(X) = 2

Can you find the Rank of XT (Transpose of X)?

Example 4: Rank of a matrix
• Find the Rank of the following matrix:

1 5 6
2 6 8
7 1 8
Example 4: Rank of a matrix (1/2)
1 5 6
X= 2 6 8
7 1 8
Rank of this matrix (3 x 3), ρ(X) ≤ 3 &

ρ(X) ≥ 2 (Column1 & Column2 are independent)

Þ a1 1 5 6 + a2 2 6 8 + a3 7 1 8 = 0

Þ a1 + 5a2 + 6a3= 0 - eq(1)

Þ 2a1 + 6a2 + 8a3= 0 - eq(2)

Þ 7a1 + 1a2 + 8a3= 0 - eq(3)

Example 4: Rank of a matrix (1/2)
Þ Using a1 = - 5a2 - 6a3 in eq(2) and eq(3)

Þ a2 + a3 = 0

Þ (a1, a2, a3) = (1, 1, -1) (a non-trivial solution)

Þ Hence, ρ(X) = 2
Full Rank
• A (m x n) Matrix X is said to be of full rank if
• ρ(X) = min(m, n)

2 6 10
X= is a 2x3 matrix and ρ(X) = 2
4 8 12

• X is a matrix of Full Rank

2 6 10
•X=
0 0 0
 1 0 0
•X= 0 1 0
0 0 1
Rank of a Diagonal Matrix
• Rank of a diagonal Matrix D
• Is equal to Number of non-zero diagonal elements in D. How?

3 0 0 0
2 0 0
0 5 0 0
X= ;Y = 0 7 0
0 0 0 0
0 0 9
0 0 0 8

1 0 0
I= 0 1 0 (𝑆𝑝𝑒𝑐𝑖𝑎𝑙 𝑐𝑎𝑠𝑒 𝑜𝑓 𝐷𝑖𝑎𝑔𝑜𝑛𝑎𝑙 𝑚𝑎𝑡𝑟𝑖𝑥 − 𝐼𝑑𝑒𝑛𝑡𝑖𝑡𝑦 𝑀𝑎𝑡𝑟𝑖𝑥)
0 0 1

Find Ranks of X, Y and I?

Rank of a Square Matrix

• In the case of square n x n Matrix Y is said to be full rank if

• det(Y) ≠ 0, i.e. ρ(Y) = n
• Why?

2 6 2 3
N= ;M=
1 3 1 5
Rank Inequalities: Sum
• ρ(X + Y) ≤ ρ(X) + ρ(Y)

• Consider ρ(X) = r, ρ(Y) = s

• X can be expressed as ∑raixi = 0, xi – independent columns of X

• Y can be expressed as ∑sbiyi = 0, yi – independent columns of Y

• (X + Y) can be written as linear combination of (r + s) vectors xi & yi

• Hence, ρ(X + Y) ≤ ρ(X) + ρ(Y)

Rank Inequalities : Difference

• Similarly, for ρ(X - Y)

• ρ(X +(-Y))

• But ρ(Y) = ρ(-Y)

Þ ρ(X + (-Y)) ≤ ρ(X) + ρ(-Y)

Þ ρ(X + (-Y)) ≤ ρ(X) + ρ(Y)

Rank Inequalities : Example(1/2)

2 6 2 3
•N= ;M=
1 3 1 5
• ρ(N) = 1, ρ(M) = 2

4 9
•M+N=
2 8
Rank Inequalities: Rank Products

• ρ(XY) ≤ min(ρ(X), ρ(Y))

Rank Inequalities : Example(1/2)

2 5 2 2
•N= ;M=
3 4 3 3
• ρ(N) = 2, ρ(M) = 1

19 19
•N*M=
18 18
Rank Inequalities : Example(2/2)

19 19
• Rank of N * M = is 1
18 18
• ρ(NM) ≤ min(ρ(N), ρ(M))

• 1 ≤ min(2, 1) = 1
Rank Inequalities
Product with orthogonal matrix:
• If C is an orthogonal matrix then ρ(AC) = ρ(A) because
• ρ(A) = ρ(ACCT) ≤ ρ(AC) ≤ ρ(A)

Submatrices:
• If Aij is a submatrix of A, then ρ(Aij) ≤ ρ(A)
Rank in Statistics
• Rank can be used as a crucial tool in Statistics in the form of
analysis
• In Simple linear Models, y = Xβ +ε
• In Multivariate analysis, as few statistical techniques depend on the X
having a full Rank
• More details will be on these will be covered as we move along in the
course
Summary on Rank
• Idea of linear independence
• What is the rank of a matrix?
• How to calculate it?
• Some rank inequalities
Determinant of a Matrix
• For every (n x n) Matrix A = (aij), det(A) can be calculated using
aij

• It is denoted as det(A) or │A│

Models use inverse of matrix <- inverse of matrix

uses idea of determinant for calculation
Determinant of a matrix – Why?
• To calculate the inverse of a square matrix

• To calculate the eigenanalyses of a matrix

• In multivariate analysis for transformations

Determinant of a matrix – How?
𝑎11 𝑎12
• In a (2 x 2) matrix, for A =
𝑎21 𝑎22

• det(A) = a11*a22 – a12*a21

Determinant of a matrix – Example
• Find the determinant of the following matrices

2 4
A=
1 −3

6 4
B=
9 6
Determinant of a matrix – Example
2 4
A=
1 −3
• det(A) = 2*(-3) – 1*4 = -10

6 4
B=
9 6
• det(B) = 6*6 – 4*9 = 0
Determinant of a matrix – How?

𝑎11 𝑎12 𝑎13

• In a (3 x 3) matrix, for A = 𝑎21 𝑎22 𝑎23
𝑎31 𝑎32 𝑎33

𝑎22 𝑎23 𝑎21 𝑎23 𝑎21 𝑎22

• det(A) = a11*│ │ − a12∗│ │+ a13*│ │
𝑎32 𝑎33 𝑎31 𝑎33 𝑎31 𝑎32
Determinant of a matrix – Example
• Find the determinant of the following matrices

2 2 4
A = 1 3 −3
4 6 7
Determinant of a matrix – Example
2 2 4
A = 1 3 −3
4 6 7
3 −3 1 −3 1 3
det(A) = 2*│ │ − 2∗│ │+ 4*│ │
6 7 4 7 4 6
= 2*(3*7 –(-3)*6) – 2*(1*7 – (-3)*4) + 4*(1*6 – 3*4)
= 2*(39) – 2*(19) + 4*(-6)
= 78 – 38 -24

=16
Co-factors in matrix
• Cofactor is a number that you get by eliminating the row and
column of a element and finding the determinant of the rest.

• Cofactor is always preceded by a ‘+’ or ‘-’ depending on the

position of the element
Co-factors in matrix
• In a (m x n) matrix A, the co-factor of the element aij

cij = (-1)i+j *det(Mij)

• Mij is the(m – 1 x n – 1) matrix formed from the rest of the

elements after removing the Row ‘i’, Column ‘j’

• Mij is called Minor of the element ‘aij’

Co-factor in matrix - Example Cofactor for a11

3 4
(-1)^(1+1) │ │
−2 −5
2 5 −1
A= 0 3 4
1 −2 −5
2−1
Cofactor of a32 = c32 = (-1)3+2 *det(M 32) = (-1) │
5 │
0 4
= -1*(8 – 0)

= -8
Adjoint or Adjugate matrix
• Adjoint matrix A is a square matrix formed by the transpose of
a co-factor matrix of A

• It is denoted as Adj A
Adjoint matrix - Example
Find the adjoint of the matrix A.

adjoint of the matrix A. = Transpose of cofactor matrix of A

3 1 −1
A = 2 −2 0
1 2 −1
Adjoint matrix - Example
Calculating co-factors of A

−2 0
Co-factor of a11 = 2
(-1) │ │=2
2 −1

2 0
Cofactor of a12 = (-1)3│ │=2
1 −1

2−2
Cofactor of a13 = (-1) │
4 │=6
1 2
Adjoint matrix - Example
Calculating co-factors of A

1−1
Co-factor of a21 = 3
(-1) │ │ = −1
2 −1

3 −1
Cofactor of a22 = (-1)4│ │ = -2
1 −1

3 1
Cofactor of a23 = (-1) │
5 │ = -5
1 2
Adjoint matrix - Example
Calculating co-factors of A

1 −1
Co-factor of a31 = (-1)4│ │ = −2
−2 0

3 −1
Cofactor of a32 = (-1) │
5 │ = -2
2 0

3 1
Cofactor of a33 = (-1) │
6 │ = -8
2 −2
Adjoint matrix - Example

2 2 6
Cofactor matrix of A, Cij= −1 −2 −5
−2 −2 −8

2 −1 −2
Adjoint of the matrix A, Adj A = (Cij)T = 2 −2 −2
6 −5 −8
Determinant of a (n x n) matrix
• Co-factors can be used to find the determinants of a (n x n)
matrix

• In a (n x n) matrix, for A, for 1 ≤ i ≤ n

• det(A) = ∑jn (aij*cij)= ai1*ci1 + ai2*ci2 + … + ain*cin

Properties of Determinants
𝑎11 𝑎12
• In a 2 x 2 matrix, A =
𝑎21 𝑎22
• If (a11, a21) and (a12, a22) are co-ordinates of point in a place,
det(A) is the area of the parallelogram by (a11, a21)’ and (a12, a22)’

• Similarly, what will det of the 3 x 3 matrix will represent?

Properties of Determinants
𝑎11 𝑎12
• In a 2 x 2 matrix, A =
𝑎21 𝑎22
• If (a11, a21) and (a12, a22) are co-ordinates of point in a place,
det(A) is the Area of the parallelogram by (a11, a21)’ and (a12, a22)’

• Similarly, what will det of the 3 x 3 matrix will represent?

• Volume of the parallelopiped formed by the ‘3’ columns of A
Properties of Determinants: Row &
Column Operations
• If we multiply a row or a column with a scalar ‘λ’, the
determinant is also multiplied by λ

2 4
A= , det(A) = -6 –(4) = -10
1 −3
Properties of Determinants: Row &
Column Operations
2 4
A= , det(A) = -6 –(4) = -10
1 −3
C1 is multiplied by scalar ‘2’ to form matrix B

4 4
B= , det(B) = -12 – 8 = -20 = 2*(-10)
2 −3
Properties of Determinants
• det(A) = det(AT)

5 2
A= , det(A) = -20 –(6) = -26
3 −4

5 3
AT = , det(AT) = -20 –(6) = -26
2 −4
Properties of Determinants
• If we multiply a n x n matrix A with scalar ‘λ’, the determinant of
the new matrix is λn *det(A)

3 5
A= , det(A) = 12 – 10 = 2
2 4

9 15
3*A = , det(AT) = 108 – 90 = 18 = 32*2 = 32*det(A)
6 12
Properties of Determinants
• If a complete row or column has zero as elements in matrix A,
det(A) = 0

0 5
A= , det(A) = 0 – 0 = 0
0 4
Properties of Determinants
• If A has two identical rows or columns, then det(A) = 0

3 3
A= , det(A) = 6 – 6 = 0
2 2
Properties of Determinants
• If A is n x n matrix and ρ(A) < n, then det(A) = 0

• Why?
Properties of Determinants
• If A is n x n matrix and ρ(A) < n, then det(A) = 0

• Why?

• A column or row can be written as a linear combination of

another column or row
Properties of Determinants
• A column or row can be written as a linear combination of
another column or row

2 4
A= , det(A) = 12 – 12 = 0
3 6
Properties of Determinants

2 4
A=
3 6

Rank of this matrix (2 x 2), ρ(X) ≤ 2

Þ a1 2 4 + a2 3 6 = 0 (Check for row vectors’ independence)

Þ 2a1 + 3a2 = 0 - eq(1)

Þ 4a1 + 6a2 = 0 - eq(2)

Properties of Determinants

Þ Solving eq(1), eq(2) gives

Þ a1 = -3, a2 = 2 ( Not a trivial solution)

Þ Hence, the vectors are linearly dependent

Þ ρ(X) = 1
Properties of Determinants
• Determinant of the diagonal matrix D with d1, d2, d3…dn

• det(D) = d1*d2*d3…*dn

𝑑1 0 0
D = 0 𝑑2 0
0 0 𝑑3
det(D) = d1*d2*d3
Properties of Determinants
• Determinant of the diagonal matrix D with d1, d2, d3…dn

• det(D) = d1*d2*d3…*dn

2 0 0
A= 0 3 0
0 0 7

3 0 0 0 0 3
det(A) = 2*│ │ − 0∗│ │+ 0*│ │
0 7 0 7 0 0

= 2*(3*7 –0*0) – 0*(0*7 – 0*0) + 0*(0*0 – 3*0)

= 2*3*7 = 42
Properties of Determinants
• Determinant of the triangular matrix T(upper or lower) with t1, t2,
t3…tn as the diagonal elements

• det(T) = t1*t2*t3…*tn

𝑡1 𝑎 𝑏
• T= 0 𝑡2 𝑐
0 0 𝑡3

det(T) = t1*t2*t3
Properties of Determinants
• Determinant of the triangular matrix T with t1, t2, t3…tn are the diagonal
elements

2 3 4
T= 0 3 5
0 0 7
3 5 0 5 0 3
det(A) = 2*│ │ − 3∗│ │+ 4*│ │
0 7 0 7 0 0
= 2*(3*7 –5*0) – 3*(0*7 – 0*5) + 4*(0*0 – 3*0)
= 2*3*7 = 42
Orthogonal Matrix
• A n x n matrix A is said to be orthogonal if
• AAT = In

• Since we already know that det(A) = det(AT) and det(In) = 1

• det(A) = +1 or -1
Block Matrix or Partitioned Matrix
• A block matrix is a matrix that is interpreted as having been
broken into sections – Blocks or Submatrices

• It can be visualised as a matrix broken into a collection of

Smaller Matrices
Block Matrix - Example

2 2 3 4
1 3 0 5
P=
8 1 3 4
4 6 5 7
• It can be broken into four (2 x 2) matrices
Block Matrix - Example
2 2 3 4
2 2 3 4
P11 = P12 = P=
1 3 0 5
1 3 0 5 8 1 3 4
4 6 5 7

8 1 3 4
P21 = P22 =
4 6 5 7

𝑃11 𝑃12
P=
𝑃21 𝑃22
Inverse of a matrix
• The inverse of a square matrix A, denoted as A-1 , such that
• AA-1 = I

• A square matrix has an inverse if det(A) ≠ 0

𝑎11 𝑎12 𝑏11 𝑏12 1 0

A= =
𝑎21 𝑎22 𝑏21 𝑏22 0 1
1 2 𝑏11 𝑏12 1 0
=
1 2 𝑏21 𝑏22 0 1
 𝑎11 𝑎12 𝑎11 𝑎21 1 0
• =
𝑎21 𝑎22 𝑎12 𝑎22 0 1
Inverse of a (2 x 2) matrix – How?
𝑎11 𝑎12
• In a (2 x 2) matrix, for A =
𝑎21 𝑎22
• det(A) = a11*a22 – a12*a21

𝑎22 −𝑎12
• Inverse of A, A-1 = 1/(det(A))*
−𝑎21 𝑎11
Inverse of a (2 x 2) matrix – Example
2 3
• Find the inverse of the matrix, A = ?
7 5
• det(A) = a11*a22 – a12*a21 = 2*5 – 3*7 = -11

𝑎22 −𝑎12
• Inverse of A, A-1 = 1/(det(A))*
−𝑎21 𝑎11

5 −3 −5/11 3/11
= 1/(-11) * =
−7 2 7/11 −2/11
Inverse of a (3 x 3) matrix – How?
𝑎11 𝑎12 𝑎13
• In a (3 x 3) matrix, for A = 𝑎21 𝑎22 𝑎23
𝑎31 𝑎32 𝑎33

• In general, for an n x n matrix,

Inverse of A, A-1 = 1/(det(A))* (𝐴𝑑𝑗𝑜𝑖𝑛𝑡 𝑀𝑎𝑡𝑟𝑖𝑥 𝑜𝑓 𝐴)

Inverse of a (3 x 3) matrix – Example
3 1 −1
• Find the inverse of the matrix, A = 2 −2 0 ?
1 2 −1
−2 0 2 0 2 −2
• det(A) = 3*│ │ − 1∗│ │+ (-1)*│ │
2 −1 1 −1 1 2
= 3(2 – 0) – 1(-2 – 0) -1(4 + 2) = 2
• Inverse of A, A-1 = 1/(det(A))* 𝑎𝑑𝑗 (𝐴)

5 −3 −5/11 3/11
= 1/(-11) * =
−7 2 7/11 −2/11
 3 1 −1
• 2 −2 0 ?
1 2 −1
• det(A) =2

• C11=2, c12=-1*(-2)=2,c13=6

• C21=-1, c22=-2, c23=-1*(5)

Inverse of a (3 x 3) matrix – Example

2 −1 −2
• Adj A = (Cij)T = 2 −2 −2
6 −5 −8
• Inverse of A, A-1 = 1/(det(A))* 𝑎𝑑𝑗 (𝐴)

2 −1 −2 1 −1/2 −1
A-1 = (½ )* 2 −2 −2 = 1 −1 −1
6 −5 −8 3 −5/2 −4
Inverse of a Matrix - Properties

• If Matrix A has an inverse B and also an inverse C, then

•B=C
Inverse of a Matrix - Properties
Proof:
• Given, AB = I & CA = I
• C = C*I
= C*(AB)
= (C*A)*B = I*B
=B
So every square matrix A has an unique Inverse
Inverse of a Matrix - Properties
• Determinant of an inverse A-1
• det(A-1) =1/ det(A)=det(A)-1

Proof:
• I = AA-1

• │I│= 1 = │AA-1│= │A││A-1│

• => │A-1│ = │A│-1
Inverse of a Matrix - Properties
• Inverse of the transpose matrix is equal to transpose of the
inverse matrix
• (AT)-1 = (A-1)T
• (AB) T

=T(B)*A(T)
Inverse of a Matrix - Properties
• If a matrix is orthogonal, then A-1 = AT

Proof:
• AAT = ATA = I (By definition of orthogonal Matrix)

• Also, AA-1 = I (By definition of Inverse Matrix)

• This implies,
• AT = A-1 in case of an orthogonal matrix
Inverse of a Matrix - Properties
• Inverse of Product of two square matrices A, B
• (AB)-1 = B-1A-1
Inverse of a Matrix - Properties
Proof:
• (AB)B-1A-1 = AIA-1 = AA-1 = I

• Similarly, B-1A-1(AB) = B-1IB = B-1B = I

• Hence,
• (AB)-1 = B-1A-1
Applications of Inverse of a matrix
• Consider a system of linear equations, y = Ax

• There exists a unique solution for x = A-1y, only if

• A is inversible i.e. A is a non-singular matrix
 Eigen Values
• Why should we learn this?
• Let's assume the Web contains 6 pages only. The author of Page 1 thinks pages 2, 4, 5,
and 6 have good content, and links to them. The author of Page 2 only likes pages 3 and
4 so only links from her page to them.

https://www.intmath.com/matrices-determinants/8-applications-eigenvalues-eigenvectors.php
 Eigen Values
• Their task was to find the "most important" page for a particular search query. For
example, if everyone linked to Page 1, and it was the only one that had 5 incoming links,
then it would be easy - Page 1 would be returned at the top of the search result.
• However, we can see some pages in our web are not regarded as very important. For
example, Page 3 has only one incoming link. Should its outgoing link (to Page 5) be worth
the same as Page 1's outgoing link to Page 5?
• The beauty of PageRank was that it regarded pages with many incoming links (especially
from other popular pages) as more important than those from mediocre pages, and it
gave more weighting to the outgoing links of important pages.
• Eigenvector = PageRank

https://www.intmath.com/matrices-determinants/8-applications-eigenvalues-eigenvectors.php
Eigen Values
• If S is an n x n matrix, then the eigen values of S are roots of
characteristic equation S,
• │S - λIn│ = 0
Eigen Values - Example
1 4
S= , for eigen values │S - λIn│ = 0
9 1
1−λ 4
=>│ │=0
9 1 −λ
=> (1 – λ)2 – 36 = 0

=> λ2 –2 λ - 35 = 0

=> (λ -7) (λ+5) = 0 => λ = 7, -5

Eigen Vectors
Ax = Sx – λx = 0

=> Sx = λx (Eigen Equation)

• x is called the Eigen Vector

• λ is called the Eigen Value

• The pair(x, λ) is called Eigen Pair

• For each value of λ, you get a corresponding eigen vector

Eigen Vector - Example
1 4
• For the previous example, S = , find the Eigen Vectors?
9 1
Eigen Vector - Example
1 4
•S= , λ = 7, -5
9 1

1 4 𝑥11 𝑥11
• For λ = 7, = 7*
9 1 𝑥12 𝑥12

=> x11 + 4x12 = 7x11

Þ9x11 + x12 = 7x12

Þ x11 = 2/3 & x12 = 1

Eigen Vector - Example
1 4 𝑥11 𝑥11
• For λ = -5, = -5*
9 1 𝑥12 𝑥12

=> x11 + 4x12 = -5x11

Þ9x11 + x12 = -5x12

Þ x11 = -2/3 & x12 = 1

Eigen Vector - Example
1 4
• For S =
9 1
• The Eigen values = 7, -5

2/3 −2/3
• The Eigen vectors = ,
1 1
Properties of Eigenanalyses
• If x is an eigen vector of S, any scalar multiple kx is also an
eigen vector
• S(kx) = λkx
Properties of Eigenanalyses
• If λi, λj are the distinct eigen values of S with eigen vectors xi, xj
• Then xi, xj are distinct vectors
Properties of Eigenanalyses
Proof:

• Say xi = xj = x, we have Sx = λix & Sx = λjx

=> (λi – λj)x = 0

• Since λi ≠ λj ; x = 0 contradicts x being a eigen vector

• Hence, xi ≠ xj
Properties of Eigenanalyses
• If xi, xj are the distinct eigenvectors of S with same eigen value λ,

linear combination of xi and xj is also an eigen vector of S

Properties of Eigenanalyses
Proof:

• Since, Sxi = λxi & Sxj = λxj ; Consider a, b are real scalars

=> Saxi = λaxi (Multiply with a) & Sbxj = λbxj (Multiply with b)

• Adding both equations,

• S(axi + axj)= λ(axi + axj)

Properties of Eigenanalyses
• If S is a symmetric matrix, eigen vectors corresponding to
distinct eigen values are orthogonal, i.e. xi‘xj = 0
• Proof?
Properties of Eigenanalyses
2 1
•S= , gives us, λ = 1, 3
1 2
2 1 𝑥11 𝑥11
• For λ = 1, = 1*
1 2 𝑥12 𝑥12
=> 2x11 + 1x12 = x11

Þ1x11 + 2x12 = x12

Þ x11 = -1 & x12 = 1

Properties of Eigenanalyses
2 1 𝑥11 𝑥11
• For λ = 3, = 3*
1 2 𝑥12 𝑥12
=> 2x11 + 1x12 = 3x11

Þ1x11 + 2x12 = 3x12

Þ x11 = 1 & x12 = 1

1
Þ −1 1 = [ -1 + 1 ] = [ 0 ]
1
Properties of Eigenanalyses
• If x and λ are eigen pair of a non-singular matrix S, then
• x is an eigen vector of S-1 with λ-1 as eigen value
Properties of Eigenanalyses
Proof:

• We have Sx = λx

Multiplying both sides with S-1

S-1Sx = S-1λx => λ-1x = S-1x

=> S-1x = λ-1x

• Hence, x is an eigen vector of S-1 with λ-1 as eigen value

Properties of Eigenanalyses - Example
Find the Eigen values and vectors for S & S-1 , where

1 1
S=
2 3
Properties of Eigenanalyses - Example
1 1
For S= , for eigen values │S - λIn│ = 0
2 3
1−λ 1
=>│ │=0
2 3 −λ
=> (3 – λ)(1 – λ) – 2 = 0

=> λ2 –4λ +1 = 0

=> λ = 2 + √3, 2 - √3
Properties of Eigenanalyses - Example
1 1
•S= , λ = 2 + √3, 2 - √3
2 3

1 1 𝑥11 𝑥11
• For λ = 2+ √3 , = (2 + √3)*
2 3 𝑥12 𝑥12

=> x11 + x12 = (2+ √3)x11

Þ2x11 + 3x12 = (2+ √3)x12

Þ x11 = (√3 – 1)/2 & x12 = 1

Properties of Eigenanalyses - Example
1 1
•S= , λ = 2 + √3, 2 - √3
2 3

1 1 𝑥11 𝑥11
• For λ = 2- √3 , = (2 - √3)*
2 3 𝑥12 𝑥12

=> x11 + x12 = (2 - √3)x11

Þ2x11 + 3x12 = (2 - √3)x12

Þ x11 = -(√3 + 1)/2 & x12 = 1

Properties of Eigenanalyses - Example
1 1
• For S =
2 3
• The Eigen values = 2 + √3, 2 - √3

• The Eigen vectors = −(√3 + 1)/2 , (√3 −1)/2

1 1
Properties of Eigenanalyses - Example
1 1
• For S =
2 3
Finding λ-1

1. 1/(2+ √3) = (2 - √3)/((2+ √3)*(2- √3))

= (2 - √3)/(22 – (√3)2) = (2 - √3)

2. 1/(2- √3) = (2 + √3)/((2+ √3)*(2- √3))

= (2 + √3)/(22 – (√3)2) = (2 + √3)

Properties of Eigenanalyses - Example
1 1
• For S =
2 3
• det(S) = 1*3 – 1*2 = 1
𝑠22 −𝑠12
• Inverse of S, S-1 = 1/(det(S))*
−𝑠21 𝑠11
3 −1
S-1 =
−2 1
Properties of Eigenanalyses - Example
3 −1
For S-1 = , for eigen values │S - λIn│ = 0
−2 1
3−λ −1
=>│ │=0
−2 1 −λ
=> (3 – λ)(1 – λ) – 2 = 0

=> λ2 –4λ +1 = 0

=> λ = 2 + √3, 2 - √3
Properties of Eigenanalyses - Example
3 −1
• S-1 = , λ = 2 - √3, 2 + √3
−2 1

3 −1 𝑥11 𝑥11
• For λ = 2+ √3 , = (2 + √3)*
−2 1 𝑥12 𝑥12

=> 3x11 - x12 = (2+ √3)x11

Þ-2x11 + x12 = (2+ √3)x12

Þ x11 = -(√3 + 1)/2 & x12 = 1

Properties of Eigenanalyses - Example
3 −1
• S-1 = , λ = 2 + √3, 2 - √3
−2 1

3 −1 𝑥11 𝑥11
• For λ = 2- √3 , = (2 - √3)*
−2 1 𝑥12 𝑥12

=> 3x11 - x12 = (2 - √3)x11

Þ-2x11 + x12 = (2 - √3)x12

Þ x11 = (√3 - 1)/2 & x12 = 1

Properties of Eigenanalyses - Example
3 −1
• For S-1 =
−2 1
• The Eigen values = 2 + √3, 2 - √3

• The Eigen vectors = −(√3 + 1)/2 , (√3 −1)/2

1 1
• Clearly, S-1 has eigen value of λ-1 with x as eigen vector
Properties of Eigenanalyses
• If x and λ are eigen pair of a non-singular matrix S, then
• x is an eigen vector of Sk with λk as eigen value
Properties of Eigenanalyses
Proof:

• We have Sx = λx

Multiplying both sides with Sk-1

Skx = Sk-1(Sx) = Sk-1(λx) = Sk-2(Sλx) = Sk-2(λ2x) … = λkx

• Hence, x is an eigen vector of Sk with λk as eigen value

 Properties of Eigenanalyses - Example
Find the Eigen values and vectors for S & S2 , where

1 1
S=
2 3
3 4
S2 =
8 11

(3- λ)(11- λ)-32=0

What is our guess/conclusion; 33-14 λ+ λ2—32=0;
Eigen values for S are λ = 2 + √3, 2 - √3 1+14 λ+ λ2 =0
Eigen values for S2 should be square of λ = 2 + √3, 2 -
√3
Λ1=7+4 √3, λ2=7-4 √3
Properties of Eigenanalyses - Example
As showed in the previous example,

• The Eigen values = 2 + √3, 2 - √3

• The Eigen vectors = −(√3 + 1)/2 , (√3 −1)/2

1 1
Properties of Eigenanalyses - Example
1 1
• For S =
2 3
Finding λ2

1. (2+ √3)2 = (22 + (√3)2 + 2*2*√3)

= 7 + 4√3

2. (2- √3)2 = (22 + (√3)2 - 2*2*√3)

= 7 - 4√3
Properties of Eigenanalyses - Example
1 1 1 1 1+2 1+3 3 4
S2= ∗ = = ,
2 3 2 3 2+6 2+9 8 11
Properties of Eigenanalyses - Example
3 4
For S2= , for eigen values │S - λIn│ = 0
8 11
3−λ 4
=>│ │=0
8 11 − λ
=> (3 – λ)(11 – λ) – 32 = 0

=> λ2 –14λ +1 = 0

=> λ = 7 + 4√3, 7 - 4√3

Properties of Eigenanalyses - Example
3 4
• S2 = , λ = 7 + 4√3, 7 - 4√3
8 11

3 4 𝑥11 𝑥11
• For λ = 7 + 4√3 , = (7 + 4√3)*
8 11 𝑥12 𝑥12

=> 3x11 + 4x12 = (7 + 4√3)x11

Þ 8x11 + 11x12 = (7 + 4√3)x12

Þ x11 = (√3 - 1)/2 & x12 = 1

Properties of Eigenanalyses - Example
3 4
• S2 = , λ = 7 + 4√3, 7 - 4√3
8 11

3 4 𝑥11 𝑥11
• For λ = 7 - 4√3 , = (7 - 4√3)*
8 11 𝑥12 𝑥12

=> 3x11 + 4x12 = (7 - 4√3)x11

Þ 8x11 + 11x12 = (7 - 4√3)x12

Þ x11 = -(√3 + 1)/2 & x12 = 1

Properties of Eigenanalyses - Example
For S2

• The Eigen values = 7 + 4√3, 7 - 4√3

• The Eigen vectors = −(√3 + 1)/2 , (√3 −1)/2

1 1

• Clearly, S2 has eigen value of λ2 with x as eigen vector

Properties of Eigenanalyses - Example
1 1
• Let’s take S = , b = 1, a = 2
2 3

1 0 1 1
• This implies, S* = aIn – bS = 2* - 1*
0 1 2 3

1 −1
S* =
−2 −1
Properties of Eigenanalyses - Example
1 −1
For S*= , for eigen values │S - λIn│ = 0
−2 −1
1−λ −1
=>│ │=0
−2 −1 − λ
=> (-1 – λ)(1 – λ) –2 = 0

=> λ2 – 3 = 0

=> λ = √3, -√3

Properties of Eigenanalyses - Example
1 −1
• S*= , λ = √3, -√3
−2 −1

1 −1 𝑥11 𝑥11
• For λ = √3 , = (√3)*
−2 −1 𝑥12 𝑥12

=> x11 - x12 = (√3)x11

Þ -2x11 - x12 = ( √3)x12

Þ x11 = -(√3 + 1)/2 & x12 = 1

Properties of Eigenanalyses - Example
1 −1
• S*= , λ = √3, -√3
−2 −1

1 −1 𝑥11 𝑥11
• For λ = -√3 , = (-√3)*
−2 −1 𝑥12 𝑥12

=> x11 - x12 = (-√3)x11

Þ -2x11 - x12 = (-√3)x12

Þ x11 = (√3 - 1)/2 & x12 = 1

Properties of Eigenanalyses - Example
1 −1
• For S* =
−2 −1
• The Eigen values = √3, -√3

• The Eigen vectors = −(√3 + 1)/2 , (√3 −1)/2

1 1
• Clearly, S* has eigen value of (a - bλ) with x as eigen vector
Properties of Eigenanalyses
• If S is a symmetric matrix, there exists an orthogonal matrix T
and diagonal matrix D, such that
• T’ST = D

• Rank of a Symmetric matrix (S) is equal to Number of non-zero

eigen values
Properties of Eigenanalyses
1 2 1 −1
• Let us take an example of S = , T = (1/√2)*
2 1 1 1

1 1
• T’ = (1/√2)* ( Check if TT’ = I)
−1 1
Properties of Eigenanalyses
1 2 1 −1
• Let us take an example of S = , T = (1/√2)*
2 1 1 1
1 1 1 2 1 −1
• T’ST = (1/√2)* * * (1/√2)*
−1 1 2 1 1 1
1+2 2+1 1 −1 3 3 1 −1
= (1/2)* * = * (1/2)*
−1 + 2 −2 + 1 1 1 1 −1 1 1
3 + 3 −3 + 3 𝟑 𝟎
= (1/2)* = =D
1 − 1 −1 − 1 𝟎 −𝟏
Properties of Eigenanalyses - Example
1 2
For S= , for eigen values │S - λIn│ = 0
2 1
1−λ 2
=>│ │=0
2 1 −λ
=> (1 – λ)(1 – λ) – 4 = 0

=> λ2 –2λ -3 = 0

=> λ = -1, 3 (Two non-zero λ)

Properties of Eigenanalyses - Example
1 2
Rank of a square matrix, when det S = ≠ 0 (-3, here) is
2 1
always n, i.e. 2

Hence, Rank of a Symmetric matrix (S) is equal to Number of

non-zero eigen values if T’ST = D
Properties of Eigenanalyses
• If D is a diagonal or triangular matrix, then the eigen values are
the diagonal elements
Properties – Eigenanalyses
3 1
S= , for eigen values │S - λIn│ = 0
0 −3
3−λ 1
=>│ │=0
0 −3 − λ
=> (3 – λ) (3 + λ) – 0 = 0

=> (λ -3) (λ+3) = 0 => λ = 3, -3

The diagonal elements are the eigen values of triangular matrix

Positive Definite Matrices
• A matrix A is said to be positive definite if all the eigen values of
the matrix are positive.

2 −1
A=
−1 2
Properties – Positive Definite Matrix
2 −1
A= , for eigen values │S - λIn│ = 0
−1 2
2−λ −1
=>│ │=0
−1 2−λ
=> (2 – λ)2 – 1 = 0

=> λ2 - 4λ + 3 = 0 => λ = 3, 1

The matrix A is a positive definite matrix

Positive Definite Matrices
• A matrix A is said to be positive definite if all the eigen values of
the matrix are positive.

2 −1
A=
−1 2
Matrix Decomposition
• A matrix decomposition is way of reducing a matrix into its
constituent parts

• It is to simplify complex matrices into simple matrices to perform

operations on decomposed matrices

• It is called matrix factorization

LU(Lower Upper Matrix Decomposition)

• There are several ways to factorize a matrix

• LU decomposition is factorization of a matrix S (n x n) into two

triangular matrices,
• One Lower Triangular Matrix, L
• One Upper Triangular Matrix, U
LU(Lower Upper Matrix Decomposition)

𝑎11 𝑎12 𝑎13

S = 𝑎21 𝑎22 𝑎23
𝑎31 𝑎32 𝑎33

1 0 0
L = 𝑙21 1 0
𝑙31 𝑙32 1
𝑢11 𝑢12 𝑢13
U= 0 𝑢22 𝑢23
0 0 𝑢33
Does [L][U] = [A]?

é 1 0 0ù é25 5 1 ù
ê ú ê ú
[L][U ] = ê2.56 1 0ú ê 0 - 4.8 - 1.56ú = ?
êë5.76 3.5 1úû êë 0 0 0.7 úû
• X-y+z=2
• X+2y+2z=3
• X-y-z=1 1 −1 1
1 2 2
1 −1 −1

A=LU => inv(A)=inv(LU)=inv(U)*inv(L)

 Using LU Decomposition to solve SLEs
Solve the following set of linear equations using LU Decomposition

é 25 5 1ù é x1 ù é106.8 ù
ê 64 8 1ú ê x ú = ê177.2 ú
ê ú ê 2ú ê ú
êë144 12 1úû êë x3 úû êë279.2úû

Using the procedure for finding the [L] and [U] matrices
é 1 0 0ù é25 5 1 ù
[A] = [L][U ] = êê2.56 1 0úú êê 0 - 4.8 - 1.56úú
êë5.76 3.5 1úû êë 0 0 0.7 úû
Spectral Matrix Decomposition
• If S is a symmetric matrix with eigen values, λ1 ≥ λ2 ≥ λ3 …≥ λn

and eigen vectors (x1, x2, x3, …, xn),

We have S = TΛT’

It is called, Spectral Decomposition with T = (x1, x2, x3, …, xn)

Singular Value Decomposition (SVD)
• Handy mathematical technique that has application to many
problems

• Given any m´n matrix A, algorithm to find matrices U, V, and W

such that
A = U W VT
U is m´n and orthonormal
W is n´n and diagonal
V is n´n and orthonormal
SVD
SVD and Inverses
• Why is SVD so useful?

• Application #1: inverses

• A-1=(VT)-1 W-1 U-1 = V W-1 UT

• Using fact that inverse = transpose
for orthogonal matrices

• Since W is diagonal, W-1 also diagonal with reciprocals of entries of W

Applications
• Linear Regression
• What is it?
• Example in excel
• Sales=f(inventory, time, price ….)
• Profit=f(launch date of product, interest rate, …)
• Churn=f(usage, did you buy other prodcuts, price)
Linear Regression
Linear Regression
• Minimize sum of squares
Linear Regression
• Take Derivative and set it to 0.
Linear Regression
Linear Regression