LINEAR ALGEBRA FOR DATA SCIENCE – ADDS 24.

1F (M03)
Lecture Note – 4

Diagonalization

Given an n x n matrix A, if we can find a linearly independent list of n eigen vectors, we may take these
eigen vectors as columns of a n x n matrix, say P. Because its columns form a linearly independent list,
this matrix P is invertible, so we may calculate the product matrix P-1AP. The result of this process will
always be a diagonal matrix. Furthermore, the entries on the main diagonal of this diagonal matrix are
precisely the eigen values of the original matrix A. This process is called as diagonalization.

Theorem

Suppose that P is a non singular 3 x 3 matrix whose columns are eigen vectors of the 3 x 3 matrix A with
eigen values λ, μ, υ (in order), then P-1AP = D or Ʌ ,where D or Ʌ is the diagonal matrix,

We say that A is diagonalizable and that P diagonalizes A. The diagonal matrix D is the diagonal form of
A and we say that A is similar to D.

Example

1) Find an invertible matrix P such that P-1AP is a diagonal matrix, where

A=

First we need to find all the eigen values of A. They are 1, 2, -1. Next, find the corresponding eigen
vectors X1, X2 , X3. Here, we need just any three particular eigen vectors say,

X1= X2 = X3 =
Other choices would be equally appropriate. Now form the matrix P with these vectors as columns

P=

Then P-1AP is a diagonal matrix. To see this in detail, first note that P is invertible because its columns
form a linearly independent list.

So P-1AP =

Exercise

1) In each case decide whether the given matrix A is diagonalizable and if so, find a matrix
P which diagonalizes A and write down the corresponding diagonal form.

A= , A=

Orthogonal Diagonalization Theorem

Suppose that A is a 3 X 3 matrix and that P is a 3 x 3 matrix whose columns c, d, e are mutually
perpendicular eigen vectors of A of unit length , then P is an orthogonal matrix which
diagonalizes A. Since PT = P-1 we can write the equation P-1AP = D in the form PTAP = D where,

D= and λ, μ and υ are the eigen values corresponding to c, d and e (in order)
Example

If A is the matrix

Find an orthogonal matrix P such that PTAP is the diagonal matrix. PT being the transpose of P.

The eigen values are 0, 3, 15. The corresponding eigen vectors are (a 2a 2a)’ (2b b -2b)’ (2c -2c
c)’ respectively.

The matrix formed by the three column vectors will be orthogonal if we take a = b = c = 1/3

Hence P = and PTAP =

Exercise

Find the orthogonal matrix P which diagonalizes A =

Powers of a matrix A

Consider D = P-1AP

Then D2 = (P-1AP) (P-1AP)

= P-1A(P P-1) AP = P-1A . IAP = P-1AAP = P-1A2P

An = PDnP-1

Similarly D3 = P-1A3P

Thus Dn = P-1An P
To obtain An pre multiply by P and post multiply by P-1,

PDnP-1 = P(P-1AnP)P-1 = (PP-1) An (PP-1)

= IAnI = An

Therefore, An = PDnP-1

 Note – when n = 1, A = PDP-1. This is known as eigen decomposition.

Exercise

1) Find a matrix P which diagonalizes the matrix

A= . Verify that P-1AP = D where D is the diagonal matrix. Hence find A6

2) Find A5 for A =

Quadratic Forms

Let A be a n x n symmetric matrix. Let Q(X) be of the form Q(X) = XTAX = ∑ ∑ .

Then Q(X) is said to be a quadratic form in the n variables X1, X2,…Xn.

Some examples for quadratic forms,

1) x2 + y2 is a quadratic form in two variables

2) x12 + 2x1x3 + 3x2x3 – x22 is a quadratic form in the three variables x1, x2, x3
3) 3x2 – 4xy + 2z2 is a quadratic form in x,y, z
Proof

Example

1) Find the symmetric 2 x2 matrix A which gives the quadratic form 3x2 – 2xy + 7y2

(x y)

A=

2) Expand

3x2 – y2 -2z2 + 3xz + 5yz

Exercise

1) Q(X) = X12 +6X1X2 – X22 – 4X2X3 + 2X32. Find the symmetric matrix A.
2) Find areal symmetric matrix C such that Q = XTCX where
Q = (x1 + x2 + x3)2

Orthonormal system of Vectors

A set of vectors X1 , X2 , …Xn are said to form an orthonormal system if

Xi T Xj = 1, if i = j

0, if i ≠ j

i.e vectors are mutually orthogonal and normalized.

Note – Column (row) vectors of an orthogonal matrix form an orthonormal system of vectors.

Singular Value Decomposition (SVD)

The Singular Value Decomposition (SVD) of a matrix is a factorization of that matrix into three matrices.

Let A be an M X N matrix. Let us suppose we can write A as the product,

A = USVT ,

Where the matrices U, S and V have the following properties.

1) The square matrix U has dimensions M X M which is the orthonormal eigenvectors of AAT (i.e
the columns of U must be normalized vectors of the matrix AAT)

2) The matrix S has dimensions M X N(the same as those of A) and is diagonal in the sense that Sij
= 0 if i ≠ j. We denote its diagonal elements by si for i = 1, 2,…p, where p = min (M, N); these
elements are termed as the singular values of A. Singular values of A are given by si = where
T T
are the eigen values of the smaller of AA and A A. Clearly, the are also some of the eigen
values of the larger of these two matrices, the remaining ones being equal to zero. (The singular
values are always non-negative, even though the eigenvalues may be negative)
 3) The square matrix V has dimensions N X N which is the orthonormal eigenvectors of ATA (i.e
the columns of V must be normalized vectors of the matrix ATA)

Example

Find the SVD for the matrix A =

The matrix A has dimension 2 x 3. Therefore, we may construct from it the 2 x 2 matrix AAT and 3 x 3
matrix ATA.

First we need to find the matrix U. For this we need to find the normalized eigen vectors of AAT

AAT =

The characteristic equation is,

│ AAT - λI│= 0

(17 – λ)2 – 64 = 0

Thus the eigen values are λ1 = 25, λ2 = 9

Since the singular values of A are given by si = and the matrix S has the same dimensions as A, we
have

S=

Where we have arranged the singular values in order of decreasing size. Now the matrix U has as its
columns the normalized eigen vectors of the 2 x 2 matrix AAT

When λ1 = 25, eigen vectors are

=
 2 2
To find Normalized eigen vector + =1

= 1/

When λ1 = 9, eigen vectors are

Similarly, the normalized eigen vector is

U=

Next we need to find matrix V. The columns of V are the normalized eigen vectors of 3 x 3 matrix ATA

ATA =

The eigen values are equal to that of AAT and the remaining eigen value is zero.

When λ1 = 25, eigen vectors are

=
The normalized eigen vector is

When λ2 = 9, eigen vectors are

The normalized eigen vector is

When λ3 = 0, eigen vectors are

The normalized eigen vector is

V=

A = USVT