DAMA50 - MATHEMATICS FOR MACHINE LEARNING

Unit1: Linear Algebra

1rst tutorial meeting notes (2023-24)

 Introduction

In machine learning we utilise vectors and matrices to represent data.

Linear algebra is a branch of mathematics concerning linear equations
and linear maps and their representations as vectors and matrices. It
provides us with the tools for studying matrices and vectors. In this
sense, Lineal Algebra is central to machine learning.
Systems of linear equations
Solve the following system of equations with respect to 𝑥𝑥, 𝑦𝑦

1) Graphical solution 2) Method of elimination: Multiply first

equation by 3 and subtract

Substituting back to the second equation we get

Systems of linear equations
A linear equation in the variables 𝑥𝑥1 ,𝑥𝑥2 ,… , 𝑥𝑥𝑛𝑛 , is an equation of the form

with 𝑎𝑎1 ,𝑎𝑎2 ,… , 𝑎𝑎𝑛𝑛 ∈ 𝐹𝐹 , where 𝐹𝐹 is a field which we usually take to be either ℝ (real
numbers) or ℂ (complex numbers).
In general, a linear system in the variables 𝑥𝑥1 ,𝑥𝑥2 ,… , 𝑥𝑥𝑛𝑛 , is a set of linear
equations usually written as

A linear system is called consistent if it has at least one solution. It is called inconsistent
if it is not consistent, i.e., it has no solution.
A linear system is called homogeneous if 𝑏𝑏1 = 𝑏𝑏2 = ⋯ = 𝑏𝑏𝑛𝑛 = 0. A homogeneous
system is always consistent (can you see why?)
Gaussian Elimination
Solving Systems of Linear Equations using Gaussian Elimination
Vectors

Vectors are fundamental objects of Linear Algebra. A vector space is a set of objects called
vectors, which may be added together and multiplied ("scaled") by numbers called scalars.
Scalars are often real numbers. From the mathematical viewpoint, any object that satisfies
these two properties can be considered as a vector.
Examples:
a) Geometric vectors b)Polynomials
Manipulating vectors

1) Vector addition:
Head to tail, at the end of 𝜐𝜐 place the start of 𝑤𝑤

2) Scalar multiplication

Linear combination of vectors

Matrices
Matrices
Matrices


Matrix Addition

A+B=B+A
Matrices


Matrix Multiplication

AB≠BA
matrix multiplication is not commutative
Identity Matrix
Inverse of a matrix

Consider a square matrix A, if there is a matrix B so that

AB = I = BA

then B is called the inverse of A and denoted by A-1.

Not every matrix A possesses an inverse A−1.

Inverse of a matrix


Inverse of a 2x2 matrix
Inverse of a matrix


Inverse of a 2x2 matrix
Inverse of a matrix


Inverse of a 2x2 matrix

A matrix is invertible if and only if its determinant is non-zero.

Determinants

Determinant of a 2x2 matrix

Determinant of a 3x3 matrix

 Matrices


Multiplication by a Scalar
Multiply a matrix A by a scalar λ
λ A = K, where Kij =λ Aij.


Transpose Matrix
For a matrix A, the matrix B with
bij = aji is called the transpose of A.
We write B = AT.
 Transpose of a matrix

The transpose of a matrix A (denoted by ) is obtained by interchanging the rows and columns of A.

Example:

A matrix B is called symmetric if

Example:
Matrix Inverse

The inverse of a square matrix denoted by 𝐴𝐴−1 is defined as follows

Provided det 𝐴𝐴 ≠0 we can calculate the inverse of a matrix using the
following formula

where the adjugate matrix is defined as follows:

Example
For a 2x2 matrix

For a 3x3 matrix A, we first calculate the cofactor matrix C.

The adjugate matrix is the transpose of C
Back to systems of Linear Equations
Linear systems - Matrices
Systems of linear equations in matrix form


Why do we need the inverse of a square matrix?

If a system of linear equations is described by a square matrix A,

Ax = b

Then by applying the inverse of A in both sides we get:

A-1A x = A-1b
Where, A-1A ≡ Ι, thus:
x = A-1b

Because we can use it to solve the system of linear equations.

 Gaussian elimination

An upper triangular system can be solved by back substitution starting at the bottom.
Solution using the augmented matrix
Linear system Augmented matrix
Linear independence


Linearly (In)dependent Vectors
Linear independence


Linearly (In)dependent Vectors (Example)
Linear independence


Linearly (In)dependent Vectors (Example)
 Basis in vector space - Rank
The span of a set of vectors 𝑣𝑣1 , 𝑣𝑣2 , ⋯ , 𝑣𝑣𝑛𝑛 denoted by span 𝑣𝑣1 , 𝑣𝑣2 , ⋯ , 𝑣𝑣𝑛𝑛 is the set
of all linear combinations of the vectors 𝑣𝑣1 , 𝑣𝑣2 , ⋯ , 𝑣𝑣𝑛𝑛 .

Basis
Consider a vector space V and a generating set A of V is called minimal if there
exists no smaller set A that spans V. Every linearly independent generating set of V
is minimal and is called a basis of V.


Rank
The number of linearly independent columns of a matrix A equals the number of
linearly independent rows and is called the rank of A and is denoted by rank(A).
Vector space basis


Basis (Example)
Vector space basis
Span (Example)

The span of with real numbers as scalars is

Rank (Example)
Row echelon form
A matrix is in row echelon form if
(1) All rows that contain only zeros are at the bottom of the matrix; correspondingly,
all rows that contain at least one nonzero element are on top of rows that contain
only zeros.
(2) Looking at nonzero rows only, the first nonzero number from the left pivot (also
called the pivot or the leading coefficient) is always strictly to the right of the pivot
of the row above it.
Reduced row echelon form
A matrix is in reduced row echelon form if it satisfies the following properties
(1) It is already in row echelon form.
(2) The leading entry in each nonzero row is a 1 (called a leading 1).
(3) Each column containing a leading 1 has zeros everywhere else.

Unlike row echelon form, the reduced row echelon form is unique. That is for each matrix
𝐴𝐴 there is only one matrix 𝑅𝑅 in reduced row echelon form that is equivalent to 𝐴𝐴.

Gaussian Elimination helps to put a matrix in row echelon form, while Gauss-Jordan
Elimination puts a matrix in reduced row echelon form.

The rank of a matrix is the number of nonzero rows in its row echelon form.
Reduced row echelon form (examples)

No. The left-most nonzero entry in the second row not equal to 1

No. The leading 1 in the second row is to the left of the leading 1
in the first row

No. The leading 1 in the third row is not the only nonzero
in the column that contains it

√ Yes. √ Yes.
 Matrices as Linear Maps
A matrix can be seen as a map on vectors. To each vector

we associate a vector

We obtain a map
Example: The map associated with the matrix
Image and Kernel

Image and Kernel of a transformation
Image and Kernel