Mathematics for Data Science

Dr. Indu Joshi

Assistant Professor at
Indian Institute of Technology Mandi

11 October 2024
 Motivation

Vector spaces and subspaces are useful in machine learning for the
following reasons:
• Geometric Interpretation: Vector spaces provide a
geometric framework to understand and visualize data,
making it easier to comprehend relationships and patterns
within the data.
• Clustering and Classification: Subspaces help in defining
decision boundaries for clustering and classification
algorithms, improving the accuracy of these models.
• Noise Reduction: By projecting data onto relevant
subspaces, noise can be filtered out, leading to cleaner
datasets and more robust models.
 Motivation (Contd.)

• Algorithm Efficiency: Operations in vector spaces, such as

projections and decompositions, are computationally efficient
and enhance the performance of machine learning algorithms.
• Similarity Measurement: Vector spaces enable the
calculation of distances and angles between data points,
which is fundamental for measuring similarity and making
predictions in algorithms like k-nearest neighbors (KNN).
 Field

• A field is a set F , on which two operations + and · (called

addition and multiplication, respectively) are defined so that
for each pair of elements x, y in F , there are unique elements
x + y and x · y (often written xy ) in F for which the following
conditions hold for all elements x, y , z in F :
• ∀a, b ∈ F , both a + b ∈ F and ab ∈ F (closure of F under
addition and multiplication)
• x + y = y + x (commutativity of addition)
• (x + y ) + z = x + (y + z) (associativity of addition)
• There is an element 0 ∈ F , called zero, such that x + 0 = x.
(existence of an additive identity)
• For each x, there is an element −x ∈ F such that
x + (−x) = 0. (existence of additive inverses)
 Field (contd.)

• x · y = y · x (commutativity of multiplication)
• (x · y ) · z = x · (y · z) (associativity of multiplication)
• (x + y ) · z = x · z + y · z and x · (y + z) = x · y + x · z
(distributivity)
• There is an element 1 ∈ F , such that 1 ̸= 0 and x · 1 = x.
(existence of a multiplicative identity)
• If x ̸= 0, then there is an element x −1 ∈ F such that
x · x −1 = 1. (existence of multiplicative inverses)
• Examples of fields include:
• The field of real numbers R
• The field of complex numbers C
• The field of rational numbers Q
 Field of Real Numbers R

• Closure
• Addition: If a, b ∈ R, then a + b ∈ R.
• Multiplication: If a, b ∈ R, then a · b ∈ R.
• Associativity
• Addition: For any a, b, c ∈ R, (a + b) + c = a + (b + c).
• Multiplication: For any a, b, c ∈ R, (a · b) · c = a · (b · c).
• Commutativity
• Addition: For any a, b ∈ R, a + b = b + a.
• Multiplication: For any a, b ∈ R, a · b = b · a.
• Identity Elements
• Additive Identity: There exists 0 ∈ R such that for any
a ∈ R, a + 0 = a.
• Multiplicative Identity: There exists 1 ∈ R such that for any
a ∈ R, a · 1 = a.
 Field of Real Numbers R

• Inverses
• Additive Inverse: For any a ∈ R, there exists −a ∈ R such
that a + (−a) = 0.
• Multiplicative Inverse: For any a ∈ R where a ̸= 0, there
exists a−1 = 1a ∈ R such that a · a−1 = 1.
• Distributivity
• Multiplication Distributes over Addition: For any
a, b, c ∈ R, a · (b + c) = a · b + a · c.
 Example- NOT a field

The set of integers Z under the usual operations of addition and

multiplication.
• Addition: Z is closed under addition, and every integer has an
additive inverse (e.g., the additive inverse of 5 is -5).
• Multiplication: Z is closed under multiplication, and
multiplication is associative and commutative. The integer 1
acts as a multiplicative identity.
• Lack of Multiplicative Inverses: However, not every non-zero
integer has a multiplicative inverse within Z. For example, 2
does not have an integer x such that 2×x=1. Because of the
absence of multiplicative inverses for all non-zero elements, Z
is not a field.
 Vector Space

• A Vector space V (R) over a field R is a set on which two

operations: vector addition (+) and scalar multiplication (·)
are defined such that:
1. (V , +) forms an Abelian group: closure, commutativity,
associativity, identity, and inverse
2. The operation (·) is defined between scalar and vectors such
that ∀a ∈ R and v ∈ V =⇒ a · v ∈ V
3. ∀a ∈ R and v , w ∈ V =⇒ a · (v + w ) = a · v + a · w
4. ∀a, b ∈ R and v ∈ V =⇒ (a + b) · v = a · v + b · v
5. ∀a, b ∈ R and v ∈ V =⇒ (ab) · v = a(b · v )
6. ∃ 1 ∈ R such that 1 · v = v ∀v ∈ V
• If all these properties are satisfied, then (V , +, ·) forms a
vector space.
 Vector Space: Example- Property 1

• V = R2 is a vector space over the field of real numbers.

• Let (a, b), (c, d) ∈ R2 . Then

(a, b) + (c, d) = (a + c, b + d) ∈ R2

Thus, closure property holds in R2

• Let (a, b), (c, d) ∈ R2 . Then

(a, b) + (c, d) = (a + c, b + d)

= (c + a, d + b)
= (c, d) + (a, b)
Thus, commutativity property holds in R2
 Vector Space: Example- Property 1

• Let (a, b), (c, d), (e, f ) ∈ R2 . Then

(a, b) + ((c, d) + (e, f )) = (a, b) + (c + e, d + f )

= (a + (c + e), b + (d + f ))
= ((a + c) + e, (b + d) + f )
= (a + c, b + d) + (e, f )
= ((a, b) + (c, d)) + (e, f )
Thus, associativity property holds in R2
 Vector Space: Example- Property 1

• Let (a, b) ∈ R2 . Then

(0, 0) + (a, b) = (0 + a, 0 + b)

= (a, b)
Thus, (0, 0) is the identity element of R2
• Let (a, b), (−a, −b) ∈ R2 . Then

(a, b) + (−a, −b) = (a + (−a), b + (−b))

= (0, 0)
Thus, (−a, −b) is the inverse element of R2
Thus, (R2 , +) is an Abelian group.
 Vector Space: Example- Property 2 and 3

• Let k ∈ R, (a, b) ∈ R2 . Then

k(a, b) = (ka, kb) ∈ R2

• Let k ∈ R, (a, b), (c, d) ∈ R2 . Then

k((a, b) + (c, d)) = k(a + c, b + d)

= (k(a + c), k(b + d))

= (ka + kc, kb + kd)
= (ka, kb) + (kc, kd))
= k(a, b) + k(c, d))
 Vector Space: Example- Property 4

• Let k, m ∈ R, (a, b) ∈ R2 . Then

(k + m)(a, b) = ((k + m)a, (k + m)b)

= (ka + ma, kb + mb)

= (ka, kb) + (ma, mb)
= k(a, b) + m(a, b)
 Vector Space: Example- Property 5

• Let k, m ∈ R, (a, b) ∈ R2 . Then

k(m(a, b)) = k(ma, mb)

= k(ma, mb)
= (k(ma), k(mb))
= ((km)a, (km)b)
= (km)(a, b)
 Vector Space: Example- Property 6

• Let (a, b) ∈ R2 . Then

1(a, b) = (1a, 1b)

= (a, b)
• All the six properties are satisfied. Therefore, (R2 , +, ·) is a
vector space.
 Vector Space: Example

• V = Mm×n is a set of matrices of order m × n. Mm×n forms a

vector space over R. Taking an example of M22 .
   
• Let a11 a12 , b11 b12 ∈ R2 . Then
a21 a22 b b22
   21 
a11 a12 b11 b12
+ ∈ R2 .
a21 a22 b21 b22
     
• Let a11 a12 , b11 b12 , c11 c12 ∈ R2 . Then
 a21 a22  b21 b22   c21 c22
a11 a12 b b12 c c
+ [ 11 + 11 12 ]
 a21 a22
  b 21 b22 c 21 c22

a a12 b + c11 b12 + c12
= 11 + 11
a21 a22 b21 + c21 b22 + c22
 Vector Space: More Examples

• V = Rn forms a vector space over R.

• V = P forms a vector space over R
where P = set of all polynomials in variable x and coefficient
in R.
• V = Pn forms a vector space over R
where Pn = set of all polynomials of degree at most n in
variable x and coefficient in R.
 Examples that are not Vector Space

• V = Set of all polynomials of degree n does not form a vector

space over R: because vector addition is not closed. Example-

v1 = 2x 3 + 5x 2 + x + 7 ∈ V

v2 = −2x 3 + 3x 2 + 4x + 1 ∈ V
v1 + v2 = 8x 2 + 5x + 8 ∈
/V
Thus, V is not a vector space as the closure property is not
satisfied.
 Examples that are not Vector Space

• R2 (R) does not form a vector space w.r.t. vector addition and
scalar multiplication defined as follows:

(a, b) + (c, d) = (a + c, b + d)

k(a, b) = (ka, kb)

This is because (R2 (R), +) is not an Abelian group.

(x1 , y 1) + (x2 , y 2) = (x1 + x2 , y 1 + 2y 2)

(x2 , y 2) + (x1 , y 1) = (x2 + x1 , y 2 + 2y 1)

(x1 , y 1) + (x2 , y 2) ̸= (x2 , y 2) + (x1 , y 1))
Commutativity Property does not hold
 Geometric Interpretation of R2 and R3

• The vector v = (x, y ) or v = (x, y , z) is identified with the

directed line segment that has initial point at the origin and
terminal point with rectangular coordinates given by the
components of v .
 Eucledian Space

• The vectors in the Eucledian space consist of n−tuples of real

numbers: x = (x1 , x2 , ....., xn )
Eucledian space is used to mathematically represent physical
space, with notions such as length, distance, and angles.
Although it becomes hard to visualize for n > 3, these
concepts generalize mathematically in obvious ways.
 Subspaces of Rn

• If Rn is a vector space over R, then a subset S of Rn

(S ⊆ Rn ) is called as a subspace of Rn , if S is also a vector
space over the same field R. To put simply:
If Rn is a vector space, then S ⊆ Rn is said to be a subspace
of V = Rn (R) if
1. 0 ∈ S
2. S is closed under vector addition: x, y ∈ S implies x + y ∈ S
3. S is closed under scalar multiplication: x ∈ S, α ∈ R implies
αx ∈ S
 Examples of Subspaces of Rn : Theorem

• Let W be a subset of Rn . Then W is a subspace of Rn iff the

following conditions hold true:
1. W is non-empty
2. For any a, b ∈ R and any x, y ∈ W ,

ax + by ∈ W
 Examples of Subspaces

• Let V (R) be a vector space. Then,

1. S = {0}
2. S = V are subspaces of V . These are called trivial subspaces
of V .
• Set of all 3 × 3 skew-symmetric or symmetric matrices is a
subset of vector space V defined as:

V = {M3×3 |M is a 3 × 3 matrix with real entries}

• The set S = {(x1 , x2 , x3 ) ∈ R3 |x1 + x2 − x3 = 0} is a subspace

of R3 .
• The set S = {(x1 , x2 , x3 ) ∈ R3 |x1 + x2 + x3 = 1} is NOT a
subspace of R3 .
 Examples of Subspaces: Proof
The set S = {(x1 , x2 , x3 ) ∈ R3 |x1 + x2 − x3 = 0} is a subspace of
R3 .
• The zero vector of R3 is (0, 0, 0), as well as, (0, 0, 0) ∈ S
• Let a, b ∈ R and (x1 , x2 , x3 ), (y1 , y2 , y3 ) are two vectors in S,
then
x1 + x2 − x3 = 0, y1 + y2 − y3 = 0
a(x1 , x2 , x3 ) + b(y1 , y2 , y3 ) = (ax1 , ax2 , ax3 ) + (by1 , by2 , by3 )
= (ax1 + by1 , ax2 + by2 , ax3 + by3 )
ax1 + by1 + ax2 + by2 − ax3 − by3
a(x1 + x2 ) + b(y1 + y2 ) − (ax3 + by3 )
= a(x1 + x2 − x3 ) + b(y1 + y2 − y3 )
=a·0+b·0=0
Therefore, S is a subspace of R3
 Examples of Subspaces: Proof

The set S = {(x1 , x2 , x3 ) ∈ R3 |x1 + x2 + x3 = 1} is NOT a

subspace of R3 .
• =⇒ (0, 0, 0) =⇒ 0 + 0 + 0 = 0 ̸= 1
• =⇒ (0, 0, 0) ∈
/S
• =⇒ S is not a subspace of R3
 Subspaces: Remarks

We have the following subspaces for R3 ,

• Origin
• A line passing through the origin
• A plane passing through the origin
• The space of R3 itself
 Subspaces: Theorems

• The intersection of any non-empty collection of subspaces of

Rn is a subspace of Rn
• The union of two subspaces of Rn is a subspace of Rn iff one
of them is contained in the other.
 Intersection of Subspaces

The intersection of any non-empty collection of subspaces of Rn is

a subspace of Rn

S1 = {(x1 , x2 , x3 ) ∈ R3 |x1 + x2 − x3 = 0}

S2 = {(x1 , x2 , x3 ) ∈ R3 |x1 = x2 = x3 }
S1 ∩ S2 = (0, 0, 0)
x1 + x2 = x3 , x1 + x2 = x2 =⇒ x1 = 0 =⇒ x2 , x3 = 0
 Linear Span

Let V (F ) be a vector space.

Let S = {v1 , v2 , ..., vn } be a non-empty subset of V . Then the set
L(S) = {c1 v1 + c2 v2 + ... + cn vn |ci ∈ R, 1 ≤ i ≤ n} is a linear span
of the set S.
Example:
1. R2 = L({(1, 0), (0, 1)})
2. R3 = L({(1, 0, 0), (0, 1, 0), (0, 0, 1)})
3. If S is empty set, then L(S) = {0}
 Linear Span: Example

S = {(1, 1, 1), (2, 1, 3)}

L(S) = {c1 (1, 1, 1) + c2 (2, 1, 3)|c1 , c2 ∈ R}
= {(c1 + 2c2 , c1 + c2 , c1 + 3c2 )|c1 , c2 ∈ R}
= {(x1 , x2 , x3 )|2x1 − x3 = x2 } =⇒ R3
2c1 + 4c2 − c1 − 3c2 = c1 + c2 = x2
 Linear Span

Let S = {(1, 0), (0, 1)} ⊆ R2

L(S) = c1 (1, 0) + c2 (0, 1), c1 , c2 ∈ R

= (c1 , c2 )
L(S) = {(c1 , c2 )|c1 , c2 ∈ R} = R2
S = {(1, 0, 0), (0, 1, 0), (0, 0, 1)}
L(S) = {(c1 , c2 , c3 )|c1 , c2 , c3 ∈ R} = R3
 Linear Span: Example

S = {(1, 2, 0), (1, 1, −1)}

L(S) = {c1 (1, 2, 0) + c2 (1, 1, −1)}
= {(c1 + c2 , 2c1 + c2 , −c2 )}
 Important Results

• Let V (F ) is a vector space and let S be a non-empty subset

of V . Then L(S) is a subspace of V .
• Let S be a non-empty subset of a vector space V . Then L(S)
is the smallest subspace of V (F ) containing S.
 Important Subspaces

 
x11 x12 ... x1n
x x22 ... x2n 
• Let X =  21  where xij ∈ R. Then
 ... ... ... 
xm1 xm2 ... xmn
1. The row space of X given by
L({(x11 , x12 ..., x1n ), (x21 , x22 ..., x2n , ..(xm1 , xm2 ..., xmn ))}) is a
subspace of Rn (R).
2. The column space of X given by
L({(x11 , x21 ..., xm1 ), (x12 , x22 ..., xm2 , ..(x1n , x2n ..., xnm ))}) is a
subspace of Rm (R).
3. The set N(X ) = {y ∈ Rn |XY = 0} is called the nullspace of
X.
4. The set R(X ) = {b ∈ Rm |XY = b} is called the range of X .
In general, column space (X ) = R(X )