MATHEMATICAL FOUNDATIONS FOR MACHINE

LEARNING
Subject Code: MAT 216

Sajith P

Department of Mathematics

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 1 / 51

Syllabus
Module – 1 (LINEAR ALGEBRA)
1 Systems of Linear Equations

¬ Matrices,

¬ Solving Systems of Linear Equations.

¬ Vector Spaces,

¬ Linear Independence

¬ Basis and Rank,

¬ Linear Mappings

¬ Matrix Representation of Linear Mappings.

¬ Change of Basis,

¬ Image space and null (Kernel) space A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 2 / 51

Text book:

¬ Mathematics for Machine Learning by Marc Peter Deisenroth,

A. Aldo Faisal, and Cheng Soon Ong

¬ Published by Cambridge University Press

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 3 / 51

 Linear Algebra

Chapter. 1

Linear Algebra

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 4 / 51

 Linear Algebra

Vectors

¬ What is a vector?

¬ vectors are objects that can be added together and multiplied by

scalars to produce another object of the same kind.

Example 1: Geometric vectors

¬ vector may be familiar from high school mathematics and physics

¬ Two geometric vectors ⃗x , ⃗y can be added, such that ⃗x + ⃗y = ⃗z

is another geometric vector

¬ Multiplication by a scalar λ⃗x is also a geometric vector.

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 5 / 51

 Linear Algebra

Vectors

Example 2: Polynomials are also vectors

¬ Two polynomials can be added together, which results in another
polynomial

¬ polynomials can be multiplied by a scalar λ ∈ R, and the result is a

polynomial as well.

¬ Therefore, polynomials are instances of vectors.

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 6 / 51

 Linear Algebra

Vectors

Example 3: Audio signals are vectors

¬ Audio signals are represented as a series of numbers.

¬ Two audio signals can be added together, and their sum is a new
audio signal

¬ If we scale an audio signal, we also obtain an audio signal.

¬ Therefore, audio signals are instances of vectors.

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 7 / 51

 Linear Algebra

Systems of Linear Equations

¬ General form
 
a11 x1 + a12 x2 + a13 x3 + . . . . . . + a1n xn = b1
 a21 x1 + a22 x2 + a23 x3 + . . . . . . + a2n xn = b2 
 
............................................
 
............................................
am1 x1 + am2 x2 + am3 x3 + . . . . . . + amn xn = bm

¬ Ax = b

¬ Every n− tuple (x1 , x2 , . . . . . . , xn ) ∈ Rn that satisfies above equations

is a solution of the linear equation system

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 8 / 51

 Linear Algebra

Systems of Linear Equations

Different cases of solutions of a system of linear equations:

¬ No solution

¬ Unique Solution

¬ Infinite number of solutions

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 9 / 51

 Linear Algebra

Geometric interpertation

¬ No solution:
Cosider x1 + x2 = 2, 2x1 + 2x2 = 6

(0, 0)

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 10 / 51

 Linear Algebra

Geometric interpertation

¬ Unique solution
:
Cosider x1 + x2 = 2, x1 − x2 = 0
Solution is (1,1)

(1, 1)

(0, 0)

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 11 / 51

 Linear Algebra

Finding inverse by Gaussian elimination

¬ Determine the inverse of

 
1 0 2 0
1 1 0 0
 
1 2 0 1
1 1 1 1

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 12 / 51

 Linear Algebra

Finding inverse by Gaussian elimination

¬ Write the augmented matrix

 
1 0 2 0 1 0 0 0
 R2 → R2 − R1
 1 1 0 0 0 1 0 0 
0  R3 → R3 − R1

 1 2 0 1 0 0 1
1 1 1 1 0 0 0 1 R4 → R4 − R1

¬ Reduce the given matrix into an unit matrix by a sequence of row

transform.

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 13 / 51

 Linear Algebra

Finding inverse by Gaussian elimination

 
1 0 2 0 1 0 0 0
 0 1 −2 0 −1 1 0 0 
¬   R → R3 − 2R2
 0 2 −2 1 −1 0 1 0  3
R4 → R4 − R2
0 1 −1 1 −1 0 0 1

 
1 0 2 0 1 0 0 0
 0 1 −2 0 −1 1 0 0 
¬ 
 0

0 2 1 1 −2 1 0 
0 0 0 1 −1 0 −1 2 R4 → 2R4 − R3

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 14 / 51

 Linear Algebra

Finding inverse by Gaussian elimination

 
1 0 2 0 1 0 0 0
 R1 → R1 − R3
 0 1 −2 0 −1 1 0 0 
¬ 
 0 0 2 1 1 −2 1 0  R2 → R2 + R3
0 0 0 1 −1 0 −1 2

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 15 / 51

 Linear Algebra

Finding inverse by Gaussian elimination

 
1 0 0 −1 0 2 −1 0
 0 R1 → R1 + R4
1 0 1 0 −1 1 0 
 R2 → R2 − R4
¬ 
 0 0 2 1 1 −2 1 0 
R3 → R3 − R4
0 0 0 1 −1 0 −1 2

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 16 / 51

 Linear Algebra

Finding inverse by Gaussian elimination

 
1 0 0 0 −1 2 −2 2
 0 R1 → R1 + R4
1 0 0 1 −1 2 −2 
 R2 → R2 − R4
¬ 
 0 0 2 0 2 −2 2 −2 
R3 → R3 − R4
0 0 0 1 −1 0 −1 2

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 17 / 51

 Linear Algebra

Finding inverse by Gaussian elimination

 
1 0 0 0 −1 2 −2 2
 0 1 0 0 1 −1 2 −2 
¬ 
 0

0 2 0 2 −2 2 −2 
R3 → R3
2
0 0 0 1 −1 0 −1 2

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 18 / 51

 Linear Algebra

Finding inverse by Gaussian elimination

 
1 0 0 0 −1 2 −2 2
 0 1 0 0 1 −1 2 −2 
¬ 
 0

0 1 0 1 −1 1 −1 
0 0 0 1 −1 0 −1 2

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 19 / 51

 Linear Algebra

Finding inverse by Gaussian elimination

 
−1 2 −2 2
 1 −1 2 −2 
¬ A−1 = 
 1 −1 1 −1 
−1 0 −1 2

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 20 / 51

 Linear Algebra

Solving Systems of Linear Equations

¬ The general form of an equation system,

a11 x1 + . . . . . . + a1n xn = b1
a21 x1 + . . . . . . + a2n xn = b2
........................
am1 x1 + . . . . . . + amn xn = bm
¬ We can write Ax = b

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 21 / 51

 Linear Algebra

Elementary Transformations

¬ Exchange of two rows

¬ Multiplication of a row with a constant
¬ Addition of two rows

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 22 / 51

 Linear Algebra

(Row Echelon Form)

¬ All rows that contain only zeros are at the bottom of the matrix
¬ The first nonzero number from the left (also called the pivot or the
leading coefficient) is always strictly to the right of the pivot of the
row above it.

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 23 / 51

 Linear Algebra

(Basic and Free Variables)

¬ The variables corresponding to the pivots in the row-echelon form are

called basic variables
¬ Other variables are free variables

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 24 / 51

 Linear Algebra

Problem

¬ Find all solutions of the following system of equations

−2x1 + 4x2 − 2x3 − x4 + 4x5 = −3

4x1 − 8x2 + 3x3 − 3x4 + x5 = 2
x1 − 2x2 + x3 − x4 + x5 = 0
x1 − 2x2 − 3x4 + 4x5 = a

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 25 / 51

 Linear Algebra

Finding inverse by Gaussian elimination

 
−2 4 −2 −1 4 −3
 4 −8 3 −3 1 2 
 1 −2 1 −1 1 0  R1 ↔ R3
¬ [A : b] =  

1 −2 0 −3 4 a

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 26 / 51

 Linear Algebra

Finding inverse by Gaussian elimination

 
1 −2 1 −1 1 0
 R2 → R2 − 4R1
 4 −8 3 −3 1 2 
¬ [A : b] = 
 −2 4 −2 −1 4 −3  R3 → R3 + 2R1
1 −2 0 −3 4 a R4 → R4 − R1

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 27 / 51

 Linear Algebra

Finding inverse by Gaussian elimination

 
1 −2 1 −1 1 0
 0 0 −1 1 −3 2 
¬ [A : b] = 
 0 0

0 −3 6 −3 
R4 → R4 − R2
0 0 −1 −2 3 a

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 28 / 51

 Linear Algebra

Finding inverse by Gaussian elimination

 
1 −2 1 −1 1 0
 0 0 −1 1 −3 2 
¬ [A : b] = 
 0 0

0 −3 6 −3 
R4 → R4 − R3
0 0 0 −3 6 a − 2

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 29 / 51

 Linear Algebra

Finding inverse by Gaussian elimination

 
1 −2 1 −1 1 0
 0 0 −1 1 −3 2 
¬ [A : b] = 
 0 0

0 −3 6 −3 
0 0 0 0 0 a+1

¬ The system equation have a solution if a + 1 = 0.

¬ The system equation have a solution if a = −1.

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 30 / 51

 Linear Algebra

Problem

¬ Find all solutions of the following system of equations

x1 − 2x2 + x3 − x4 + x5 = 0
0x1 + 0x2 + x3 − x4 + 3x5 = −2
0x1 + 0x2 + 0x3 + x4 − 2x5 = 1

¬ To get particular solutions, set free varibles x2 = 0 and x5 = 0 and

solving
¬ x4 = 1, x3 = −1, x1 = 2

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 31 / 51

 Linear Algebra

Particular solution is

   
x1 2

 x2  
  0 

¬ 
 x3 =
  −1 

 x4   1 
x5 0

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 32 / 51

 Linear Algebra

To find general solution

¬ Set free variable x2 = 1, x5 = 0

   
x1 2

 x2  
  1 


 x3 =
  0 

 x4   0 
x5 0

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 33 / 51

 Linear Algebra

To find general solution

¬ Set free variable x2 = 0, x5 = 1

   
x1 2

 x2  
  0 


 x3 =
  −1 

 x4   2 
x5 1

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 34 / 51

 Linear Algebra

Set of all solutions

¬ Set free variable x2 = 0, x5 = 1

       
x1 2 2 2

 x2  
  0 


 1 


 0 

X =
 x3 =
  −1  + λ1 
  0  + λ2 
  −1 

 x4   1   0   2 
x5 0 0 1

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 35 / 51

 Linear Algebra

Problem

¬ Find 
all solutions
 of the system
 of equations
 AX = 12X where
x1 6 4 3
X =  x2  ∈ R3 , A= 6 0 9 
x3 0 8 0
Pn
i=1 xi =1

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 36 / 51

 Linear Algebra

Solution
¬ AX = 12X
¬ AX − 12IX = 0
¬ [A − 12I ]X = 0
 
 6 4 3 1 0 0 
   
0
 6 0 9  − 12  0 1 0  X =  0 
 
0 8 0 0 0 1 0
 
 6 4 3 12 0 0 
   
0
 6 0 9  −  0 12 0  X =  0 
 
0 8 0 0 0 12 0
    
−6 4 3 x1 0
 6 −12 9   x2  =  0 
0 8 −12 x3 0 A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 37 / 51

 Linear Algebra

Solution cont..

¬ Reduce the coefficient matrix into echelone form

 
−6 4 3
 6 −12 9 
R2 → R2 + R1
0 8 −12
 
−6 4 3
 0 −8 12 
0 8 −12 R3 → R3 + R2
 
−6 4 3
 0 −8 12 
0 0 0
A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 38 / 51

 Linear Algebra

Solution cont..

¬ Reduce the coefficient matrix into echelone form

    
−6 4 3 x1 0
 0 −8 12   x2  =  0 
0 0 0 x3 0

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 39 / 51

 Linear Algebra

Solution cont..

¬ −8x2 + 12x3 = 0

¬ 8x2 = 12x3

¬ x2 = 32 x3

¬ −6x1 + 4x2 + 3x3 = 0

¬ −6x1 + 4( 23 x3 ) + 3x3 = 0

¬ −6x1 + 9x3 = 0

¬ x1 = 23 x3
A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 40 / 51

 Linear Algebra

Solution cont..

¬ x1 = x2 = 32 x3

¬ Given x1 + x2 + x3 = 1

3
¬ 2 x3 + 23 x3 + x3 = 1

1
¬ x3 = 4

3
¬ x1 = x2 = 8

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 41 / 51

 Linear Algebra

Problem

¬ Find all solutions of the following system of equations

2w + 3x + 4y + 5z = 1
4w + 3x + 8y + 5z = 2
6w + 3x + 8y + 5z = 1
x1 − 2x2 − 3x4 + 4x5 = a

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 42 / 51

 Linear Algebra

Finding inverse by Gaussian elimination

 
2 3 4 5 1
R → R2 − R1
¬ [A : b] =  4 3 8 5 2  2
R3 → R3 − 3R1
6 3 8 5 1

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 43 / 51

 Linear Algebra

Finding inverse by Gaussian elimination

 
2 3 4 5 1
¬ [A : b] =  0 −3 0 −5 0  R3 → R3 − 2R2
0 −6 −4 −10 −2

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 44 / 51

 Linear Algebra

Finding inverse by Gaussian elimination

 
2 3 4 5 1
¬ [A : b] =  0 −3 0 −5 0 
0 0 −4 0 −2

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 45 / 51

 Linear Algebra

    
2 3 4 5 x1 1
¬ [A : b] =  0 −3 0 −5   x2  =  0 
0 0 −4 0 x3 −2

−1
¬ −4y = −2 =⇒ y = 2
−5t
¬ −3x − 5z = 0 Let z = t =⇒ x = 3
−1
¬ 2w + 3x + 4y + 5z = 1 =⇒ w = 2 .

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 46 / 51

 Linear Algebra

Problem

¬ Find all solutions of the system of equations AX = b where

     
x1 1 1 −1 −1 1
 x2   2 5 −7 −5   −2 
X = 
 x3  A=
 2 −1 1
 b= 
3   4 
x4 5 2 −4 2 6

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 47 / 51

 Linear Algebra

 
1 1 −1 −1 1
 2 5 −7 −5 −1  R2 → R2 − 2R1
¬ [A : b] =   R → R3 − 2R1
 2 −1 1 3 4  3
R4 → R4 − 5R1
5 2 −4 2 6

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 48 / 51

 Linear Algebra

 
1 1 −1 −1 1
 0 3 −5 −3 0 
¬ [A : b] = 
5 2  R3 → R + R2

 0 −3 3
0 −3 1 7 1 R4 → R4 − R2

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 49 / 51

 Linear Algebra

 
1 1 −1 −1 1
 0 3 −5 −3 0 
¬ [A : b] = 
 0

0 −2 2 2 
0 0 −4 4 1 R4 → R4 − 2R3

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 50 / 51

 Linear Algebra

 
1 1 −1 −1 1
 0 3 −5 −3 0 
¬ [A : b] = 
 0

0 −2 2 2 
0 0 0 0 −3

¬ System of equation is inconsistant

A link to tex.sx

Sajith P MATHEMATICAL FOUNDATIONS FOR MACHINE LEARNING 51 / 51