UnSupervised Learning

Basic Linear Algebra Concepts for Machine Learning

 Matrix & Vector

 Diagonal Matrix
 Orthogonal Matrix
 Symmetric Matrix
 Eigen Value & Eigen Vector
 Principal Component Analysis
 Singular Value Decomposition
•Important definitions and notations of scalars, vectors, matrices, and tensors.
•Creating vectors and matrices using NumPy.
•Transposing a matrix(2D array) and coding them.
•Addition and broadcasting in matrices using NumPy

Matrix and Vectors

Arguably two of the most important concepts that you would encounter throughout your Machine
Learning journey.
An array of numbers is known as vectors whereas a matrix is 2-dimensional vectors which are
generally expressed in uppercase.
 In Machine Learning terms, a vector is the target variable in a supervised learning problem where
the features form the matrix in the data. Several operations like multiplication, transformation, rank,
conjugate, etc., could be performed with the matrix.
Two vectors of equal shape and with same number of elements could be added and subtracted.
 Scalar: Any single numerical value is a scalar as
shown in the image above. It is simply denoted by
lowercase and italics. For example: n Matrix: A matrix is a 2-D array of shape (m×n) with
m rows and n columns.

Vector: An array of numbers(data) is a Each element can be reached via its row and column and is denoted by
vector. You can assume a column in a dataset a subscript. For example, A₁,₁ returns the element at 1st row and 1st
to be a feature vector. column.
A matrix is denoted by uppercase, italics, and bold variable name. For
example: A

A vector is conventionally denoted by a

lowercase, italics and bold type variable(feature)
name. For example: x

Tensor: Generally, an n-dimensional array where n>2 is called

a Tensor. But a matrix or a vector is also a valid tensor.
Transposing a matrix is becomes a very important topic as it further contributes to calculating the inverse of
a matrix and other data manipulation where you want to mirror the axes of your dataframe.
Transposing a matrix converts a row vector to a column vector and vice-versa, it is denoted by the
superscript T:

Here is what a square matrix(identical number of rows and

columns) looks like after transposition with interchanged axes:
And if the matrix is not square, here is how
the shape of the matrix changes:
 Addition and Broadcasting in Matrices

Addition is just another simple operation that you would want to perform among many others.
We can add matrices of the same shape:
A+B=C
Each cell in matrix A gets added to the corresponding element in matrix B and the sum is stored at the
corresponding position in the resulting matrix C as shown in the figure on the left.
Now, you’d ask what if we have to add 2 matrices of different shape or add a scalar to a matrix, well
NumPy has got us covered for that with broadcasting:
Broadcasting
 Broadcasting is a technique that adds a scalar or a different shaped vector to a matrix by extending itself to all the
elements of the matrix. The scalar is added to each and every element of the matrix which is the reason it is called
“broadcasting”.
Let’s say we have 2 matrices of different shapes like in the
image below:

In this case, the matrix B(the smaller matrix) extends itself to

match the shape of matrix A(bigger matrix) like this:
Now we have the basic understanding of vectors, matrices, and tensors, it’s time to introduce you to a very important
fundamental concept of linear algebra — Dot product(Matrix Multiplication) and how it’s linked to solving
system of linear equations. And I say it’s important because it’s being widely used in almost all the major Machine
Learning algorithms.
•Dot product of a Matrix and a vector
•Dot product of two matrices
•Properties of Matrix multiplication
•Systems of Linear Equations
•Linear Regression Prediction Model Equation
Dot Product of a matrix and a vector
Unlike addition or subtraction, the product of two matrices is not calculated by multiplying each cell of one
matrix with the corresponding cell of the other but we calculate the sum of products of rows of one matrix
with the column of the other matrix as shown in the image below :
This matrix multiplication is also called Dot Product. Here is the formalization of the method:

Important Rule: You can only multiply two matrices if the number of columns of the first matrix is equal to the number of
rows of the second matrix.
If the shape of one matrix is (m×n) and the shape of the other one should be (n×t)(t ≥ 1), then the resulting product
matrix would have the shape (m×t) is shown below
Here’s a more comprehensive and notational form of a dot product between a matrix and a vector, Suppose we are
multiplying matrix A (3×2) with a vector B(2×1):

The resultant matrix will be a 3×1 vector calculated as

Let’s look at an example:

Multiplying 3 X2 matrix with vector

The resultant matrix calculated by following the above rule

Let’s code them using Numpy
Firstly, import the NumPy package in your workspace,

and then create a 2D matrix as discussed in the example

above or any other matrix structure you want to try it out

with, then create a vector ensuring the number of rows

being equal to the number of columns in the matrix.

You can then use the dot() method in the numpy package to

which you’d pass the matrix and the vector. It will return

the resultant vector for you as shown.

Dot product of two matrices.
We can then go ahead and multiply 2 matrices, here is an example of a 4×3 matrix A multiplied with 3×2 matrix B:

following the method as shown below:

Just like we coded the previous example, here you will be creating two 2D arrays and then using the dot method
of the array, you can calculate the dot product:
Code:
Properties of Matrix multiplication
Now that matrix multiplication is all clear, let’s look at a few properties of matrix multiplication explained below:

Here’s a task for you: You can create the matrices(A, B, or C) as per your imagination and then check for each rule by first
calculating the L.H.S. and then evaluate the R.H.S. with it. See, if they hold true.
We will check for transposition rule for dot
product here

You can see that we have created two

matrices A and B and then first solved for L.H.S. by

first calculating the dot product of the 2 matrices and

then

transposed them using the T method and evaluated it

against R.H.S. where the positions of the matrices

interchanged and they were first transposed

and then multiplied using the dot() method.

System of Linear Equations
Playing a vital role in linear algebra to help solve a wide variety of problems, a system of equations is basically defined as a
set of multiple equations. They can further be described using matrices, let’s see how.

Set of 2 equations with 2 unknowns. 

So, a system of equations has two main components:
1.Number of equations
2.Number of unknowns
The number of unknowns is the dimensionality of the problem at hand and the number of equations is the number of lines
in (2D) or n-dimensional planes.
Here is a system of 4 equations and 4 unknowns:

System of 4 equations with 4 unknowns x1, x2, x3, x4.

Matrix Representation of a system of linear equations

Now, we can use matrices to describe a system of linear equations which are of the form Ax = b. Consider this system of
linear equations:

Here, we have n unknowns from x₁ to xₙ and we have m number of rows(equations here).

We can deduce that the variables(from x₁ to xₙ) can be summarized in a vector x.

And we can have the As(coefficient(weight) for each variable) in another

matrix A
The L.H.S. of the system can then be described using the notation below

The dot product of A and vector x will give us the required

output vector b

Lets take an example of how this would happen for a

system of 2 equations:
Linear Regression
A practical example of what we learned today can be seen in the implementation of a linear regression model prediction
equation as follows:

here, which can further be written in a vectorized form like: yˆ=hθ(x)=θ·x

•ŷ is the predicted value. •θ is the model’s parameter vector with feature weights.
•n is the number of features. •x is the instance’s feature vector, containing x0 to xn, with x0
•xi is the ith feature value. always equal to 1.
•θj is the jth model parameter •θ · x is the dot product of the vectors θ and x which is equal to the
(including the bias term θ0 and the R.H.S. in the equation above.
feature weights θ1, θ2, ⋯, θn). •hθ is the hypothesis function, using the model parameters θ.
Conclusion:
You can hence realize the importance of the dot product and matrices & vectors in Machine Learning as every operation from
a high-level is being executed by these fundamentals under the hood.
With Dot product helping us to represent the system of equations, we will discuss about identity and inverse matrices.
These two types of matrices help us to solve the system of linear equations .
We will solve a system of 2 equations using NumPy basing
Identity Matrices
A special kind of matrix that has its main diagonal cells filled
with ones(1s) and the rest of the cells filled with zeros. Here is
what a 3×3 identity matrix looks like:
Properties:
The identity matrix is analogous to 1(in scalar) which is to
signify that applying(multiplying) the identity matrix to a vector
or matrix has no effect on the subject.
For example: Ix = x where, I is the Identity matrix x is a vector
Inverse Matrices
The inverse of a matrix A is a matrix which when multiplied with A itself, returns the Identity matrix. It is denoted by A⁻¹.

The intuition is that if we apply a linear transformation to the space with a matrix A, we can

revert the changes by applying A⁻¹ to the space again.

Note: If the determinant of the matrix is zero, then it will not have an inverse; the matrix is

then said to be singular. Only non-singular matrices have inverses.

Solving System of Linear Equations
Example:
we can represent a system of linear equations using matrices. Let’s say we have a system of equations as shown below, now
Now, we can use inverse matrices to solve them. this system is first needed to be represented in a format where it
For any equation Ax = b, we can simply multiply A⁻¹ on both can be represented in the form of Ax = b using the method on
sides of the equation and we’ll be left with an Identity matrix the right.
that doesn’t have any effect on x and thus our x would
be A⁻¹b as shown:
After moving all the unknown terms to the left and constants to the right, we can now write the matrix form of the above
system

Now, all we need to do is create these matrices and vectors in code using NumPy and then find out x = A⁻¹b