Introduction to Matrix Algebra

I. Definitions
A matrix is a collection of numbers ordered by rows and columns. It is customary to enclose the
elements of a matrix in parentheses, brackets, or braces. For example, the following is a matrix:

1 1

A 1 2
1 3

This matrix has three rows and two columns, so it is referred to as a 3 by 2 matrix. The elements of
a matrix are numbered in the following way:

a11 a12

A a 21 a 22
a

31 a32
That is, the first subscript in a matrix refers to the row and the second subscript refers to the column.
A vector is a special type of matrix that has only one row (called a row vector) or one column (called
a column vector).
A scalar is a matrix with only one row and one column.
A symmetric matrix is a square matrix (i.e. a matrix with as many rows as columns) in which aij = aji
for all i and j. Matrix A is symmetric:

1 2

A
2 1
A diagonal matrix is a symmetric matrix where all the off-diagonal elements are 0. Matrix A is
diagonal:

2 0

A
0 2
An identity matrix is a diagonal matrix with only 1s on the diagonal. The identity matrix is almost
always denoted as I:

1 0

I
0 1

II. Matrix Addition and Subtraction

To add two matrices, they both must have the same number of rows and they both must have the same
number of columns. The elements of the two matrices are simply added together, element by element,
to produce the results. That is, for C = A + B:

1 1
1 2
11 1 2 2 3
, B
, then C

2 3 2 6 5 6
2 2
3 4

If A

Matrix subtraction works in the same way, except that elements are subtracted instead of added.

III. Matrix Multiplication

There are several rules for matrix multiplication.
The first concerns the multiplication between a matrix and a scalar. Here, each element in the product
matrix is simply the scalar multiplied by the element in the matrix.

1 1

1 1 2 2

, and scalar = 2, then X 2X 2

If X
1 2 2
1 2

Matrix manipulation with a scalar is commutative. That is, it doesnt matter whether the scalar
premultiplies the matrix or vice versa, hence X = X.
The next rule involves the multiplication of a row vector by a column vector. To perform this, the row
vector must have as many columns as the column vector has rows.

1

If row vector z 2 3 4, and column vector y 2
3

1

Then zy 2 3 4 2 2(1) 3(2) 4(3) 20
3

Or we could write more generally as:

zy z1

z2

y1

3

z 3 y 2 z1 y1 z 2 y 2 z 3 y3 z k y k
k 1

y
3

The multiplication is equal to the sum of the kth element of the row vector multiplied by the kth
element of the column vector, over all elements. The product of a row vector multiplied by a column
vector will be a scalar.
The multiplication of vectors of different lengths is not feasible, such is the case below. Notice that
the row vector has three columns while the column vector has only two rows.

1
3 4
2

All other types of matrix multiplication involve the multiplication of a row vector and a column
vector. For matrix multiplication to be feasible, the first matrix must have as many columns as the
second matrix has rows (note that this is the requirement for multiplying a row vector by a column
vector). The resulting matrix will have as many rows as the first matrix and as many columns as the

second matrix. More generally, as long as multiplication is feasible, matrix A of dimension nm (n

rows, m columns), multiplied by matrix B of dimension pk (p rows, k columns) will result in matrix C
of dimension nk (n rows, k columns). Consider matrices A and B:

1 1

1 1
,
A 1 2 , B
0 2
1 3

1 1
1(1) 1(0) 1(1) 1(2) 1 3

1 1

1(1) 2(0) 1(1) 2(2) 1 5

Then C AB 1 2
1 3 0 2 1(1) 3(0) 1(1) 3(2) 1 7

Or we could write more generally for this case as:

a11 a12
a b a b

b11 b12 11 11 12 21
a21b11 a22b21
C AB a 21 a22
b21 b22

31 a32
a31b11 a32b21

2
a1k bk1
a11b12 a12b22 k21

a21b12 a22b22 a 2 k bk1
k 1
a31b12 a32b22 2
a b
3k k1
k 1

b
k 1

a 2 k bk 2

k 1

2
a3k bk 2

k 1

1k k 2

Because A has 3 rows and 2 columns and B has 2 rows and 2 columns, the matrix multiplication is
feasible and the resulting matrix will have 3 rows and 2 columns.
Because of these requirements, matrix multiplication is usually not commutative. That is, usually AB
BA. And even if AB is feasible, BA may not be feasible. For these reasons, the terms pre-multiply
and post-multiply are often encountered in matrix algebra while they are rarely encountered in scalar
algebra.
One special case of matrix multiplication involves the identity matrix, I. The identity matrix operates
as the number 1 does in scalar algebra. That is, any vector or matrix multiplied by an identity matrix is
simply the original vector or matrix. Hence yI = y, XI = X, etc. Note, however, that a scalar
multiplied by an identify matrix becomes a diagonal matrix with the scalars on the diagonal:

1 0 0

I
0 1 0
This should be verified by reviewing the rules above for multiplying a scalar and a matrix.
It should be noted at this stage that an idempotent matrix is a square matrix which if multiplied by
itself yields itself, such is case of diagonal matrix A above:
Identity, projection, residual making and centering matrices are examples of idempotent matrices:

1 0 1 0 1 0

II
0 1 0 1 0 1

P X ( X ' X ) 1 X '

PP X ( X ' X ) 1 X ' X ( X ' X ) 1 X ' X ( X ' X ) 1 X '

M I P I X ( X ' X ) 1 X '
MM ( I P)( I P) II PP 2IP I P 2P I P
M 0 ( I i(i' i) 1 i' )

M 0 M 0 ( I i(i' i) 1 i' )( I i(i' i) 1 i' ) II Ii(i' i) 1 i' i(i' i) 1 i' I i(i' i) 1 i' i(i' i) 1 i' I i(i' i) 1 i'

I
i ( i 'i ) 1 i '
i ( i 'i ) 1 i '

i ( i 'i ) 1 i '

IV. Matrix Transpose

The transpose of a matrix is denoted by a prime (A) or a superscript t or T (At or AT). The first row of
a matrix becomes the first column of the transpose matrix, the second row of the matrix becomes the
second column of the transpose, etc. Thus:

a11 a12

a11
If A a 21 a 22 , A'
a12
a

31 a32

a21
a22

a31

a32

1 1

1 1 1

Let A 1 2 , A'
1
2
3

1 3

The transpose of a row vector will be a column vector, and the transpose of a column vector will be a
row vector. The transpose of a symmetric matrix is simply the original matrix. For example:

1 0
1 0
, I '

I
0 1
0 1
Some important results involving transposes are:

( A' )' A
( AB)' B' A'

( ABC )' C ' B' A'

( A B)' A' B'
( A B C )' A' B'C'

V. Determinant
The determinant is a function of elements of a square matrix. Determinants do not exist for nonsquare matrices. A determinant of two rows and two columns and is called a determinant of the
second order. Consider matrix A:

a12
a

A 11
a21 a 22
The numbers a11 a22 in A comprise the principle diagonal, while the numbers a12 a21 make up the
secondary diagonal. By definition, the value of a second-order determinant is given by the product of
the principle diagonal minus the product of the secondary diagonal. So we have:

det( A) a11a22 a12a21,

A determinant with three rows and three columns is called a third-order determinant. A n x n
determinant is an nth-order determinant. Higher order determinants can be solved either by applying
the Sarrus Law or by expansion.

VI. Matrix Inverse

In scalar algebra, the inverse of a number is that number which, when multiplied by the original
number, gives a product of 1. Hence, the inverse of x is simply 1/x. In matrix algebra, the inverse of a
matrix is that matrix which, when multiplied by the original matrix, gives an identity matrix. The
inverse of a matrix is denoted by the superscript -1. Hence:

A1 A I AA 1
A matrix must be square to have an inverse, but not all square matrices have an inverse. In some cases,
the inverse does not exist.
To provide an example of inverse computation, consider square matrix J:

j
J 11
j21

j12 2 0

j22 0 1

Its inverse is J-1, and we are trying to determine the unknown elements a, b, c and d:

a b

J 1
c d
2 0 a b 1 0

I
JJ 1
0
1
c
d
0
1

We may express this as four equations and four unknowns (a, b, c and d) by multiplying out the
matrices and equating to the identity matrix:

j11a j12c 1
j 21a j 22c 0
j11b j12 d 0
j 21b j 22 d 1
Or by replacing with actual elements of J we can express this more simply as:

2a 1
c0
2b 0
d 1
Solving the equations yields the inverse expression for J:

1 / 2 0

J 1
0 1
In this case minimal manipulation of the equations is required to obtain a solution.
It is not always the case, however, and the general formula for an inverse of a 2 by 2 matrix is:

j22 j12
1

j11
j11 j22 j21 j12 j21

J 1

Applying to J we obtain the same outcome:

J 1

1 0
1

2(1) (0)(0) 0 2

1 1 0 1 / 2 0

2 0 2 0 1

Note that from the formula we may also write:

J 1

1 j22 j12

j11
det( J ) j21

Therefore, if the determinant of J is zero, J-1 is not well defined and we say that matrix J is noninvertible.
Important properties of the inverse of a matrix are:

( A 1 ) 1 A
( A' ) 1 ( A 1 )'
( AB ) 1 A 1 B 1
I 1 I
Where A and B are square matrices and I is an identity matrix.

VII. Rank
The column rank of a matrix A is the maximum number of linearly independent column vectors of A.
The row rank of A is the maximum number of linearly independent row vectors of A.
A result of fundamental importance in linear algebra is that the column rank and the row rank are
always equal. Therefore the number of linearly independent rows or columns is called the rank of A.
Consider the matrix A:

 2 3

A
0 1
The pair of columns is linearly independent, so it is of rank 2. Similarly, the transpose of A:

2 0

A'
3 1
It is also of rank 2. Indeed, since the column vectors of A are the row vectors of the transpose of A,
the statement that the column rank of a matrix equals its row rank is equivalent to the statement that
the rank of a matrix is equal to the rank of its transpose.
However, consider the matrix B:

2 1

A
2
1

The first column is simply the second column multiplied by 2 so this matrix is of rank 1.
If columns or rows are linearly independent, the matrix has a rank equal to its dimension and we say
that the matrix is of full rank.
Important properties of the rank of a matrix are:

Rank ( A) Rank ( A' )

Rank ( AB ) min( Rank ( A), Rank ( B))
Rank ( A' A) Rank ( AA' ) Rank ( A)
VIII. Trace
The trace of a square matrix is the sum of the terms on its leading diagonal. For example:

2 1

A
2 7
Writing trace A as Tr(A) = 2 + 7 = 9. Important properties of the trace of a matrix are:

Tr (cA) cTr ( A)
Tr ( A' ) Tr ( A)
Tr ( A B) Tr ( A) Tr ( B)
Tr ( I N ) N
Where c is a scalar, A and B are square matrices and IN is a n x n identity matrix.