Chapter III: Solving linear system of

equations Ax = b
Numerical Analysis I

2nd Engineering ST, USTHB

1
 Contents

1 Introduction 3

2 Direct methods for solving systems of linear equations 3

2.1 Review 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.2 LU factorization method . . . . . . . . . . . . . . . . . . . . . . . . 8
2.2.1 Principle of the method . . . . . . . . . . . . . . . . . . . . 8
2.3 Review 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.4 Cholesky factorization method . . . . . . . . . . . . . . . . . . . . . 17
2.4.1 Principle of the method . . . . . . . . . . . . . . . . . . . . 17
1 Introduction
A system of m linear equations in n unknowns x1 , x2 , . . . , xn is a set of equations of the form:



 a11 x1 + a12 x2 + · · · + a1n xn = b1

 a21 x1 + a22 x2 + · · · + a2n xn = b2

(S) . (1)

 ..


 a x + a x + ··· + a x = b

m1 1 m2 2 mn n m

where the coefficients (aij )1≤i≤m,1≤j≤n , bi are given numbers.

The matrix form associated with the system (1) is given by:

Ax = b,

where      
a a12 · · · a1n x1 b1
 11     
 a21 a22 · · · a2n   x2   b2 
A= . .. .. , x= .. , b= .. .
     
 .. . ··· .   .   . 
     
am1 am2 · · · amn xn bm

Remark 1 Let the system Ax = b be given

• If b ̸= 0, then the system is called a non-homogenous system of linear equations.

• If b = 0, then Ax = 0, and the system is called a homogenous system of linear equations.

• If Ax = b has a solution, then the system is called consistent.

• If Ax = b has no solution, then the system is called inconsistent.

2 Direct methods for solving systems of linear equations

Here, direct methods refer to techniques used to find the exact solution of a system of linear
equations in a finite number of steps, without relying on approximations or iterative processes.
These methods typically involve algebraic manipulation or matrix operations to solve for the
unknowns directly.
2.1 Review 1
Definition 1 (Square matrix) is a matrix that has the same number of rows as columns.

1. Diagonal matrix: A diagonal matrix D = [dij ] is a square matrix with dij = 0

whenever i ̸= j.

Example 1 If D is a diagonal matrix of size n × n, it can be expressed in the following

form:  
d11 0 0 ··· 0
 

 0 d22 0 ··· 0 
D= 0 0 d33 ··· 0 ,
 
 .. .. .. . . . .. 

 . . . . 
0 0 0 · · · dnn

where d11 , d22 , d33 , . . . , dnn are the elements of the main diagonal.

2. Identity matrix: The identity matrix of order n, In = [δij ], is a diagonal matrix whose
diagonal entries are all 1s. When the size of In is clear, this matrix is generally written
simply as I.

Definition 2 (Triangular matrix) is a square matrix that has all zero entries except those
either on and above or on and below the main diagonal.

1. Upper-triangular matrix: is a square matrix in which all the entries below the main
diagonal are zero,
A = [aij ], where aij = 0, for all i > j.

Example 2 For example, a 3 × 3 upper-triangular matrix looks like this:

 
a11 a12 a13
 
U =
 0 .
a22 a23 
0 0 a33

2. Unit upper-triangular matrix: is an upper-triangular matrix where all diagonal

elements are equal to 1.
 Example 3 For example, a 3 × 3 unit upper-triangular matrix looks like this:
 
1 a12 a13
 
U =  0 1 a23 

.
0 0 1

3. Lower-triangular matrix: is a square matrix in which all the entries above the main
diagonal are zero,
A = [aij ], where aij = 0, for all i < j.

Example 4 For example, a 3 × 3 lower-triangular matrix looks like this:

 
a11 0 0
 
A=
 a21 a22 0 .
a31 a32 a33

4. Unit lower-triangular matrix: is a lower-triangular matrix where all diagonal ele-

ments are equal to 1.

Example 5 Here is an example of a 3 × 3 unit lower-triangular matrix

 
1 0 0
 
A =  a21 1 0 

.
a31 a32 1

Definition 3 (Invertible matrix) An n × n matrix A is said to be nonsingular (or invert-

ible) if an n × n matrix A−1 exists with AA−1 = A−1 A = I. The matrix A−1 is called the
inverse of A. A matrix without an inverse is called singular (or noninvertible).

Definition 4 (Transpose of a matrix) The transpose of an n × m matrix A = [aij ] is the

m × n matrix AT = [aji ], where for each i, the ith column of AT is the same as the ith row
of A.

Example 6 Let !
1 2 3
A= .
4 5 6
Then, the transpose of matrix A is given by:
 
1 4
 
At = 
 2 5 .

3 6

Theorem 1 The following operations involving the transpose of a matrix hold whenever the
operation is possible:

1. (AT )T = A. 4. (cA)T = cAT , for any scalar c.

2. (A + B)T = AT + B T .

3. (AB)T = B T AT . 5. If A−1 exists, then (A−1 )T = (AT )−1 .

Definition 5 (Leading principal minors) Leading principal minors of order k, denoted

as ∆k (A), are defined as the determinants of the leading principal submatrices of a square
matrix A of order n × n. Specifically, the leading principal submatrix of order k is formed by
taking the first k rows and the first k columns of A, for each k = 1, 2, . . . , n.

Example 7 1. Consider the matrix A of order 3 × 3:

 
a11 a12 a13
 
A=
a21 a22 a23  .

a31 a32 a33

The leading principal minors of A are:

 
! a11 a12 a13
a11 a12  
∆1 (A) = a11 , ∆2 (A) = det , ∆3 (A) = det  a 21 a 22 a 23
 = det(A).
a21 a22  
a31 a32 a33

2. Consider the matrix:  

1 2 3
 
B=
 4 5 6.

7 8 9

The leading principal minors are calculated as follows:

- For k = 1: ∆1 (B) is the determinant of the top-left 1 × 1 submatrix,
 ∆1 (B) = a11 = 1.

- For k = 2: ∆2 (B) is the determinant of the top-left 2 × 2 submatrix,

!
1 2
∆2 (B) = det = −3.
4 5

- For k = 3: ∆3 (B) is the determinant of the entire matrix B,

 
1 2 3
 
∆3 (B) = det 
 4 5 6 = det(B) = 0.

7 8 9

3. Consider the matrix:

 
−3
4 2 3
 
0 5 6 9
C=
 .
 0 0 −2 1 

0 0 0 7

The leading principal minors are calculated as follows:

∆1 (C) = c11 = 4.

!
4 2
∆2 (C) = det = 20.
0 5
 
4 2 3
 
∆3 (C) = 
0 5 6 = −40.
0 0 −2
 
4 2 −3 3
 
0 5 6 9
∆4 (C) = det 
  = det(C) = −280.
0 0 −2 1 


0 0 0 7
Remark 2 Suppose A is n × n matrix:

1. If any row or column of A has only zero entries, then det(A) = 0.

2. If A has two rows or two columns the same, then det(A) = 0.

3. If B is also an n × n matrix, then det(AB) = det(A) det(B).

4. det(AT ) = det(A).

5. When A−1 exists, det(A−1 ) = 1

det(A)
.
Qn
6. If A is an upper-triangular, lower-triangular, or diagonal matrix, then det(A) = i=1 aii .

2.2 LU factorization method

LU factorization, also known as LU decomposition, is a mathematical method used in numer-
ical analysis and linear algebra to factor a square matrix into the product of two triangular
matrices: a lower-triangular matrix L and an upper-triangular matrix U . This technique is
particularly useful for solving systems of linear equations, and computing determinants.

2.2.1 Principle of the method

LU factorization method
Given a system of linear equations Ax = b, where A is a square matrix of order
n × n.
1. LU factorization of A: To perform LU factorization of a matrix A, we ex-
press it as the product of a lower-triangular matrix L and an upper-triangular
matrix U , such that:
A = LU,

where the main diagonal of either L or U consists of all ones.

2. Solving the system: The system Ax = b becomes


Ly = b
Ax = b ⇔ L |{z}
Ux = b ⇔
U x = y.
y

This approach breaks the problem into two simpler triangular systems. First,
solve the system Ly = b for y, and then solve the system U x = y for x.
Remark 3 Doolittle’s reduction method and Crout’s reduction method are both techniques
used for LU factorization of matrices, which is essential for solving systems of linear equa-
tions. Below is a comparison of the two methods.

1. Doolittle’s reduction method: The matrix L in LU factorization of the matrix A is

unit lower-triangular matrix (the diagonal elements of L are 1s).

2. Crout’s reduction Method: The matrix U in LU factorization is unit upper-triangular

matrix (the diagonal elements of U are 1s).

Theorem 2 A square matrix A can be decomposed into a product of a lower-triangular ma-

trix L and an upper-triangular matrix U if all leading principal minors of A are non-zero.

Example 8 Consider the following linear system of equations to solve with Doolittle’s re-
duction method: 
 2x1 + x2 + 2x3 = 10


6x1 + 4x2 = 26


 8x + 5x + x = 35
1 2 3

The matrix form of this system is given by Ax = b, where

     
2 1 2 x1 10
     
A=
 6 4 0 ,
 x=
 x 2
,
 b=
 26 .

8 5 1 x3 35

1. Step 1: LU factorization of matrix A

Firstly, we check if all leading principal minors of A are non-zero.
 
! 2 1 2
2 1  
∆1 (A) = 2 ̸= 0, ∆2 (A) = det = 2 ̸= 0, ∆3 (A) = det  6 4 0 = −2 ̸= 0.
6 4  
8 5 1

Therefore, A can be decomposed into the form A = LU , where L is a unit lower-

triangular matrix and U is an upper-triangular matrix that defined by
   
1 0 0 u11 u12 u13
   
L=
 l21 1 ,
0  U =
 0 .
u22 u23 
l31 l32 1 0 0 u33
 Now, we want to find the elements of L and U , and then
    
2 1 2 1 0 0 u11 u12 u13
    
A = LU ⇔ 
 6 4 0  =  l21 1
  0 
 0
 u22 u23 

8 5 1 l31 l32 1 0 0 u33
 
u11 u12 u13
 
=  l21 u11
 l21 u12 + u22 l21 u13 + u23 .

l31 u11 l31 u12 + l32 u22 l31 u13 + l32 u23 + u33

Therefore,  

 u11 = 2 
 u11 = 2

 




 u12 = 1 


 u12 = 1
 



 u13 = 2 


 u13 = 2
 
l21 u11 = 6 l21 = 3

 


 

l21 u12 + u22 = 4 ⇒ u22 = 1
 
u23 = −6
 


 l21 u13 + u23 = 0 



 




 l31 u11 = 8 


 l31 = 4
 
l31 u12 + l32 u22 = 5 l32 = 1

 


 

 
l31 u13 + l32 u23 + u33 = 1 u33 = −1
 

Thus, we have the following L and U matrices:

   
1 0 0 2 1 2
   
L= 3 1 0 

, U =  0 1 −6 

.
4 1 1 0 0 −1

2. Step 2: Solving the system Ax = b


Ly = b
Ax = b ⇔ L |{z}
Ux = b ⇔
U x = y.
y

Firstly, we solve the system Ly = b, and then we solve the system U x = y.

 (a) Solve the lower-triangular system Ly = b
    
1 0 0 y1 10
    
Ly = b ⇔  3 1 0   y2  =  26 
    
.
4 1 1 y3 35

 
 y1 = 10  y1 = 10

 

⇒ 3y1 + y2 = 26 ⇒ y2 = −4
 

 4y + y + y = 35  y = −1

1 2 3 3

 
10
 
Thus, the solution to the system Ly = b is y = 
 −4 .

−1
(b) Solve the upper-triangular system U x = y
    
2 1 2 x1 10
    
Ux = y ⇔ 
 0 1 −6   x2  =  −4  .
   
0 0 −1 x3 −1

 
 2x1 + x2 + 2x3 = 10  x3 = 1

 

⇒ x2 − 6x3 = −4 ⇒ x2 = 2
 
− x3 = −1

 
 x =3
1

 
3
 
The solution to the system is: x = 
 2 .

1

Example 9 Consider the previous example of a linear system of equations that we will solve
using Crout’s reduction method.

1. Step 1: LU factorization of matrix A

We decompose the matrix A into the form A = LU , where L is a lower-triangular
matrix and U is a unit upper-triangular matrix defined by
   
l11 0 0 1 u12 u13
   
L=
 l21 l22 0 , U =
 0 1 .
u23 
l31 l32 l33 0 0 1
 Now, we want to find the elements of L and U , then
    
2 1 2 l11 0 0 1 u12 u13
    
A = LU ⇔ 
 6 4 0  =  l21
  l22 0  0
 1 u23 

8 5 1 l31 l32 l33 0 0 1
 
l11 l11 u12 l11 u13
 
=  l21
 l21 u12 + l22 l21 u13 + l22 u23 .

l31 l31 u12 + l32 l31 u13 + l32 u23 + l33

Therefore,  

 l11 = 2 
 l11 = 2

 




 l21 = 6 


 l21 = 6
 



 l31 = 8 


 l31 = 8
  1
l11 u12 = 1 u12 =

 


 
 2
(1) l21 u12 + l22 = 4 ⇒ l22 = 1

 



 l31 u12 + l32 = 5 

 l32 = 1

 




 l11 u13 = 2 


 u13 = 1
 
l21 u13 + l22 u23 = 0 u23 = −6

 


 

 
l31 u13 + l32 u23 + l33 = 1 l33 = −1
 

Thus, we have the following L and U matrices:

   
1
2 0 0 1 2
1
   
L=
 6 1 0 , U =
 0 1 −6 .

8 1 −1 0 0 1

2. Step 2: Solving the system Ax = b


Ly = b
Ax = b ⇔ L |{z}
Ux = b ⇔
U x = y.
y

Firstly, we solve the system Ly = b, and then we solve the system U x = y.

 (a) Solve the lower-triangular system Ly = b
    
2 0 0 y1 10
    
Ly = b ⇔  6 1 0   y2  =  26 
    
.
8 1 −1 y3 35

 
 2y1 = 10  y1 = 5

 

⇒ 6y1 + y2 = 26 ⇒ y2 = −4
 
 8y + y − y = 35
 
 y =1
1 2 3 3

 
5
 
Thus, the solution to the system Ly = b is y = 
 −4 .

1
(b) Solve the upper-triangular system U x = y
    
1
1 2
1 x1 5
    
Ux = y ⇔ 
 0 1 −6   x2  =  −4  .
   
0 0 1 x3 1

 
1
 x1 + 2 x2 + x3 = 5  x3 = 1

 

⇒ x2 − 6x3 = −4 ⇒ x2 = 2

 

 x3 = 1  x =3
1

 
3
 
 2 .
The solution to the system is: x =  
1

2.3 Review 2
Definition 6 (Eigenvalues of a square matrix) Let A be a square matrix of order n.
The eigenvalues of A are the roots of the characteristic polynomial associated with A.
The characteristic polynomial P (λ) is defined as:

P (λ) = det(A − λIn ),

where, In is the identity matrix of order n × n, and λ represents the eigenvalue of A.

The polynomial P (λ) is a polynomial in λ, and the roots of this polynomial are the eigenvalues
of A.

Example 10 Suppose we have the matrix A of order 2 × 2:

!
4 1
A= .
2 3

The characteristic polynomial P (λ) is given by:

!
4−λ 1
P (λ) = det(A − λI2 ) = det .
2 3−λ
Now, calculate the determinant:

P (λ) = (4 − λ)(3 − λ) − (1)(2) = (4 − λ)(3 − λ) − 2 = λ2 − 7λ + 10.

Thus, the characteristic polynomial is:

P (λ) = λ2 − 7λ + 10.

The eigenvalues of A are the roots of this polynomial. For the equation:

λ2 − 7λ + 10 = 0.

The discriminant is given by:

∆ = (−7)2 − 4 × 1 × 10 = 9 > 0.

We have ∆ > 0, thus the equation λ2 − 7λ + 10 = 0 has two distinct real roots
√ √
−(−7) + ∆ 7+ 9 7+3
λ1 = = = ,
2×1 2 2
√ √
−(−7) − ∆ 7− 9 7−3
λ1 = = = .
2×1 2 2
The solutions are:

λ1 = 5 and λ2 = 2

Thus, the eigenvalues of A are λ1 = 5 and λ2 = 2.

Definition 7 (Symmetric matrix) A symmetric matrix is a square matrix that is equal
to its own transpose. In other words, a matrix A is symmetric if: AT = A.

Definition 8 (Positive definite matrix) A matrix A is positive definite if xT Ax > 0 for

every n-dimensional vector x ̸= 0.

Remark 4 Several conditions are equivalent for a matrix to be positive definite:

1. All leading principal minors of the matrix are strictly positive.

2. All the eigenvalues of the matrix are strictly positive.

Example 11 Consider the matrix:

 
1 2 3
 
A=
2 1 .
2
3 2 −1
The leading principal minors of A are:

∆1 (A) = 1 > 0,
!
1 2
∆2 (A) = det = −3 < 0,
2 1
∆3 (A) = det(A) = 14 > 0.

Since the first and third leading principal minors are strictly positive, but the second is not
strictly positive, the matrix A is not positive definite matrix.

Example 12 Consider the matrix:

!
2 −1
A= .
−1 2
To verify that the matrix A is positive definite, we can use the following methods:

1. Quadratic Form: For any non-zero vector:

!
x1
x= .
x2
 ! !
T 2 −1 x1
x Ax = (x1 , x2 ) = (2x1 − x2 )x1 + (−x1 + 2x2 )x2 = 2x21 + 2x22 − 2x1 x2 .
−1 2 x2
Rearranging the terms gives

xT Ax = x21 + x22 − 2x1 x2 + x21 + x22 = (x1 − x2 )2 + x21 + x22 > 0.

Since this quadratic form is strictly positive for any non-zero vector x, the matrix A is
positive definite.

2. Leading principal minors: Calculate the leading principal minors and check if they
are all strictly positive.

∆1 (A) = 2 > 0.
∆2 (A) = det(A) = 3 > 0.

Since both leading principal minors are strictly positive, the matrix A is positive definite.

3. Eigenvalues of A: Compute the eigenvalues of A. A matrix is positive definite if all

its eigenvalues are strictly positive.
The characteristic polynomial of A is given by:

P (λ) = det(A − λI2 ),

where I2 is the identity matrix of order 2 × 2 and λ represents the eigenvalue of A.

Now, we determine the eigenvalues
!
2−λ −1
P (λ) = 0 ⇔ det(A − λI2 ) = 0 ⇔ det = (2 − λ)2 − 1 = 0.
−1 2−λ

Thus, we find: λ1 = 1 and λ2 = 3.

The eigenvalues of A are 1 and 3, both strictly greater than 0, confirming that the
matrix is positive definite.
2.4 Cholesky factorization method
Cholesky factorization is a method of decomposing a symmetric positive definite matrix
into the product of a lower triangular matrix with positive entries on the diagonal and
its transpose. Specifically, if A is a symmetric positive definite matrix, then the Cholesky
factorization expresses as:
A = LLT .

2.4.1 Principle of the method

Cholesky factorization method

Given a system of linear equations Ax = b, where A is a symmetric positive
definite matrix of order n × n.
1. Cholesky factorization of A: Factor the matrix A into the form

A = LLT ,

where L is a lower-triangular matrix with strictly positive entries on the

diagonal, and LT is the transpose of L, which is an upper-triangular matrix.

2. Solving the system: The system Ax = b becomes


Ly = b
LT x = b ⇔
Ax = b ⇔ L |{z}
LT x = y.
y

This breaks the problem into two simpler triangular systems. First, solve
the system Ly = b for y, and then solve the system LT x = y for x.

Theorem 3 If A ∈ Mn (R) is a symmetric positive definite matrix, then there exists at least
one lower-triangular matrix L such that

A = LLT .

Moreover, if the diagonal elements lii of L are strictly positive, the decomposition is unique.
Example 13 Consider the following linear system of equations to solve using the Cholesky
factorization method: 
 2x1 + x2 + 2x3 = 1


x1 + 3x2 + x3 = 1


 2x + x + 3x = 1
1 2 3

The matrix form of this system is given by Ax = b, where

     
2 1 2 x1 1
     
A=
 1 3 1 ,
 x=
 x 2
,
 b=
 1 .

2 1 3 x3 1
Check if the matrix A is symmetric and positive definite.

• The matrix A is symmetric, since AT = A.

• A is a positive definite matrix since all its leading principal minors are strictly positive.
∆1 (A) = a11 = 2 > 0,
!
2 1
∆2 (A) = det = 5 > 0,
1 3
 
2 1 2
 
∆3 (A) = det 
 1 3 1  = 5 > 0.

2 1 3

Now, we can apply the Cholesky decomposition.

1. Step 1: Cholesky factorization of A

Factor the matrix A into the form

A = LLT ,

where L is a lower-triangular matrix with strictly positive entries on the diagonal, and
LT is the transpose of L.
The matrix L is defined by

 
l11 0 0
 
L=
 l21 l22 0  , with l11 , l22 , l33 > 0.
l31 l32 l33
 Now, we want to find the elements of L, and then
    
2 1 2 l11 0 0 l11 l21 l31
    
A = LLT ⇔ 
 1 3 1  =  l21 l22 0   0 l22 l32 
   
2 1 3 l31 l32 l33 0 0 l33
 
l2 l11 l21 l11 l31
 11 
=  l21 l11 2 2 .
 l21 + l22 l21 l31 + l22 l32 
2 2 2
l31 l11 l31 l21 + l32 l22 l31 + l32 + l33

Therefore, 
2

 l11 =2

√


 l21 l11 = 1 
l11 = 2

 
 


 l31 l11 = 2 

 √

 
 l21 = 2/2
l11 
l21= 1 √

 

 
 l

2 2 31 = 2
l21 + l22 =3 ⇒ p .

 
 l22 = 5/2

 l31 l21 + l32 l22 = 1 

l32 =0

 

  

 l l
11 31
  = 2 

l33 =1

  

 (((
l l + l l = 1
 ( (
21 31 22 32

 ( ( (

 (
 2 2 2
l31 + l32 + l33 =3

Thus, we get
 √   √ √ √ 
2
2 0 0 2 2
 √ q   q2 
L= 2
2 5
2 ,
0  LT = 
 0
5
2
0  .
√

2 0 1 0 0 1

2. Step 2: Solving the system Ax = b


Ly = b
LT x = b ⇔
Ax = b ⇔ L |{z}
LT x = y.
y

Firstly, we solve the system Ly = b, and then we solve the system LT x = y.

 (a) Solve the lower-triangular system Ly = b
 √    
2 0 0 y1 1
 √ q    
2 5
Ly = b ⇔ 
 2 2
0 
  y2
  =  1 .
√
  
2 0 1 y3 1

 √ 
√1
 2y1 =1  y1 =
 

√ q  2
2 5 √1
⇒ 2
y1 + y
2 2
=1 ⇒ y2 = 10
 √

 

 y =0
2y1 + y3 = 1 3

 
√1
 2 
Thus, the solution to the system Ly = b is y =  √1 .
 10 
0
(b) Solve the upper-triangular system LT x = y
 √ √ √    
2
2 2 x1 √1
 q2    2 
LT x = y ⇔ 
 0
5
2
0   x2  =  √1
   10
.

0 0 1 x3 0

 √ √ √ 
2 √1
2x + x2 + 2x3 =
1
 x3 = 0
 2


 q 2 
⇒ 5 1 ⇒ 1
x
2 2
= √
10
x2 = 5
 
 
 x = 2
x3 = 0

1 5

 
2
 5 
The solution to the system is: x =  1 .
 5 
0