Vector/Matrix Norms and Iterative Linear Solvers

Daniel R. Reynolds
SMU Math3315/CSE3365, Spring 2010

In these notes we will introduce the ideas of vector and matrix norms, which can be used to
measure the size of vectors and matrices. With these tools, we will then discuss simple iterative
solvers for linear equations, which are increasingly used in scientific computing for solving very
large (and usually sparse) linear systems.

1 Vector Norms
Let us assume that we have vectors x ∈ IRn and y ∈ IRn , and a constant scalar α ∈ IR. Then
we can define the norm of a vector with the following definition.
Definition 1 (Vector Norm). A vector norm on IRn is a function k · k : IRn → IR that satisfies
the three properties:
a. kxk ≥ 0 for all x ∈ IRn , and kxk = 0 if and only if x is the zero vector.
b. kαxk = |α|kxk.
c. kx + yk ≤ kxk + kyk [called the “triangle inequality”].
While this may seem a bit esoteric, consider that if x ∈ IR1 , i.e. if x is a real number, the
absolute value function, | · |, satisfies all of the above properties, and therefore constitutes as a
norm. Another familiar norm would be the Euclidean norm for vectors x ∈ IR2 .
Example 1 (Euclidean norm on IR2 ). Consider the vector hx, yi ∈ IR2 . Then the function
p
k hx, yi k = x2 + y 2
satisfies all three properties of the norm:
p
a. x2 + y 2 ≥ 0, so x2 + y 2 ≥ 0. Moreover, this equals zero only when both x = 0 and y = 0.
p p p
b. kc hx, yi k = k hcx, cyi k = (cx)2 + (cy)2 = c2 (x2 + y 2 ) = |c| x2 + y 2 = |c| k hx, yi k.
c. It is a bit more involved to verify the triangle inequality, and uses the Law of Cosines from
Calculus III, i.e.
hx, yi · hp, qi = k hx, yi kk hp, qi k cos(θ),
where θ is the angle between the two vectors hx, yi and hp, qi. With this tool, we have
k hx, yi + hp, qi k2 = k hx + p, y + qi k2 = (x + p)2 + (y + q)2
= x2 + p2 + 2xp + y 2 + q 2 + 2yq = (x2 + y 2 ) + (p2 + q 2 ) + 2(xp + yq)
= k hx, yi k2 + k hp, qi k2 + 2 hx, yi · hp, qi
= k hx, yi k2 + k hp, qi k2 + 2k hx, yi kk hp, qi k cos(θ)
≤ k hx, yi k2 + k hp, qi k2 + 2k hx, yi kk hp, qi k = (k hx, yi k + k hp, qi k)2 .
Therefore, since both k hx, yi+hp, qi k2 and (k hx, yi k+k hp, qi k)2 are positive, and k hx, yi+
hp, qi k2 ≤ (k hx, yi k + k hp, qi k)2 , we have the desired result,
k hx, yi + hp, qi k ≤ k hx, yi k + k hp, qi k.
In the more general case of vectors x ∈ IRn , we have a few frequently-used norms:
n
!1/p
X
p
lp -norm: kxkp = |xi | , for p ∈ [1, ∞),
i=1
l∞ -norm: kxk∞ = max |xi |.
i=1:n

Two of the most commonly used lp -norms are the l2 -norm (or “2-norm” for short),
n
!1/2 v u n
X uX
2
kxk2 = xi =t x2i ,
i=1 i=1

which is a generalization of the Euclidean norm from IR2 to n-dimensional vectors, and the l1
norm,
n
X
kxk1 = |xi |.
i=1
To gain a little insight into how these norms measure the size of a vector in different ways,
Figure 1 shows the unit ball in IR2 for the l1 , l2 and l∞ norms, given by
Bp = {x ∈ IRn : kxkp = 1} ,
i.e. these are the sets of all vectors having length 1 in the given norm.

1
B
8

B2

B1

−1 1

−1

Figure 1: The unit balls B1 , B2 and B∞ for vectors in IR2 .

Example 2. Compute the l1 , l2 and l∞ norms of the vector x = [2, 4, −3, 1, −6].
kxk1 = 2 + 4 + 3 + 1 + 6 = 16,
p √
kxk2 = 22 + 42 + 32 + 12 + 62 = 66,
kxk∞ = max{2, 4, 3, 1, 6} = 6.
An important benefit of having a norm for a vector x ∈ IRn is that we can use it to measure
convergence of sequences of vectors.
Definition 2 (Vector Convergence). If we have a sequence of vectors {x(k) } and a vector x, all
in IRn , and
lim kx(k) − xk = 0,
k→∞

then we say that the sequence x(k)

converges to x in IRn . Moreover, this convergence can be
measured in any norm, i.e. if the above limit equals 0 when measured in one norm on IRn , the
limit will also equal 0 when measured in any other norm on IRn .
Example 3. The vectors  
(k) 1 2 3 6 1
x = , ,− , ,−
k k k k k
converge to the vector x = [0, 0, 0, 0, 0], since
 
(k) 1 2 3 6 1 13
lim kx − xk1 = lim + + + + = lim = 0.
k→∞ k→∞ k k k k k k→∞ k

2 Matrix Norms
In addition to defining norms to measure the size of vectors, we may define norms to measure
the “size” of a matrix.
Definition 3 (Matrix Norm). A matrix norm on IRn×n is a function k · k : IRn×n → IR that
satisfies the same three requirements as vector norms:
a. kAk ≥ 0 for all A ∈ IRn×n , and kAk = 0 if and only if A is the matrix of all zeros.
b. kαAk = |α|kAk.
c. kA + Bk ≤ kAk + kBk.
A special class of matrix norms are those that are defined by a vector norm on IRn .
Definition 4 (Induced Matrix Norm). Given a vector norm k · k∗ , we may define the corre-
sponding induced matrix norm according to the formula

kAk∗ = max kAxk∗ ,

x∈B∗

i.e. we may define the “size” of a matrix as the maximum amount it can stretch a unit vector
in a given norm.
For the three main vector norms on IRn we may compute their induced matrix norms, each of
which may be computed by a specific formula:
n
X
l1 -matrix norm: kAk1 = max kAxk1 = max |ai,j |,
x∈B1 j=1:n
i=1
n
X
l∞ -matrix norm: kAk∞ = max kAxk∞ = max |ai,j |,
x∈B∞ i=1:n
j=1

l2 -matrix norm: kAk2 = max kAxk2 = max |σi |,

x∈B2 i=1:n
where σi are the singular values of A (the eigenvalues of AT A). The first two are easy to
compute: to compute the l1 -matrix norm we merely take the l1 vector norm of each column of
A, and pick the largest resulting column sum; to compute the l∞ -matrix norm we add up the
rows and pick the largest resulting row sum. The l2 -matrix norm is less easily computed, and
we will not focus on that norm here.

Example 4. Compute the l1 and l∞ norms of the matrix

 
1 4 −2 4
 2 −4 −8 6 
A=  5
.
0 0 1 
−2 6 7 1

For the 1-norm, we add up the absolute value of each matrix column. The first column has sum
1 + 2 + 5 + 2 = 10, the second column 4 + 4 + 0 + 6 = 14, the third 2 + 8 + 0 + 7 = 17 and the
fourth is 4 + 6 + 1 + 1 = 12. The largest of these is 17, so kAk1 = 17.

Similarly, the row sums are 1 + 4 + 2 + 4 = 11, 2 + 4 + 8 + 6 = 20, 5 + 1 = 6 and 2 + 8 + 7 = 17,

so kAk∞ = 20.

Matrix norms have a number of special properties (you may verify many of these yourself):

• kIk = 1,

• kAxk ≤ kAkkxk,

• kABk ≤ kAkkBk,

• kAk1 = kAT k∞ , and kAk∞ = kAT k1 ,

• kAk2 = kAT k2 .

3 Stationary Iterations
We now consider the problem of solving the linear system Ax = b, where A ∈ IRn×n , x, b ∈ IRn ,
and now where n is very large. For problems of such size, the O(n3 ) complexity of Gaussian
Elimination becomes prohibitively large, and we may wish to look elsewhere for methods to solve
the linear system. It is in this regime of very large problems where we must leave the comfort of
provably stable and robust algorithms that directly solve the system (e.g. Gaussian Elimination
or LU factorization), and instead investigate methods that may work on some problems, but fail
on others.
To this end, we will follow the lead from our study of nonlinear solvers, specifically fixed-point
iterations, and attempt to approximate the solution x ∈ IRn to Ax = b using a sequence of
approximations {x(k) }, in the hopes that

(i) x(k) → x, and

(ii) given x(k) it is easy to compute x(k+1) .

This again brings up the questions

• How do we generate the iterates x(k) ?

 • Does the sequence converge?

• Under what conditions (if any) is this convergence guaranteed?

• How fast do these iterates converge?

The simplest of all iterative linear solvers (and the only ones that we will discuss in this class)
are so-called stationary iterations.

Definition 5 (Stationary Iteration). Given some nonsingular matrix Q, called the “splitting
matrix”, and an initial guess x(0) at the solution to Ax = b, a stationary iteration will compute
the iterates x(1) , x(2) , x(3) , . . ., recursively using the formula

Qx(k) = (Q − A)x(k−1) + b, k = 1, 2, . . . , (1)

i.e. at each iteration k, we solve the system Qx(k) = c(k−1) for the new guess x(k) , where the
right-hand-side is computed using the old guess as c(k−1) = (Q − A)x(k−1) + b.

We note that if the stationary iteration converges, i.e. if x(k) → x, then we may take the limit
of both sides of equation (1) to obtain

lim Qx(k) = lim (Q − A)x(k−1) + b, (2)

k→∞ k→∞

but since both x(k) → x and x(k−1) → x as k → ∞, equation (2) becomes

Qx = (Q − A)x + b ⇔ Qx = Qx − Ax + b ⇔ Ax = b.

So if the stationary iteration converges to any vector x, that vector x must solve the original
equation Ax = b!
In choosing the splitting matrix Q, we have the following considerations:

(1) The system Qx(k) = (Q − A)x(k−1) + b should be easy to solve, so that we may efficiently
generate the sequence {x(k) }. In other words, Q should have a desirable structure, so that
we may use a fast (low complexity), direct solver to compute the iterates.

(2) The matrix Q should generate a sequence {x(k) } that converges [rapidly] to the solution
x for an arbitrary initial guess x(0) .

Two extreme scenarios are Q = I and Q = A. In the former scenario, item (1) above is
easily met, since once the right-hand-side c(k−1) has been computed, the new iterate is given by
Ix(k) = c(k−1) , or x(k) = c(k−1) , so there is no solve involved whatsoever. However, it is unlikely
that such a simple method will work well at satisfying item (2).
In the latter scenario, item (2) is easily met, since Ax(1) = (A − A)x(0) + b = b, so the first guess
x(1) solves the system (i.e. we have extremely fast convergence). However, if you already have a
fast solver for the splitting matrix Q = A, you would obviously not be using an iterative method
in the first place, so most likely item (1) will not be met.
3.1 Convergence of Stationary Iterations
To understand more about how to choose Q, let’s first investigate how these methods work.
Investigating the stationary iteration (1), we have
Qx(k) = (Q − A)x(k−1) + b
⇒
x(k) = Q−1 (Q − A)x(k−1) + Q−1 b
⇔
x(k) = (I − Q−1 A)x(k−1) + Q−1 b.
If we now define the error vector at iteration k as e(k) = x(k) − x, and use both the above
equation and the fact that x solves Ax = b, we have
e(k) = x(k) − x = (I − Q−1 A)x(k−1) + Q−1 b − x
= (I − Q−1 A)x(k−1) + Q−1 Ax − x
= (I − Q−1 A)x(k−1) − (x − Q−1 Ax)
= (I − Q−1 A)x(k−1) − (I − Q−1 A)x
= (I − Q−1 A)(x(k−1) − x)
= (I − Q−1 A)e(k−1) .
If we name the matrix I − Q−1 A as G and the vector Q−1 b as d, and take norms of both sides
of the above equation, we have
ke(k) k = kGe(k−1) k ≤ kGkke(k−1) k.
So, if kGk < 1, we will have ke(k) k < ke(k−1) k, and therefore ke(k) k → 0 as k → ∞. Moreover,
we can measure kGk using any matrix norm we wish, since if the iteration converges for any
norm on IRn , it will converge for all norms on IRn , due to the equivalence of norms in IRn .

Note: if we were to think of the function g(x) = Gx + d as a fixed-point function that now oper-
ates on vectors x ∈ IRn , the ‘derivative’ of g(x) is just the matrix G. Moreover, the equivalent
of our requirement that |g 0 (x)| < 1 for convergence of scalar-valued fixed-point iterations is just
kGk < 1!

As we noted earlier, a subordinate matrix norm is merely a measurement of how much a ma-
trix stretches vectors. Therefore we may regard the requirement that kGk < 1 as saying that
multiplying a vector by G makes the vector smaller, since
kGk < 1 ⇒ kGxk ≤ kGkkxk < kxk.
This property of G to shrink vectors that it multiplies is an example of a contraction.
Definition 6 (Contraction). Let h be a function that takes as input a vector in IRn and outputs
a vector in IRn , i.e. h : IRn → IRn . The function h is called a contraction if for some subset
D ⊂ IRn ,
kh(x) − h(y)k < kx − yk, for all x, y ∈ D.
A matrix A ∈ IRn×n is just a linear function that takes a vector x ∈ IRn as input, and returns
the vector (Ax) ∈ IRn as output. Therefore, any matrix G ∈ IRn×n such that kGk < 1, is by
definition a contraction for all vectors in IRn .
3.2 Common Stationary Iterations
Let us first consider dissecting our matrix A into three familiar components, A = D + L + U :

(i) D is the diagonal of A, i.e. Ai,j where i = j.

(ii) U is the strictly upper-triangular portion of A, i.e. Ai,j where i < j.

(iii) L is the strictly lower-triangular portion of A, i.e. Ai,j where i > j.

3.2.1 Richardson Iteration

We have already briefly discussed the simplest of stationary iterations, wherein Q = I. This is
formally called the Richardson iteration, in which the vectors x(k) of our sequence are computed
with the recursion

x(k) = (I − A)x(k−1) + b = x(k−1) − Ax(k−1) + b = x(k−1) + (b − Ax(k−1) ).

This method only converges for an arbitrary initial guess x(0) when the matrix A satisfies
kI − Ak < 1, which unfortunately is usually untrue for most matrices; however the iterates are
very simple to compute.

Algorithm: [Richardson Iteration]

We take as inputs the matrix A, the rhs b, the initial guess at the
solution x, the desired residual tolerance tol, and the maximum
allowed number of iterations maxit.

function x = Richardson(A,b,x,maxit,tol)
for k=1:maxit
r = b - A*x
if (norm(r) < tol)
return
end
x=x+r
end

3.2.2 Jacobi Iteration

The next stationary iteration improves on Richardson incrementally. Instead of choosing Q = I
(the easiest linear system to solve), the Jacobi iteration chooses Q = D, the diagonal values
extracted from A (the second-easiest linear system to solve). With this choice, the iterations of
our sequence are computed via the formula

Dx(k) = (D − A)x(k−1) + b
⇒
x(k) = D−1 (D − A)x(k−1) + D−1 b
= (I − D−1 A)x(k−1) + D−1 b
= x(k−1) − D−1 Ax(k−1) + D−1 b
= x(k−1) + D−1 (b − Ax(k−1) ).
This method converges for an arbitrary initial guess x(0) when A satisfies kI − D−1 Ak < 1. This
requirement is true for all diagonally-dominant matrices A. Moreover, the iterates are still easy
to compute, since each iteration requires only a single diagonal solve, in addition to computation
of the residual vector r = b − Ax.

Definition 7 (Diagonally Dominant Matrices). A matrix A ∈ IRn×n is called diagonally-

dominant when
n
X
|ai,i | > |ai,j |, for all matrix rows i = 1 : n,
j=1,j6=i

i.e. when the absolute sum of all off-diagonal entries in a row is smaller than the absolute value
of the diagonal entry, and this property holds for all rows of the matrix.

Algorithm: [Jacobi Iteration]

We take as inputs the matrix A, the rhs b, the initial guess at the
solution x, the desired residual tolerance tol, and the maximum
allowed number of iterations maxit.

function x = Jacobi(A,b,x,maxit,tol)
n = length(b)
for k=1:maxit
r = b - A*x
if (norm(r) < tol)
return
end
for i=1:n
x(i) = x(i) + r(i)/A(i,i)
end
end

3.2.3 Gauss-Seidel Iteration

The last stationary iteration that we will discuss improves on the Jacobi iteration one more
level. Instead of only choosing Q to be the diagonal of A, the Gauss-Seidel iteration chooses
Q = D + L, i.e. it uses the entire lower-triangular portion of the matrix, instead of just the
diagonal.

Note: alternate formulations of Gauss-Seidel choose the upper-triangular portion of A instead

(Q = D + U ), though that is less often used in practice than the method described here.

With the choice of Q = D + L, the iterations of our sequence are computed via the formula

(D + L)x(k) = (D + L − A)x(k−1) + b
⇒
x(k) = (D + L)−1 (D + L − A)x(k−1) + (D + L)−1 b
= (D + L)−1 (D + L − D − L − U )x(k−1) + (D + L)−1 b
= −(D + L)−1 U x(k−1) + (D + L)−1 b,
or equivalently,

(D + L)x(k) = (D + L − A)x(k−1) + b
⇒
x(k) = (D + L)−1 (D + L − A)x(k−1) + (D + L)−1 b
= (I − (D + L)−1 A)x(k−1) + (D + L)−1 b
= x(k−1) − (D + L)−1 Ax(k−1) + (D + L)−1 b
= x(k−1) + (D + L)−1 (b − Ax(k−1) ).

This method converges for an arbitrary x(0) when A satisfies the property kI −(D +L)−1 Ak < 1,
or equivalently when A satisfies k(D + L)−1 U k < 1. It can be easily shown that this property
holds for all diagonally-dominant matrices A, as well as for some matrices with more significant
off-diagonal components. While each iterate does require a lower-triangular solve (forward sub-
stitution) to compute, these operations may still be significantly less costly than full Gaussian
Elimination, especially for problems that converge relatively rapidly in only a few iterations.

Algorithm: [Gauss-Seidel Iteration]

We take as inputs the matrix A, the rhs b, the initial guess at the
solution x, the desired residual tolerance tol, and the maximum
allowed number of iterations maxit.

function x = GaussSeidel(A,b,x,maxit,tol)
n = length(b)
for k=1:maxit
r = b - A*x
if (norm(r) < tol)
return
end
for j=1:n
y(j) = r(j)/A(j,j)
x(j) = x(j) + y(j)
for i=j+1:n
r(i) = r(i) - A(i,j)*y(j)
end
end
end

4 Problems
1. Show that the l∞ norm for vectors x ∈ IR2 satisfies the three properties of a norm.

2. Compute the l1 and l∞ matrix norms for the matrix

 1 
2 2 3 1
 2 −5 −2 1 
A=  1 12 −6 −2  .

2 1 5
3 2 6 25
3. Verify that kIk = 1 for any induced matrix norm.

4. Use the definition of an induced matrix norm to verify that kAxk ≤ kAkkxk for any vector
norm.

5. Show that kAk1 = kAT k∞ for an arbitrary A ∈ IR3×3 .

6. Consider the linear system

    
5 2 0 1 x1 5
 2 −5 − 1 1   x2   6 
2
  x3  =  −8
    .
 1 2 −6 −2 
3 1 −1 8 x4 2

Can you determine whether the Richardson, Jacobi or Gauss-Seidel iterations will converge
for this problem?

7. Write the stationary iteration matrix G for the alternate formulation of the Gauss-Seidel
iteration, that uses the upper-triangular portion of A as the splitting matrix.

8. Compute the cost for one iteration of the Jacobi iteration. To do this you will need to
compute the cost of the matrix-vector product Ax, as well as the cost of computing the
norm of r.