FIXED POINT ITERATIONS

EIRIK HOEL HØISETH

1 Introduction
Let D ⊂ Rn and F : D 7→ Rn be a continuous vector valued mapping in n variables1 .
Our goal is the solution of an equation

F(x) = 0. (1)

Assume the root finding problem (1) has the equivalent form

G(x) = x, (2)

for G : D 7→ Rn , i.e. x∗ ∈ D is a solution of (1) if and only if it is a solution of (2).

We say that x∗ is a fixed point of G. A simple example of this is

G(x) = x + F(x).

A straightforward idea for the solution of fixed point equations (2) is that of fixed
point iterations. Starting with some point x(0) ∈ Rn (preferably an approximation
to a solution of (2)), we define the sequence {x(k) }∞ n
k=0 ⊂ R by the recursive relation

x(k+1) = G(x(k) ). (3)

This idea is based on the observation that this sequence will become stationary after
some index k if and only if x(k) is a fixed point of G. Moreover it seems reasonable
that x(k) is close to a fixed point if x(k) ≈ x(k+1) . However we still need to develop
sufficient conditions on G for when this function has a fixed point, and when the
iteration (3) actually works.

2 Main results
The first result gives a sufficient condition for G to have a fixed point. We first
define a convex domain/set
1
Notation: For two sets A, B we write A ⊂ B if and and only if x ∈ A implies x ∈ B. So A ⊂ A
is true. Some people use the notation ⊆ instead.

1
Definition 1. A domain D is convex if for any x, y ∈ D,

(1 − t)x + ty ∈ D for all t ∈ [0, 1].

Theorem 2 (Brouwer’s fixed-point theorem). Let D ⊂ Rn be closed, bounded and

convex. If G : D 7→ D is continuous it has a fixed point.

The general proof of this is well outside the scope of this course. Note that for the
special case n = 1, i.e. scalar functions, this theorem states that if G : [a, b] 7→ [a, b]
where a ≤ b ∈ R then G has a fixed point if it is continuous.
The next theorem is very important. It gives sufficient conditions for both the
existence and uniqueness of the fixed point, as well as for the fixed point iterations to
converge towards this fixed point. First, recall the definition of a Cauchy sequence

Definition 3. {x(k) }∞ n
k=0 in some subset of R is Cauchy in some norm k · k if
elements of the sequence become arbitrarily close together as the sequence progresses.
To be specific, for any  > 0 there exists a positive integer n0 such that for all q, r ∈ N
such that q, r > n0
kx(q) − x(r) k < 

Theorem 4 (Banach’s fixed-point theorem). Assume that D ⊂ Rn is closed, and

that G : D 7→ D is a contraction. That is, there exists 0 ≤ C < 1 such that

kG(x) − G(y)k ≤ Ckx − yk,

for all x, y ∈ D, where k · k is any norm on Rn . Then the function G has a unique
fixed point x∗ ∈ D.
Now let x(0) ∈ D be arbitrary and define the sequence {x(k) }∞
k=0 ⊂ D by
(k+1)
x := G(x(k) ). Then we have the estimates

kx(k+1) − x∗ k ≤ Ckx(k) − x∗ k,
Ck
kx(k) − x∗ k ≤ kx(1) − x(0) k,
1−C
C
kx(k) − x∗ k ≤ kx(k) − x(k−1) k.
1−C

In particular the sequence {x(k) }∞ ∗

k=0 converges to x .

2
Proof. Assume that x∗ and y∗ are two fixed points of G in D. Then

kx∗ − y∗ k = kG(x∗ ) − G(y∗ )k ≤ Ckx∗ − y∗ k.

From 0 ≤ C < 1, it follows that kx∗ − y∗ k = 0 and thus x∗ = y∗ . Thus the

fixed-point (if it exists is unique).
Now let let x(0) ∈ D be arbitrary and define inductively x(k+1) := G(x(k) ). Then
we have for all k, m ∈ N the estimate
m
X
(k+m) (k)
kx −x k≤ kx(k+j) − x(k+j−1) k
j=1
m
X
≤ C j−1 kx(k+1) − x(k) k
j=1
∞
X
≤ C j−1 kx(k+1) − x(k) k (4)
j=1
1
= kx(k+1) − x(k) k
1−C
C
≤ kx(k) − x(k−1) k
1−C
Ck
≤ kx(1) − x(0) k.
1−C
It is clear from (4) that {x(k) }∞
k=0 is a Cauchy sequence by letting q = k + m, and
r = k in the definition above. Thus this sequence has a limit x∗ ∈ Rn . In fact
x∗ ∈ D since D is closed, which implies that any Cauchy sequence in D has a
limit in D. Now note that the function G is by definition Lipschitz-continuous with
Lipschitz-constant C, which again shows that G is continuous. Thus
 
∗ (k)
x = lim x = lim x (k+1) (k)
= lim G(x ) = G lim x (k)
= G(x∗ ).
k→∞ k→∞ k→∞ k→∞

This shows that x∗ is a fixed point of G. The different estimates claimed in the
theorem now follow easily from the fact that G is a contraction and from (4) by
letting m → ∞. (The norm is a continuous mapping, so the left hand side of this
equation tends to kx∗ − x(k) k).

The next result gives a local condition on G for there to exist a set where
Banach’s fixed point theorem is applicable. First, recall that

3
Definition 5. A neighbourhood of a point x is a set D that contains an open set
containing x.

Theorem 6. Assume that G : Rn 7→ Rn is continuously differentiable in a neigh-

bourhood of a fixed point x∗ of G, and that there exists a norm k · k on Rn with
subordinate matrix norm k · k on Rn×n such that

kJG (x∗ )k < 1

where JG is the Jacobian of G. Then there exists a closed neighbourhood D of x∗

such that G is a contraction on D. In particular, the fixed point iteration x(k+1) =
G(x(k) ) converges for every x(0) ∈ D to x∗ .

Proof. Because of the continuous differentiability of G there exists ρ > 0 and a

constant 0 < C < 1 such that

kJG (x)k ≤ C for all x ∈ Bρ (x∗ ) := {y ∈ Rn : ky − x∗ k ≤ ρ}

Let now x, y ∈ Bρ (x∗ ). Then

Z 1
G(y) − G(x) = JG (x + t(y − x))(y − x)dt.
0

Thus
Z 1
kG(y) − G(x)k ≤ kJG (x + t(y − x))(y − x)k dt
0
Z 1
≤ kJG (x + t(y − x))kky − xk dt
0
Z 1
= kJG (x + t(y − x))k dt ky − xk (5)
0
≤ Cky − xk

In particular we obtain with y = x∗ the inequality

kx∗ − G(x)k = kG(x∗ ) − G(x)k ≤ Ckx∗ − xk ≤ Cρ < ρ.

Thus x ∈ Bρ (x∗ ) implies G(x) ∈ Bρ (x∗ ). This shows that G is in fact a contraction
on Bρ (x∗ ). The final claim of the theorem is a simple application of Banach’s fixed
point theorem.

4
 We now make some remarks regarding the spectral radius of a matrix A ∈ Rn×n .
The first is that every subordinate matrix norm on Rn×n satisfies the inequality
ρ(A) ≤ kAk.
This is a simple observation if the largest eigenvalue, i.e. λ such that |λ| = ρ(A), is
real. Then letting x be a corresponding normalized eigenvector such that kxk = 1
ρ(A) = |λ| = |λ|kxk = kλxk = kAxk ≤ kAkkxk.
Conversely you can show that for any  > 0 there exists a norm k · k on Rn such
that the subordinate matrix norm k · k on Rn×n satisfies
kAk ≤ ρ(A) + .
Together these results imply that there exists a subordinate matrix norm k · k on
Rn×n such that kAk < 1, if and only if ρ(A) < 1. Thus the main condition of the
previous theorem can be equivalently reformulated as
ρ(JG (x∗ )) < 1.

The following lemma gives a practical way of determining when and where a
differentiable function is a contraction, in order to apply Banach’s fixed point theo-
rem.
Lemma 7. Assume that G : D 7→ D, with D ⊂ Rn convex, is continuously differ-
entiable and let 0 ≤ C < 1. Let k · k be a norm on Rn with subordinate matrix norm
k · k on Rn×n . If
kJG (x)k ≤ C,
for all x ∈ D, then G is a contraction using this norm.

Proof. Choose any x, y ∈ D. The same computation as in (5) then shows that
kG(y) − G(x)k ≤ Cky − xk

Banach’s fixed point theorem then applies on D, provided D is closed. This is

usually the easiest method to prove that G is a contraction. We remark that in
n = 1 the criterion
kJG (x)k ≤ C
becomes |G0 (x)| ≤ C for some non-negative constant C < 1.

5
3 Convergence rate and order
We briefly mention two different ways to measure how fast an algorithm converges.

Definition 8. Let the sequence {x(k) }∞ n ∗ n

k=0 ⊂ R converge to x ∈ R . If there exists
a sequence {β (k) }∞
k=0 ⊂ R which converges to 0, and a positive constant C such that

kx(k) − x∗ k ≤ C|β (k) | for sufficiently large k

Then {x(k) }∞ ∗
k=0 is said to converge to x with rate of convergence O(β
(k)
).

Note: β (k) is usually chosen as a1k or k1a for some positive constant a.
A central concept is the order of convergence.

Definition 9. We say that the sequence {x(k) }∞ n

k=0 ⊂ R converges to x ∈ R
∗ n

with convergence order p ≥ 1, if x(k) → x∗ and there exists a positive constant

0 < C < +∞ (0 < C < 1 if p = 1) such that

kx(k+1) − x∗ k ≤ Ckx(k) − x∗ kp

We say that an iterative method on the form x(k+1) = G(x(k) ) has order p if the se-
quence {x(k) }∞ ∗ ∗
k=0 it generates converges to the solution x = G(x ) with convergence
order p.

We in particular mention the following cases:

• p = 1 is called linear convergence. Roughly the number of ”correct digits”

increases by a constant number.

• p = 2 is called quadratic convergence. Roughly the number of ”correct digits”

doubles in each iteration.

• p = 3 is called cubic convergence. Roughly the number of ”correct digits”

triples in each iteration.

If one aims for reasonable high accuracy, higher order methods are (all else being
about equal) very desirable.

6
4 Convergence order of fixed point methods
From Banach’s fixed point theorem, we are guaranteed (at least) linear convergence
for the fixed point iteration. Now let us return to fixed point iterations for the case
of n = 1. The following result tells us when we can expect higher convergence order.
Theorem 10. Assume that G : R 7→ R is p-times continuously differentiable with
p ≥ 1 in a neighbourhood of a fixed point x∗ of G. Furthermore assume that

0 = G0 (x∗ ) = G00 (x∗ ) = . . . = G(p−1) (x∗ ), if p ≥ 2,

G0 (x∗ ) < 1, if p = 1.

Then the fixed point sequence x(k+1) = G(x(k) ) converges to x∗ with (at least) order
p, provided that the starting point x(0) is sufficiently close to x∗ . If in addition

G(p) (x∗ ) 6= 0

this convergence order is precisely p.

Proof. First note that Theorem 6 shows that the fixed point sequence indeed con-
verges to x∗ for suitable starting points x(0) . A Taylor expansion of G at the fixed
point x∗ now shows that
p−1
∗ ∗
X G(s) (x∗ )
s G(p) (ξ) (k)
p
x (k+1) (k)
− x = G(x ) − G(x ) = x(k) − x∗ + x − x∗
s=1
s! p!

for some ξ between x(k) and x∗ . The sum on the left will be empty in the case p = 1.
Since G(s) (x∗ ) = 0 for 1 ≤ s ≤ p − 1, this further implies that

G(p) (ξ) (k) p

x(k+1) − x∗ = x − x∗
p!

Because G(p) is continuous, there exists C > 0 (with C < 1 for p = 1) such that

G(p) (ξ)
≤C
p!
for ξ sufficiently close to x∗ , Thus
p
x(k+1) − x∗ ≤ C x(k) − x∗ ,

7
and therefore the convergence order of the sequence {x(k) }∞
k=0 is (at least) p. If

G(p) (x∗ ) 6= 0

then again because G(p) is continuous, there exist K > 0 such that

G(p) (ξ)
≥K
p!
for ξ sufficiently close to x∗ . Thus
p
x(k+1) − x∗ ≥ K x(k) − x∗ ,

which implies that the convergence order cannot be higher than p. Thus the con-
vergence order is precisely p.

Note: From the proof of Theorem 10 we expect that close to the fixed point x∗

∗ G(p) (x∗ ) (k) p

x (k+1)
−x ≈ x − x∗ ,
p!
when
0 = G0 (x∗ ) = G00 (x∗ ) = . . . = G(p−1) (x∗ ) but G(p) (x∗ ) 6= 0.