Chapter-3: Solution of Nonlinear Equations

-We want to find x such that f(x)=0.

Example: f(x)=x-tan(x)=0
Example: x-a*sin(x)=b
-There may be many approximate solutions even though the exact solution is unique. (Because of
roundoff errors.)
4 3 2 4
Example: P 4 ( x)=x −4 x +6 x −4 x+1=(x−1)
-If you use marc-32, you will find many zeros in the interval [0.975, 1.035]
3-1 Bisection (Interval Halving) Method
-If f(x) is a continuous function on the interval [a,b] and if f(a)f(b)<0, then f(x) must have a zero in (a,b).
Bisection Method:
1- Compute c=0.5*(a+b)
2- If f(a)f(c)<0 then f(x) has a zero in [a,c] ⇒ b← c (assign c to b)
3- Else ⇒ a←c (assign c to a f (x) has a zero in [c , b])
4- Stop if f(c)=0. (c is the zero of f(x))
5- Go to step 1.
-It is quite unlikely that f(c) will be exactly 0 in the computer because of
roundoff errors.
Stopping Criteria (stop when one of them is satisfied)
1-The maximum number of steps (M)
2- |b−a|< δ δ is a positive number.
3- |f (c)|< ϵ ϵ is a positive number.
a r c b

a c b
|f (c )|< ϵ but |b−a|> δ
Criterion |b−a|< δ failure

|b−a|< δ but |f (c )|> ϵ The function is not continuous but

we can not verify in advance.
Criterion |f (c )|< ϵ failure
-Error Analysis
-Let us denote the successive intervals that arise in the process by [a 0 , b0 ] ,
[a 1 , b 1 ] and so on.
⇒ a 0 ≤a1 ≤a 2≤a 3 ≤…an ≤b0 , b 0≥b1 ≥b 2≥b 3≥…≥bn≥…≥a 0
1
and (b n+1−a n+1)= (bn−an) n≥0
2
-Since the sequence an is increasing and bounded above, it converges.
Likewise, bn converges.
−n
⇒(b n −a n)=2 (b 0−a0)
−n
⇒ lim (b n −a n )=lim 2 (b0 −a0 )=0
n→∞ n→ ∞
⇒ r=lim a n=lim b n
n →∞ n→∞

2
f (a n )f (b n)≤0 ⇒ lim f (a n )f (bn )=(f (r)) ≤0 ⇒ f (r )=0
n→∞
-The best estimate at the nth stage is not an or bn but the midpoint of the
interval cn =(an +b n)/2
 Example: 2∗x−tan(x) in the interval [0.5,1 .5] Find the root.
Solution:
a0=0.5, b0 =1.5 ⇒ c 0 =1, f (a0 )=0.45, f (b0 )=−11.1, f (c 0 )=0.44
a1=1, b1 =1.5 ⇒ c 1=1.25, f (a1 )=0.44, f (b1 )=−11.1, f (c 1 )=−0.51
a2=1, b2 =1.25 ⇒ c 2 =1.125, f (a2 )=0.44, f (b2)=−0.51, f (c 2)=0.16
a3=1.125, b3=1.25 ⇒ c 3=1.1875, f (a3 )=0.16, f (b3 )=−0.51, f (c 3 )=−0.1
a4 =1.125, b 4 =1.1875 ⇒ c 4 =1.15625, f (a4 )=0.16, f (b4 )=−0.1, f (c 4 )=0.04
a5=1.15625, b5 =1.1875 ⇒ c 5 =1.171875, f (a5 )=0.04, f (b5 )=−0.1, f (c 5 )=−0.02
a6=1.15625, b 6=1.171875 ⇒ c 6 =1.1640625, f (a6 )=0.04, f (b6 )=−0.02, f (c 6 )=0.0066118
 -Example: x−1−2x in the interval [0,1]
Solution:
a0=0, b0 =1 ⇒ c 0 =0.5, f (a0)=∞ , f (b0 )=−1, f (c 0 )=0.58
a1=0.5, b1=1 ⇒ c 1 =0.75, f (a1 )=0.58, f (b1 )=−1, f (c 1)=−0.35
a2=0.5, b2 =0.75 ⇒ c 2=0.625, f (a2 )=0.58, f (b2 )=−0.35, f (c 2)=0.058
a3=0.625, b3=0.75 ⇒ c 3 =0.6875, f (a3 )=0.058, f (b3)=−0.35, f (c 3 )=0.156
a4 =0.625, b 4 =0.6875 ⇒ c 4 =0.65625, f (a4 )=0.058, f (b4 )=0.156, f (c 4 )=−0.0522
a5=0.625, b5=0.65625 ⇒ c 5=0.640625, f (a5 )=0.058, f (b5 )=−0.0522, f (c 5 )=0.00197
a6=0.640625, b6 =0.65625 ⇒ c 6 =0.6484375, f (a6 )=0.00197, f (b6 )=−0.0522, f (c 6 )=−0.025
a7=0.640625, b7 =0.6484375 ⇒ c 7=0.6445313, f (a7 )=0.00197, f (b7 )=−0.025, f (c 7 )=−0.0117
3.2 Newton’s Method
Newton’s method is a general procedure that can be applied in many
diverse situations. When applied to the problem of locating a zero of a real-
valued function of a real variable, it is often called the Newton-Raphson
iteration. It is faster than the bisection and the secant method (convergence
is quadratic). Unfortunately, the method is not guaranteed always to
converge. Therefore, it is combined with other slower methods.
(2)
-Let r be a zero of f(x) and let x be an approximation to r. If f (x) exists and
continuous, then by Taylor’s Theorem,
2
0=f (r)=f (x)+hf ' (x)+O(h )
2 f ( x)
where h=r-x. If h is small, we can ignore O (h ) ⇒ h≃−
f ' (x)
f (x)
If x is approximation to r, then x− should be a better approximation to r.
f ' (x)
Newton’s Method:
-Begin with an estimate x0 of r and compute
f (x n)
x n +1=x n− n≥0
f ' (x n )
Graphical Interpretation

l (x)=−29+10 x

l (x) is the linearization of the

f (x n)
function f (x).
l (x) is a good approximation
r of f (x) if x close to c

xn +1=2.9 x n =5
(2) 2 (3) 3
f (x)=f (c)+ f ' (c)(x−c)+0.5 f (c)(x−c) +(1/6)f (c)(x−c) +…Taylor series ex.
l ( x)=f (c)+ f ' (c )(x−c) First Order Taylor series expansion.
⇒ l (x)=f (xn )+ f ' (x n )(x−xn ) c= xn
 x1=−1.53 r=0 x 0=1.2 x2=3.82

Example of nonconvergence of Newton’s method.

-For convergence of Newton’s method, x0 must be sufficiently close to zero,

or f (x) must have a prescribed shape (convex and increasing)
Error Analysis
(2)
Assume f is continuous, f (r)=0≠f ' (r)
f (x n ) f (x n) e n f ' (x n)−f (x n)
en +1 =x n+1−r=x n− −r=en− =
f ' (x n) f ' (x n) f ' (x n)

By Taylor’s Theorem, we have

1 2 (2)
0=f (r)=f ( x n −e n )=f ( x n )−e n f ' ( x n )+ en f ( ξn )
2
where ξ n is a number between x n and r .
1
⇒ en f ' (x n)−f (x n)= f (2) (ξn)e 2n
2
(2)
f
1 ( ξn) 2 1 f (r) 2
(2)
2
⇒ en+1= e n≃ e n=Ce n
2 f ' (xn) 2 f ' (r)
⇒ The rate of convergence is quadratic. Does e n converge to 0 ?
Choose δ small enough so that δ C (δ )<1. Having fixed δ , set ρ =δ C (δ )
Suppose we start iteration with a point x 0 satisfying |x 0−r|≤δ .
Then |e 0|≤ δ and |ζ 0−r|≤ δ
2
⇒|x 1−r|=|e1|≤e0 C (δ )=|e 0||e0|C (δ )≤|e 0|δ C (δ )=|e0|ρ <|e0|≤ δ
-If we repeat the argument
|e 1|≤ρ|e0|
2
|e 2|≤ρ|e1|≤ρ |e0|
3 n
|e 3|≤ρ|e2|≤ρ |e0| ⇒ |en|≤ρ |e0| →0.

-Theorem: Let f (2)(x) be continuous function and let r be a simple zero of

f(x). Then, there is a neighborhood of r and a constant C such that if
Newton’s method is started in that neighborhood, the successive points
become steadily closer to r and satisfy
|x n+1−r|≤c (x n −r)2 (n≥0)
-Theorem: If f(x) belongs to C 2 (R), is increasing, is convex, and has a zero,
then the zero is unique, and the Newton’s iteration will converge to it from
any starting point.
Proof:
(2)
f (x) is convex ⇒ f ( x)>0, increasing ⇒ f ' (x)>0.
(2)
f
1 ( ξn ) 2
⇒ e n+1= e n >0
2 f ' (xn )
⇒ xn >r for n≥1 (x n=e n +r)

Since f ( x) is increasing, f ( x n )>f (r)=0 (n≥1)

f ( xn )
e n+1=en − ⇒ en+1 <en ⇒ en is decreasing and bounded below by zero.
f ' ( xn )
⇒ lim en =0 ⇒ lim x n=r
n→∞ n→∞

Example: Find an efficient method for computing square roots based on the
use of Newton’s method.
2
Solution: Let x= √ R ⇒ x −R=0
2
Use Newton's method on the function f (x )=x −R ⇒ f ' ( x)=2 x .
f (x n ) x2n −R 1 R
⇒ the iteration formulea : x n+1=x n− =xn − = (x n + )
f ' (x n ) 2 xn 2 xn
Lets compute √ 17 using the Newton's method. Let x 0=4.
⇒ x 1=4.12, x 2=4.123106, x 3=4.1231056256177,
x 4=4.123105625617660549821409856,

Implicit Functions
G (x , y)=0 Find y values for the given x values.
G(x , y k )
⇒ y k +1= y k −
∂G
(x , y k )
∂y

-This method can be used to construct a table of the function y(x)

Example: Produce a table of x versus y for the function
7 5 3 3
G (x , y)=3 x +2 y −x + y −3=0

Solution: ∂G
(x , y)=10 y 4 +3 y2
∂y
7 5 3 3
3 x +2 y −x + y −3
⇒ y n+1= y n − 4 2
10 y +3 y
 i x y
−−−−− −−−−− −−−−−
0 0.0 1.000000
1 0.1 1.000077
2 0.2 1.000612
⋮ ⋮ ⋮
20 2.0 −2.810639
⋮ ⋮ ⋮
80 8.0 −19.92635
⋮ ⋮ ⋮
100 10.0 −27.23685
Systems of Nonlinear Equations
Newton’s method for systems of nonlinear equations follows the same
strategy that was used for a single equation.
Let f 1 (x1 , x 2)=0
f 2 (x1 , x 2)=0 find x 1 and x2
Suppose that (x 1 , x 2) is an approximate solution of previous equations. Let us
compute h 1 and h2 so that (x 1+h1, x 2+h2) will be a better approximate solution.
-Using only linear terms in the Taylor expansion, we get
∂f 1 ∂f1
0=f 1 (x 1 +h1 , x 2 +h2 )≃f 1 (x1, x2 )+h 1 +h2
∂ x1 ∂ x2
∂f2 ∂f 2
0=f 2 ( x1 +h1 , x2 +h2 )≃f 2 (x 1, x 2)+h1 +h2
∂ x1 ∂ x2
−1
∂f1 ∂f1 ∂f1 ∂f 1
h
[]
⇒ 1 =−
h2
∂ x1
∂f2
∂ x1
[ ] ∂ x2
∂f2
∂ x2
[ f 1 ( x1 , x 2 )
]
f 2 ( x1 , x 2 )
, J=
[ ]
∂ x1
∂f2
∂ x1
∂ x2
∂f 2
∂ x2

J is the Jacobian matrix of f 1 and f 2 .

Newton’s method fo two nonlinear equations in two variables is
−1
∂f 1 k k ∂f1 k k
k +1
x1
[ ][ ][ ]
k +1
x2
=
k
x 1 h1
k
+ k
x 2 h2
k
h
h2
k

[] [
where 1k =−J
−1
k

k
k
f 1 (x1 , x 2 )
]
k
f 2 (x1 , x 2 )
, J
−1
=
[∂ x1
( x1 , x 2 )

∂f 2 k k
∂ x1
( x1 , x 2 )
∂ x2
( x1 , x2 )

∂f2 k k
∂ x2
( x1 , x2 ) ]
-The system with more than two variables:
f i (x1, x 2, x 3, …, x n)=0 (1≤i≤n)
Let X =(x 1, x 2, x3, …, xn)T , F=(f 1, f 2, f 3, …, f n)T
Taylor expansion of F :
0=F (X + H )≃F (X )+ F ' (X) H , H =(h1, h2, h3, …, h n)T
∂fi
F ' (X) is the nxn Jacobian matrix J (X ) with elements ; namely ,
∂xj
∂f 1 ∂f 1 ∂f1

[ ]
…
∂ x1 ∂ x2 ∂ xn
F ' (X)= ⋮ ⋮ ⋮ ⋮
∂f n ∂f n ∂fn
…
∂ x1 ∂ x2 ∂ xn
⇒ H =−F ' (X )−1 F (X )
(k +1) (k) (k )
X =X +H
Example:
T
-Starting with (1,1,1) carry out six iteration of Newton’s method for finding of
the root of the nonlinear system:
xy=z +1
2 x 1=x , x 2= y , x3 =z
2 2 2
xyz+ y =x +2 x1 x 2 −x3 −1
x
e + z=e +3
Solution:
y

x2 x1 −2 x3
[ x 2 x
e −e + x 3 −3
1 ]
F ( X )= x1 x 2 x3 −x21 + x 22 −2

[
⇒ F ' ( X)= x2 x 3−2 x 1 x1 x3 +2 x 2 x1 x2
e
x1
−e
x2
1 ]
n x1 x2 x3
−−− −−−−−− −−−−−− −−−−−−
1 1.7547911 1.5363988 1.1455950
⋮ ⋮ ⋮ ⋮
6 1.7776717 1.4239606 1.2374710
3.3 Secant Method
f (x n)
Newton's method : x n+1 =x n− (uses derivative. Need to compute derivative)
f ' (x n)
2
f (x n)
Steffensen's iteration: x n+1=x n−
f ( x n +f (x n))−f (xn)
(Need to compute f (x n) and f ( x n + f (x n )) )
f (x n)−f (x n−1)
Secant method: Replaces f ' (x n) with f ' (x n)≃
x n−xn −1
xn −x n−1
[
⇒ x n +1=x n−f (x n)
]
f (x n)−f (x n−1)
To compute x n+1 , we need x n and xn−1 . However evaluation of each new x n+1
requires only one evaluation of f .
Error Analysis en=x n−r
f (x n) x n−1−f (x n−1) xn f (x n) en−1−f (x n−1)en
en +1=x n+1−r= −r=
f (x n)−f (x n−1) f (x n)−f (x n−1)
x n− xn−1 f (x n)/e n−f (x n−1)/en−1
=
[ ][
f (x n)−f (x n−1) x n−x n−1 ] e n en−1

By Taylor’s Theorem,
1
f ( x n )=f (r +e n )=f (r )+e n f ' (r )+ e2n f (2) (r)+O(e3n ), f (r)=0.
2
f (x n ) 1 (2) 2
⇒ =f ' (r)+ e n f (r)+O (e n ) Changing the index to n−1 yields
en 2
f ( x n−1) 1 (2) 2
=f ' (r)+ e n−1 f (r)+O (en−1)
en−1 2
1
⇒ f (x n )/ en−f (xn−1 )/e n−1= (en −en−1 )f (2) (r)+O (e 2n )−O(e2n−1), x n−x n−1=e n−en−1
2
f (x n)/e n−f ( xn−1)/e n−1 1 (2) (2)
1 f (r)
⇒
[ xn− xn−1 ] ≃ f (r)
2
⇒ e n+1 ≃
2 f ' (r )
e n e n−1 =C e n e n−1
What is the order of convergence?
α
Assume |e n+1|∼ A|en| where A is a positive constant
This means taht the ratio |e n+1|/( A|e n|n ) tends to 1 as n→∞ and implies α −order
convergence.
α −1 1/ α
⇒|en|∼ A|en−1| and |en−1|∼( A |en|)
1 1
α −1/ α 1/ α 1+ α −1 1− α + α
⇒|en+1|= A|e n| ∼|C||en|A |en| ⇒A |C| ∼ |en| →1
1
1+ α −1
⇒A |C| is a nonzero constant while e n →0 as n→∞ .
1
⇒ 1− α + α =0 ⇒ α =(1+ √ 5)/ 2≃1.62 ⇒ the rate of convergence is superlinear.
(2) 0.62
f (r )
A=|C|
1
1/(1+ α ) 1/ α α −1
=|C| =|C| =|C| =
0.62
|
2 f ' (r)|
⇒|en+ 1|≃ A|en|(1+ √5)/2 .
-The Secant method converges slower than the Newton’s method but
Secant method requires evaluation of only f(x). Newton’s method requires
evaluations of f(x) and f'(x) at every step.
3.5 Computing Zeros of Polynomials
n n−1 2
P(z)=a n z +an−1 z +…+a2 z + a1 z+a0
a k and the variable z may be comlex number or variables. If a n≠0 then P has
degree n . Find the zeros of P .

Theorem 1: (Fundamental Theorem of Algebra)

-Every non-constant polynomial has at least one zero in the complex field.
Let P is a nth degree polynomial.
⇒ P(z)=(z−c)q (z)+r where r is a complex number and q is a polynomial of
degree n−1.
P(c)=r . If r=0 then c is a zero of P and we have P (z)=(z−c)q(z).
Thus (z−c) is a factor of P (z).

Theorem : A polynomial of degree n has exactly n zeros in the complex

plane (consequence of Fundamental Theorem of Algebra)
P(z)=(z−r 1)(z−r 2)…(z−r n)q n . where q n is a constant.
Theorem: All zeros of the nth degree polynomial lie in the open desk
whose center is at the origin of the complex plane and whose radius is
ρ =1+|an|−1 max |a k|
0≤k <n

Proof:
Let c= max |a k| so that c|an|−1=ρ−1. If c=0 our result is trivially true.
0≤k <n
Hence, assume c >0. Then ρ >1. If |z|≥ρ If d=e−f ⇒ |d|≤|e|+|−f|=|e|+|f|
n n−1 n n−1 n n
|P(z)|≥|a n z |−|a n−1 z +…+a1 z+a 0|≥|a n z |−c ∑ |z|k >|a n z |−c|z |(|z|−1)−1
k =0
n n
=|an z |{1−c|a n|−1 (|z|−1)−1 }≥|a n z | {1−c|an|−1 (ρ −1)−1 }=0 n−1 k 1−a
n
∑ a =
⇒ If |z|≥ρ |P(z)|>0 ( there is no zero if |z|≥ρ ) k =0 1−a

Example: Find a disk centered at the origin that contains all the zeros of the
4 3 2
polynomial P(z)=z −4 z +7 z −5 z−2
Solution:
ρ=1+|a 4|−1 max |a k|=8 ⇒|z i|<8 where |zi|'s are the zeros of P(z)
0≤ k <4
 n n −n −(n−1) −2 −1
Let S (z)=z P (1/ z)=z [ a n z +an−1 z +…+a 2 z +a1 z +a 0]
=a n +a n−1 z+a n−2 z 2 +…+ a2 z n−2+a 1 z n−1 +a0 z n
The condition P (z 0)=0 is equivalent to the condition S (1/ z 0)=0
Theorem: If all the zeros of S( z) are in the disk {z :|z|≤ρ }, then all the nonzero
zeros of P(z) are outside the disk {z :|z|≤ρ−1}.
Example: P(z)=z 4 −4 z 3+7 z 2−5 z−2 Find a ring that all zeros of P(z) lie in it.
Solution:
ρ p=1+|a 4|−1 max |a k|=8
0≤k < 4

S(z)=1−4 z+7 z −5 z 3−2 z 4

2

7 9
⇒ ρ s=1+|a 4|−1 max |ak|=1+ =
0≤k <4 2 2
1 2
⇒ ρ <|z i|< ρ p . ⇒ <|z i|<8 where |zi|' s are zeros of P(z).
s 9
Horner’s Algorithm
If a polynomial p and a complex number z 0 are given, Horner's algorithm will
p(z)− p(z0)
produce the polynomial q(z)= .
z− zo
The degree of polynomial q is one less than the degree of p.
p (z)=(z− z0) q(z)+ p(z0)
n −1
Let q be represented by q(z)=b 0 +b1 z+…+bn−1 z
⇒ bn−1=a n
bn−2=a n−1+ z 0 bn−1
⋮ ⋮ ⋮
b 0=a 1+ z 0 b1
p(z 0)=a0 + z 0 b0

an an −1 a n−2 a n−3 … a1 a0
z0 z o bn−1 z 0 b n−2 z 0 b n−3 … z0 b1 z 0 b0
----------------------------------------------------------------------------------------
b n−1 b n−2 b n−3 b n−4 … b0 b−1

b−1= p (z 0 )
Example: P(z)=z 4 −4 z 3 +7 z 2−5 z−2. Evaluate p (3) using Horner's algorithm.

1 −4 7 −5 −2
3 3 −3 12 21
----------------------------------------------------
1 −1 4 7 19
⇒ p(3)=19
3 2
⇒ p( z)=(z−3)(z −z +4 z+7)+19
-Horner’s algorithm is also used for deflation (factorization)
If z 0 is zero of p then p(z)=(z−z 0)q (z)

Example: Deflate the polynomial P(z)=z 4 −4 z 3 +7 z 2−5 z−2 using the fact
that 2 is one of its zero.
Solution:
1 −4 7 −5 −2
2 2 −4 6 2
----------------------------------------------------
1 −2 3 1 0
4 3 2 3 2
⇒ z −4 z +7 z −5 z−2=( z−2)(z −2 z +3 z+1)
-Horner’s algorithm can be used in finding the Taylor expansion of a
polynomial about any point.
n n−1 2 n n−1
P(z)=a n z +a n−1 z +…+a 2 z +a1 z+a0 =c n (z−z 0) +c n−1 ( z−z 0) +…+c 0
(k )
p (z 0)
⇒ ck =
k!
-If we apply the Horner's algorithm to the p with z 0
p(z)− p( z0 )
q(z)= =cn (z−z 0)n−1+ cn−1 (z−z 0 )n −2 + … +c 1
z− z0
⇒ c 1=q(z 0 )
This process is repeated until all coefficients c k are found.
Example:
4 3 2
Find the Taylor expansion of p( z)= z −4 z +7 z −5 z−2 about the point z0 =3.
Solution:
1 −4 7 −5 −2
3 3 −3 12 21
----------------------------------------------------
1 −1 4 7 19
3 3 6 30
----------------------------------------------------
1 2 10 37
3 3 15
----------------------------------------------------
1 5 25
3 3
----------------------------------------------------
1 8
4 3 2
⇒ p( z)=(z−3) +8( z−3) +25(z−3) +37(z−3)+19
-Newton’s iteration can be carried out on a polynomial.
f (z k )
z k +1=z k − We know c 0= p(z 0), c 1= p ' (z 0)
f ' (z k )
Example:
4 3 2
p( z)= z −4 z +7 z −5 z−2 Start with z 0 =0 and find a zero of the polynomial

1 −4 7 −5 −2
0 0 0 0 0
----------------------------------------------------
1 −4 7 −5 −2
0 0 0 0
----------------------------------------------------
1 −4 7 −5
⇒ c 0=−2= p(z 0 ) c1=−5= p ' ( z0 )
p(z 0 ) −2
⇒ z1 = z 0 − =0− =−0.4
p ' (z0 ) −5
-If we Execute the algorithm on a computer, we get
k p (z k ) p ' (z k ) zk
-----------------------------------------------------
1 −2 −5 −0.4
⋮ ⋮ ⋮ ⋮
5 0.00000 −9.85537 −0.27568
Theorem: Let z k and z k+1 be two successive iterates when Newton's method is
applied to a polynomial p of degree n . Then there is a zero of p within distance
n|z k −z k+1| of z k in the complex plane.
n

Proof: Let r 1 , r 2 , r 3, …, r n be the zeros of p then p ( z)=c ∏ ( z−r j )

j=1
n n n n
p ( z) −1 p ( zn)
p ' ( z)=c ∑ ∏ ( z−r i )=∑ = p ( z) ∑ ( z−r k ) , z n+1=z n −
k=1 i=1 k=1 z−r k k =1 p ' ( zn)
i≠k

We want to show that for any z( z k ) there is an index j for which

|z−r j|≤n|p ( z)/ p ' ( z)|. If no index j satisfies the desired inequality, then for all j
|z−r j|>n|p ( z)/ p ' ( z)|.
n n
1 1 1
|z−r j| < |p ' ( z)/ p( z)|= |∑ ( z−r k )−1|≤ ∑ |z−r k|−1
−1
n n k=1 n k =1
But this is not possible because average of n numbers cannot be greater than each
of them.
-Laguerre Iteration
Converge faster than Newton’ method (third-order convergence
A=− p ' ( z)/ p( z)
2 (2)
B= A − p (z)/ p(z)
C=n−1 [ A±√ (n−1)(nB− A 2) ]
znew =z+1/C

Theorem: İf p is a polynomial of degree n, if z is any complex number, and

C is computed as in Laguerre’s algorithm, then p has a zero in the complex
plane within distance √ n/C of z .
CHAPTER-4 Solving Systems of Linear Equations
-We want to solve the systems of linear equations having the form
a 11 x 1 +a 12 x2 +a 13 x3 + … +a 1 n xn =b1
⋮ ⋮ ⋮ ⋮ ⋮ ⋮
a n 1 x1 +a n 2 x2 +a n 3 x3 + … +a nn x n =b n in matrix form:
a 11 a 12 a 13 … a 1 n x 1 b1

[⋮ ⋮ ⋮
][ ] [ ]
⋮ ⋮ ⋮ =⋮
a n 1 a n 2 a n 3 … a nn x n bn
⇒ AX=b

X b
A

-A 1xn matrix is called a row vector.

-An mx1 matrix is called a column vector.
-If A is a matrix, the notation aij, (A)ij, or A(i,j) is used to denote the element
at the intersection of the ith row and jth column.
T
The transpose of a matrix is denoted by A and is the matrix defined by
T
( A )ij =a ji
 T
-If A = A , we say that A is symetric.
-If A is a matrix and λ is a scalar, then λA is defined by (λA)ij= λaij. If A=(aij)
and B=(bij) are mxn matrices then (A+B) is defined by (A+B)ij=aij+bij.
-If A is a mxp matrix, B is a pxn matrix, then AB is an mxp matrix defined by
p
( AB)ij= ∑ a ik b kj ( 1≤i≤m , 1≤ j≤n )
k =1
-Let Ax=b , Bx=d
-If two systems have precisely the same solution, we call them equivalent
systems. Thus, to solve a system a system of equations, we can instead
solve any equivalent system.
Elementary Operations:
i) Interchanging two equations in the system : E i ⇔ E j
ii) Multiplying an equation by a nonzero number : λ E i → E i
iii) Adding to an equation a multiple of some other equation : E i + λ E j → E i
Theorem: If one system of equations is obtained from another by a finite
sequence of elementary operations, then two systems are equivalent.
-Matrix Properties:
1 0 0 … 0

[
The nxn matrix I = 0 1
⋮ ⋮
0 0
IA= A = AI for any matrix
0
⋮
0
A
⋮ ⋮
… 1
]
… 0 is called an identity matrix.

of size nxn .
-If AB=I, then we say that B is a right inverse of A and A is a left inverse of
B.
1 0
[ 1 0 0
1 1 0 ][ 0 1
α β][ =
1 0
]
0 1
= AB

A is the left inverse of B and B is the right inverse of A

Theorem: A square matrix can poses at most one right inverse.
Theorem: If A and B are square matrices such that AB=I, then BA=I.
Proof: Let C=BA-I+B then AC=ABA-AI+AB=A-A+I=I, thus C is a right
inverse of A. By the previous theorem B=C; hence, BA=I ⇒ BA= AB=I .
We then call B as the inverse of A and say that A is invertible or nonsingular.
 −1
-If A is invertible, then Ax=b has the solution x= A b
Elementary Matrix Operations
-An elementary matrix is defined to be nxn matrix that arises when an
elementary operation is applied to the nxn identity matrix.
Elementary Operations (Matrix)
i) The interchange of two rows in A : A s At
ii) Multiplying one row by a nonzero constant λ A s → A s
iii) Adding to one row a multiple of another: A s + λ A t → A s
Examples:
1 0 0 a 11 a12 a13 a 11 a12 a 13

[ ][ ][
0 0 1 a 21 a 22 a23 = a 31 a32 a 33
0 1 0 a 31 a32 a33 a 21 a22 a 23 ] A2 A3

1 0 0 a 11 a12 a 13 a 11 a12 a 13

[ ][ ][
0 λ 0 a 21 a22 a 23 = λ a21 λ a 22 λ a23
0 0 1 a 31 a32 a 33 a 31 a32 a 33 ] λ A 2→ A 2

1 0 0 a 11 a12 a 13 a 11 a 12 a13

[ ][ ][
0 1 0 a 21 a22 a 23 = a 21
0 λ 1 a 31 a32 a 33
a 22 a 23
λ a21 +a 31 λ a22 +a 32 λ a23 +a 33 ] A3 + λ A 2 → A3

Elementary matrices
-If we apply elementary row operations m times to A, we get
E m E m−1 E m−2 …E 2 E1 A
If A is invertible E m E m−1 E m−2 … E 2 E 1 A=I ( possible )
−1
⇒ A = E m E m−1 E m−2 … E 2 E 1
Example
1 2 3 1 0 0 1 2 3 1 0 0
[ ] [ ]
A= 1 3 3 ,
2 4 7
E 1= −1 1 0
0 0 1 [ ] [ ]
⇒ E1 A = 0 1 0 ,
2 4 7
E 2= 0 1 0
−2 0 1
1 2 3 1 −2 0 1 0 3
[ ] [
⇒ E 2 E1 A =0 1
0 0
0 ,
1
E 3= 0
0 0 1]
1 0 ,
[ ]
⇒ E 3 E 2 E 1 A= 0 1 0
0 0 1
1 0 −3 1 0 0
[ ]
E4 = 0 1
0 0
0 ,
1 [ ⇒ E 4 E 3 E 2 E 1 A= 0 1

1 0 −3 1 −2
0 0 ]0
1
0 1 0 0 1 0 0
−1

[ ][
⇒ A = E4 E3 E2 E1 I = 0 1
0 0
0 0
1 0
1
0 ][ ] [ ]
0 0 1 0 −1 1 0
1 −2 0 1 0 0 1
9 −2 −3
[
= −1
−2
1
0 ]
0
1
4.2 LU and Cholesky Factorizations
a 11 a12 a13 … a1 n x1 b1

[ a 21
a 31
⋮
an 1
a22
a32
⋮
an 2
a23
a33
⋮
an 3
…
…
⋮
…
a2 n
a3 n
⋮
ann
][ ] [ ]
x2

⋮
xn
b2
x 3 = b3
⋮
bn
⇒ AX=b
−1
⇒ X= A b

If A is a diagonal matrix

a 11 0 0 … 0 x1 b1 b 1 /a11

[ 0
0
⋮
0
a22
0
⋮
0
0
a 33
⋮
0
…
…
⋮
…
0
0
⋮
a nn
][ ] [ ] [ ]
x2

⋮
xn
b2
x 3 = b3
⋮
bn
b 2 /a 22
⇒ X = b /a
⋮
3 33

b n /a nn

If aii =0 for some index i , and if b i=0, then x i can be any real number. If a ii =0,
no solution of the system exists.
If A is a lower triangular matrix
b1
x 1=
a 11 0 0 … 0 x1 b1 a11

[ a 21
a 31
⋮
an 1
a 22
a 32
⋮
an 2
0
a 33
⋮
an 3
…
…
⋮
…
0
0
⋮
a nn
][ ] [ ]
x2 b2
x3 = b3
⋮
xn bn
⋮
x 2=

x i=
−a 21 x1 +b 2
a 22
i−1
bi −∑ aij x j
j=1
aii
If A is a upper triangular matrix
bn
x n=
a 11 a 12 a 13 … a1 n x1 b1 a nn

[ 0
0
⋮
0
a 22
0
⋮
0
a 23
a 33
⋮
0
…
…
⋮
…
a2 n
a3 n
⋮
a nn
][ ] [ ]
x2 b2
x3 = b3
⋮
xn bn
⋮
x n−1=

x i=
b n−1−a n−1 , n x n
a n−1, n−1
n
bi − ∑ a ij x j
j=i+1
a ii
-There is still another simple type of systems.We want
Example:
a11 a12 0 x1 b 1 a 31 0 0 x1 b 3

[ ][ ] [ ] [
a21 a22 a23 x2 = b 2
a31 0 0 x3 b 3 ][ ] [ ]
⇒ a 11 a12 0 x2 = b 1
a 21 a22 a23 x3 b 2

We need to sove this system not in the usual order 1, 2, 3 but rather in the
order 3, 1, 2.
-Assume p1 is the row of A that has zeros in positions 2,3,...n
p2 is the row of A that has zeros in positions 3,4,...n
:
pn is the row of A that has zeros in positions n

If we reorder rows of A as p 1 , p2 ,… p n , the resulting matrix would be lower

triangular. ( p 1, p2 ,… p n ) is called permutation vector.
-LU Factorization
Suppose A=LU where L is the lower triangular matrix and U is the upper
triangular matrix.
Ax=b ⇒ LUx=b . Let Ux=z
⇒ Ax=LUx=Lz=b ( L is known and b is known.) We can find z .
Then Ux=z ( U and z are known.) We can find x .

l11 0 0 … 0 u 11 u12 u 13 … u1 n a 11 a12 a13 … a1 n

Suppose A=LU = l
⋮ [
l 21
31

l n1
l 22
l 32
⋮
l n2
0
l 33
⋮
ln 3
…
…
⋮
…
0
0
⋮
lnn
][ 0
0
⋮
0
u22
0
⋮
0
u 23
u 33
⋮
0
…
…
⋮
…
⋮ ][
u 2n a 21
u3 n = a 31
⋮
u nn a n1
a22
a32
⋮
a n2
a23
a33
⋮
an3
…
…
⋮
…
a 2n
a3 n
⋮
a nn
]
When this is possible, we say that A has an LU decomposition.
-L and U are not uniquely determined.
If we choose l ii =1 L is called unit lower triangular matrix. If we choose uii=1
U is called unit upper triangular matrix.
n min(i , j)
aij = ∑ lis u sj = ∑ l is u sj
s=1 s=1
-Each step in this process determines one new raw of U and one new
column in L.
At step k, we assume rows 1,2,3,…,k-1 have been computed in U and
columns 1,2,3,…,k-1 have been computed in L.
k−1
If i= j=k ⇒ a kk = ∑ l ks u sk +l kk ukk ( l kk or u kk can be cumputed using this formulea)
s=1
k−1
a kj = ∑ l ks u sj +l kk u kj ( k +1≤ j≤n ) u kj can be computed if l kk ≠0
s=1
k −1
aik = ∑ lis u sk +l ik ukk ( k +1≤i≤n ) l ik can be computed if u kk ≠0
s=1
If we choose l ii =1, the algorithm is called as Doolittle's factorization. If we choose
T
u ii=1, the algorithm is called as Crout's algorithm. When U =L , l ii =uii , the
algorithm is called Choelsky's factorization. ( A should be real, symmetric
and positive definite)
Example: Find the Doolittle, Crout, and Choelsky factorizations of the matrix
60 30 20 l 11 0 0 u11 u12 u13

[
A = 30 20 15
20 15 12 ] [L= l 21 l22 0
l 31 l32 l33 ] [ U= 0
0
u22 u23
0 u33 ]
a) Dolittle's algorithm (l11 =l22 =l 33=1)
⇒ l 11 u11 =a11 =60 ⇒ u11 =60
⇒ l 11 u12=a 12=30 ⇒ u12=30 , ⇒ l 11 u 13=a 13=20 ⇒ u13 =20
20 1
⇒ l 21 u 11=a 21=30 ⇒ l 21=0.5 , ⇒ l31 u11 =a31=20 ⇒l 31= =
60 3
⇒ l 21 u 12+l 22 u 22=a22 =20 ⇒ u22=20−l 21 u12 =20−15=5
⇒ l 21 u 13 +l 22 u 23=a 23=15 ⇒ u23 =15−0.5∗20=5
1
⇒ l 31 u12 +l 32 u22 =a32=15 ⇒ l 32=(15− 30)/5=1
3
20 1
⇒ l 31 u13 +l 32 u23 +l 33 u33 =a33 =12 ⇒ u33 =(12− −5)=
3 3
 1 0 0 60 30 20
[
⇒ A=LU = 0.5 1 0 0 5 5
][
1/3 1 1 0 0 1/3 ]
60 0 0 1 0.5 1/3 √ 60 0 0 √ 60 0 0 1 0.5 1/3
[
⇒U = 0 5 0 0 1
0 0 1/3 0 0][ 1 =0
1 0
][
√5 0 0
0 √ 3/3 0
√5 0 0 1
0 √ 3/3 0 0 ][1
1 ][ ]
60 0 0 1 0.5 1/3
[
⇒ A= 30 5 0
][
0 1
20 5 1/3 0 0
1
1 ] Crout's factorization

1 0 0 √ 60 0 0 √ 60 0 0 1 0.5 1/3
[
⇒ A= 0.5 1 0 0
1/3 1 1 0 ][ √5 0 0
0 √ 3/3 0 ][
√5 0 0 1
0 √ 3/3 0 0
1
1 ][ ]
√ 60 0 0 √ 60 √ 15 √ 60/3
[
= √ 15 √5 0 0
√ 60/3 √ 5 √3/3 0 ][√5 √5
0 √ 3/3
= ^
L
]
^
L
T
Choelsky's factorization
Theorem: If all n leading principle minors of the nxn matrix A are
nonsingular, then A has a LU decomposition.

Choelsky Factorization
-If A is a real, symmetric and positive definite matrix, then it has a unique
factorization, A= L LT , in which L is lower triangular with a positive diagonal.
T T
( A is symmetric if A= A . A is positive definite if x A x >0 for every nonzero
vector x .
k−1
2 2
a kk = ∑ l ks +l kk
s=1
k−1
a ik = ∑ l is k ks +l ik l kk ( k+1≤i≤n )
s=1
4.3 Pivoting and Construction an algorithm
-Basic Gaussian Elimination
6 −2 2 4 x1 12
Example

First step:
[ 12 −8
3 −13
−6 4
6
9
][ ] [ ]
10 x 2
3 x3
1 −18 x 4
=
34
27
−38

-Substract 2 times the first eqution from the second

-Substract 1/ 2 times the first eqution from the third
-Substract −1 times the first eqution from the fourth
-The numbers 2,1/2,−1 are called multipliers for the first step.
6 (first row first column) is called the pivot element for this step.
In the first step, row-1 is called the pivot row.

6 −2 2 4 x1 12

[ 0 −4
0 −12
0 2
2
8
][ ] [ ]
2 x2
1 x3
3 −14 x 4
=
10
21
−26
-Next step: (row 2 is used as the pivot row, -4 is the pivot element.)
-Substract 3 times the second row from the third row.
-Substract -0.5 times the second row from the fourth row.
-The multipliers: 3, -0.5
6 −2 2 4 x1 12

[ 0 −4 2 2
0
0
0 2 −5
0 4 −13
][ ] [ ]
x2
x3
=
10
−9
x 4 −21

-Final step: (row 3 is the pivot row, 2 is the pivot element.)

-Substract 2 times the third row from the fourth row.
-The multiplier: 2.

6 −2 2 4 x1 12
⇒
[0
0
0
−4 2 2 x2
0 2 −5 x3
0 0 −3 x 4
][ ] [ ]
= 10
−9
−3
Equivalent to the original system.
 x1 1 1 0 0 0 6 −2 2 4
x
x3
x4
[][ ] [
⇒ x= 2 =
−3
−2
1
L=

Multipliers at the first step

2
0.5
1
3
−1 −0.5
0
1
2
0
0
1
] [
U=
0 −4 2 2
0 0 2 −5
0 0 0 −3
] ⇒ A= LU

Multipliers at the second step

Multiplier at the third step
It is easy to show that A=LU Let u1 , u 2 , u3 ,u 4 be the first, second, third and fourth
rows of U and A 1 , A 2 , A 3 , A 4 be the first, second, third and fourth rows of A ,
respectively.
⇒ U 2= A 2−2 A 1 , A 1=U 1
⇒ A 2 =U 2 +2 A 1 =U 2 +2U 1=2U 1 +U 2
⇒ U 3=( A 3−0.5 A 1)−3 U 2 ⇒ A 3=0.5 A 1+3 U 2+U 3=0.5U 1 +3 U 2 +U 3
⇒ U 4=(A 4 + A 1)+0.5 U 2−2 U 3 ⇒ A 4=−U 1−0.5U 2 +2U 3 +U 4

Theorem: If all the pivot elements are nonzero in the process of Basic
Gauss Elimination, then A=LU.
Pivoting
-Basic Gauss Elimination method sometimes fail.
Example: 0 1 x1 1
[ ][ ] [ ]
=
1 1 x2 2

-Fails. Because there is no way of adding a multiple of the first equation to

the second in order to obtain 0 coefficient of x1 in the second equation.
Example:
ϵ 1 x1 = 1
[ ][ ] [ ]
1 1 x2 2
ϵ is a small number. FAILS
−1
2−ϵ
ϵ 1 x1 1 x2 =
⇒
[ 0 1− ϵ
−1 ][ ] [
=
x2 2−ϵ
−1] 1− ϵ
−1
≃1
−1
x 1 =(1−x 2 ) ϵ ≃0
−1 −1
In computer, if ϵ is small enough, 2− ϵ will be computed to be the same as −ϵ .
1 1−2 ϵ
The correct solution : x1 = ≃1, x 2 = ≃1.
1− ϵ 1− ϵ
The problem is not the smallness of a 11 . Rather, it is the smallness of a 11 relative
to the elements in its row.
Example:
1 ϵ
−1
x1
[ ][ ] [ ]
−1
= ϵ ϵ is a small number. Simple Gaussian algorithm produces:
1 1 x2 2
−1
−1 2− ϵ
1 ϵ x1
[ ][ ] [ ] x 2=
−1
ϵ ≃1
= −1 ⇒ 1− ϵ
−1
0 1− ϵ
−1
x2 2− ϵ
x 1= ϵ−1− ϵ−1 x 2≃0 wrong.
If we change the order of equatioms:
1 1 x1 2 1 1 x1 2
[ ][ ] [ ] [
ϵ 1 x2
=
1
⇒
0 1−ϵ x 2 ][ ] [
=
1−2 ϵ ]⇒ x 2=1, x 1=1.

1 1 x1 2 1 1 x1 2
[ 1 ϵ
−1
x2][ ] [ ] [
= −1
ϵ
⇒ −1 ][ ] [
0 ϵ −1 x 2
= −1
ϵ −2 ] ⇒ x 2=1, x 1=1.
-For the reason of economy in computing, we prefer not to move the rows of
the matrix around in the computers’s memory. Instead, we simply choose
the pivot rows in a logical manner. Instead of using rows in the order 1, 2, 3,
…,n-1 as pivot rows, we use p1, p2, p3,…,pn-1.
-In the first step multipliers of row p1 will be subtracted from rows p2,...,pn
-In the next step multipliers of row p2 will be subtracted from rows p3,...,pn-1 ;
and so on.
Gaussian Elimination with Scaled Row Pivoting
(Avoids choosing small pivot elements compared to other elements of the
pivot row)
The algorithm consists of two parts: a factorization phase and a solution
phase.
Factorization Phase:
Ax=b PAx=Pb Permutation matrix P derived from the permutation array p .
⇒ PAx=LUx=Pb Let Ux=z ⇒ Lz=Pb (Find z) ⇒ Ux=z (Find x)
Factorization Phase:
si =max|a ij|=max {|ai 1|,|a i 2|,|a i3|,…,|a i n|} (1≤i≤n)
1≤ j≤n
 |a i 1|
First step: Select the row as the pivot row for which is largest.
si
Set the permutation vector ( p 1 , p2 ,…, p n ) to (1, 2,3,…, n ). Then select an index
|a p , 1|
j for which j
is maximal and interchange p1 with p j in the permutation array
spj

a p ,1
p . Then substract i
times p1 from row pi forr 2≤i≤n
a p ,1
1

kth step: Find an index j for which |a p j k|/s p j is maximal. Interchange p k with
p j in the array p , and then substract a p k /a p k times row p k from row pi for
i k

k +1≤i≤n .
Example: Find PA=LU using Gaussian Elimination with scaled row pivoting.
2 3 −6 p=(1,2,3) and s=(6,8,3)
[
A = 1 −6 8
3 −2 1 ] ⇒|a pi 1|/ s pi={2/6 ,1/8 ,3/3}
⇒ row 3 is pivot row. ⇒ p=(3,2,1)

2/3 13/3 −20/3 ⇒|a p2 2|/s p2 =(16/3)/8=16 /24

[
A 1= 1/3 −16/3 23/3
3 −2 1 ] ⇒|a p3 2|/ s p3 =(13/3)/6=13/18⇒(13/18)>(16/24)
⇒ row 1 is pivot row. ⇒ p=(3,1,2)

2/3 13/3 −20/3

3 [
A 2= 1/3 −16 /13
−2
−7/13
1 ]
1 0 0 3 −2 1 3 −2 1
⇒ PA= 2/3
[1
1/3 −16 /13
0 0
1 0 ][ 13/3
0 ][
−20 /3 = 2 3 −6
−7 /13 1 −6 8 ]
-We can use the same algorithm to find the inverse of a matrix. Apply the
elementary matrix operations to I.(E n E n−1 E n −2 … E 2 E 1 I = A−1)
-We want