2022T1 MATH2089 – Numerical Methods Lectures Frances Kuo

§3 Least Squares and Polynomial Interpolation

Linear least squares
9
Motivation. 8

Data fitting by least squares 7

6
Lots of data with possible measurement
5
errors 4

Take line or curve of “best fit” 3

1
−2 0 2 4 6 8 10 12

Polynomial interpolation
Approximating a complicated function by 20

polynomial interpolation 0

Assume function values are known exactly −20

Fit polynomial through data points exactly −40

Benefit: easy to evaluate, differentiate, −60

and integrate −80

−100
−3 −2 −1 0 1 2 3 4

Key concepts.
Least squares: overdetermined linear system Au = y (A is m×n with m > n)
Method 1: normal equations (A⊤ A) u = A⊤ y
Cholesky or LU fact. + forward & back subs. = O(m n2 ) flops
Issue: squaring condition number
Method 2:
QR factorization + back substitution = O(m n2 ) flops
Benefit: numerically preferable
Matlab backslash command: u = A \ y
Polynomial interpolation:
Standard representation: p(x) = a0 + a1 x + a2 x2 + · · · + an xn
Lagrange polynomials: p(x) = y0 ℓ0 (x) + y1 ℓ1 (x) + · · · + yn ℓn (x)
Equally spaced points – large error near end points
Chebyshev points – minimize the error

References. Rao 5.1 – 5.5, 5.9, 5.10

MATLAB M-files. exlsq1.m exlsq2.m exqr.m expolint.m excheb.m

lagpol.m horner.m

1
Data fitting
Given data points (xj , yj ) for j = 1, . . . , m
Consider a polynomial of degree n − 1 =⇒ n unknown coefficients
y(x) = a0 + a1 x + a2 x2 + · · · + an−1 xn−1

m equations and n unknowns:

y(xj ) = yj , j = 1, . . . , m


 a0 + a1 x1 + a2 x21 + · · · + an−1 xn−1
1 = y1
a0 + a1 x2 + a2 x22 + · · · + an−1 xn−1 = y2


2
=⇒ ..


 .

a0 + a1 xm + a2 x2m + · · · + an−1 xn−1 = ym

m
 
 2 n−1
  y1
1 x1 x1 · · · x1 a0
1 x2 x2 · · · xn−1  a1   y2 
 
2 2
=⇒  . . . .  ..  =  ..  =⇒ Au = y
    
 .. .. .. . .. ..  .   . 
 
1 xm x2m · · · xn−1 m an−1
A u
ym
y

m = n =⇒ polynomial interpolation: fit a curve through the data exactly

Polynomial interpolation Linear least squares
20 9

8
0

−20
6

−40 5

4
−60

−80
2

−100 1
−3 −2 −1 0 1 2 3 4 −2 0 2 4 6 8 10 12

m > n =⇒ linear least squares: find a curve of best fit to the data by
minimizing the sum of squared residuals
m m
X X
2
rj2 := y(xj ) − yj = krk22 = kAu − yk22
j=1 j=1

It is called “linear” least squares because the parameters a0 , . . . , an−1 to

be solved enter linearly.
Matlab u = A \ y for both!

2
Example. Use Matlab to find the line of best fit y(x) = α + β x to the data
j 1 2 3 4 5
xj 0 0.5 1 1.5 2
yj 1.0 0.6 0.4 0.1 0.08
Answer. In this case, we have m = 5, n = 2, and the linear system Au = y, with
   
1 x1 y1
1 x2    y2 
  α  
A= 1 x3 ,
 u= , y= y3  .

1 x4  β y4 
1 x5 y5
xdat = [0:0.5:2];
ydat = [1.0 0.6 0.4 0.1 0.08];
xdat = xdat(:);
ydat = ydat(:);
A = [ones(size(xdat)) xdat];
u = A \ ydat

u =
0.9040
-0.4680
Thus α = 0.904 and β = −0.468. The line of best fit is y(x) = 0.904 − 0.468 x.
x = linspace(0,2,100);
y = u(1) + u(2)*x;
plot(x,y,’b-’,xdat,ydat,’m*’);

Line of best bit

1.2

0.8

0.6

0.4

0.2

−0.2
0 0.5 1 1.5 2

3
Example. Use Matlab to find the quadratic of best fit y(x) = α + βx + γ x2 to
the data in the previous example.
Answer. In this case, we have m = 5, n = 3, and the linear system Au = y, with

1 x1 x21
   
y1
2
 
1 x2 x2  α y2 
2
   
A= 1 x3 x3 , u =  β  , y = y3  .
1 x4 x24 
  
γ y4 
2
1 x5 x5 y5

Define xdat and ydat as before. Then compute

A = [ones(size(xdat)) xdat xdat.^2];

u = A \ ydat

u =
0.9983
-0.8451
0.1886

Thus the quadratic of best fit is y(x) = 0.9983 − 0.8451 x + 0.1886 x2 .

x = linspace(0,2,100);
y = u(1) + u(2)*x + u(3)*x.^2;
plot(x,y,’b-’,xdat,ydat,’m*’);

Quadratic of best bit

1

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0
0 0.5 1 1.5 2

4
Example. Modify the Matlab commands in the previous example to obtain a
polynomial interpolation to the data. What is the degree of the polynomial?
Answer. We need a polynomial of degree 4 to interpolate 5 points. So we have
m = n = 5, and the linear system Au = y, with
1 x1 x21 x31 x41
     
a0 y1
1 x2 x22 x32 x42  a1  y2 
 2 3 4
    
A= 1 x3 x3 x3 x3 , u = a2
 , y = y3  .
1 x4 x24 x34 x44 
   
a3  y4 
2 3 4
1 x5 x5 x5 x5 a4 y5

Define xdat and ydat as before. Then compute

A = [ones(size(xdat)) xdat xdat.^2 xdat.^3 xdat.^4];
u = A \ ydat

u =
1.0000
-1.5400
2.2467
-1.7600
0.4533
The interpolating polynomial is y(x) = 1 − 1.54 x + 2.2467 x2 − 1.76 x3 + 0.4533 x4 .
x = linspace(0,2,100);
y = u(1) + u(2)*x + u(3)*x.^2 + u(4)*x.^3 + u(5)*x.^4;
plot(x,y,’b-’,xdat,ydat,’m*’);

Polynomial interpolation
1

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0
0 0.5 1 1.5 2

5
Example. Approximate the data from the previous examples by an exponential
function y(x) = λ eµ x .
Answer. The data values yield a system of nonlinear equations in λ and µ:


 λ eµ x1 = y1
λ eµ x2 = y2


..


 .

 µ xm
λe = ym
Instead of solving this nonlinear least squares problem (which requires iterative
methods), we can convert this into a linear least squares problem by taking loga-
rithms:
y(x) = λ eµ x =⇒ ln(y(x)) = ln(λ eµ x ) = ln(λ) + µ x
We obtain the linear system


ln(λ) + µ x1 = ln(y1 )

ln(λ) + µ x2 = ln(y2 )

..


 .

ln(λ) + µ xm = ln(ym )


Thus    
1 x1 ln(y1 )
1 x2     ln(y2 ) 
ln(λ)
A = . ..  , u= , b =  . .
   
 .. .  µ  .. 
1 xm ln(ym )
Define xdat and ydat as before. Then compute
A = [ones(size(xdat)) xdat];
u = A \ log(ydat)
lambda = exp(u(1))
mu = u(2)

u =
0.1176
-1.3686

lambda =
1.1247

mu =
-1.3686

6
Thus λ = e0.1176 = 1.1247 and µ = −1.3686.
The exponential approximation is y(x) = 1.1247 e−1.3686 x .
x = linspace(0,2,100);
y = lambda * exp(mu*x);
plot(x,y,’b-’,xdat,ydat,’m*’);

Exponential approximation
1.4

1.2

0.8

0.6

0.4

0.2

0
0 0.5 1 1.5 2

Note that we obtained the least squares solution for the converted linear problem,
which is not the same as the least squares solution for the original nonlinear
problem.

Example. Matlab exlsq1.m covers the previous four examples.

Example. Matlab exlsq2.m produces the following plot.

Linear least squares
9

1
−2 0 2 4 6 8 10 12

7
 Two n × 1 vectors u and v are orthogonal ⇐⇒ u⊤ v = u · v = 0
An m × n matrix Q is orthogonal ⇐⇒ Q⊤ Q = In
Q = [q 1 q 2 · · · q n ] has orthonormal columns q i , i.e.,
(
q⊤
i q j = 0 for all i 6= j (the column vectors are orthogonal)
2
q⊤
i q i = kq i k2 = 1 for all i (the column vectors have unit length)

kQk2 = 1
kQAk2 = kAk2 for any A
If Q is not square, then QQ⊤ 6= Im in general.
If Q is square, then Q⊤ Q = In = QQ⊤ , and so Q−1 = Q⊤ .

QR factorization: write a real m × n matrix A, with m > n, in the form

   
R R
A=Q = [Y Z] =YR
0 0

R is an n × n upper triangular matrix.

Q is an m × m orthogonal matrix.
Q = [Y Z], where Y is an m × n matrix and Z is an m × (m − n) matrix.
 ⊤  ⊤ ⊤
  
Y Y Y Y Z I n 0
Q⊤ Q = Im =⇒ [Y Z] = =
Z⊤ Z ⊤Y Z ⊤Z 0 Im−n
Thus Y ⊤ Y = In , Z ⊤ Z = Im−n , and Y ⊤ Z = 0.
Hence both Y and Z are orthogonal matrices.
A⊤ A = (Y R)⊤ (Y R) = R⊤ Y ⊤ Y R = R⊤ R
Cost: O(m n2 ) flops
Matlab [Q, R] = qr(A)

Example. Matlab exqr.m

2 2 − 23 − 13 − 23 −3 −6 − 23 − 13
      
" #
2  −3 −6
 −2 −8  =  − 23 − 13   0 6 = 2
− 23
     
3 3 
0 6
1 −2 − 13 − 23 2
3 0 0 − 13 − 23 R
A Q Y

8
 The singular values σ1 ≥ σ2 ≥ · · · ≥ σn ≥ 0 of an m × n real matrix A are the
square-roots of the eigenvalues of A⊤ A .
A has full column rank ⇐⇒ columns of A are linearly independent ⇐⇒
σn > 0
r
Recall kAk2 := max |λi (A⊤ A)| = σ1
1≤i≤n

Define condition number for a rectangular matrix to be

s
σ1 max1≤i≤n |λi (A⊤ A)|
κ2 (A) := =
σn min1≤i≤n |λi (A⊤ A)|

max1≤i≤n |λi (A)|

If A is symmetric, then κ2 (A) = .
min1≤i≤n |λi (A)|

Exercise. Consider the QR factorization of an m × n real matrix A. Show that

(i) B = A⊤ A is symmetric
(ii) κ2 (B) = [κ2 (A)]2
(iii) κ2 (R) = κ2 (A)

9
Linear least squares
m data points and n unknowns, m > n (more equations than unknowns)
Au = y  

1 x1 x21 ··· xn−1

a0
 y 1
1
1 x2 x22 ··· xn−1  a1 
 y 2

2  
. .. .. ..  ..  = 
 .. 
 
..

 .. . . . .   .   . 
1 xm x2m ··· n−1
xm an−1 ym
A u y

A is an m × n matrix
A has full column rank
Residual r := Au − y
Pm
Find u to minimize krk22 = r ⊤ r = 2
j=1 rj

Solution method 1: use normal equations

Au = y =⇒ (A⊤ A) u = A⊤ y

Define n × n matrix B = A⊤ A
A full column rank =⇒ B symmetric and positive definite
Solve Bu = A⊤ y by Cholesky or LU factorization with forward & back
substitutions
Cost: O(m n2 ) flops (dominated by computing B)
Issue: κ2 (B) = κ2 (A⊤ A) = [κ2 (A)]2 , squaring condition number!
Solution method 2: use QR factorization
   
R R
A=Q = [Y Z] =YR
0 0

Au = y =⇒ (Y R) u = y =⇒ (Y ⊤ Y )Ru = Y ⊤ y =⇒ Ru = Y ⊤ y
A full column rank ⇐⇒ R non-singular
Solve Ru = Y ⊤ y using back substitution
Cost: O(m n2 ) flops (dominated by QR factorization)
Benefit: κ2 (R) = κ2 (A), condition number unchanged, so numerically
preferable
Concerns: QR factorization is more costly than Cholesky or LU factoriza-
tion; Q (and therefore Y ) is generally not sparse
Matlab u = A \ y

10
Exercise. Outline the steps in the two equivalent methods for finding the least
square solution to Au = y and count the flops in each step.

11
 Polynomial interpolation
Pn
Polynomial of degree n: p(x) = i=0 ai xi =⇒ n + 1 coefficient
Basis functions: xi for i = 0, . . . , n
n + 1 data points: (xj , yj ) for j = 0, . . . , n
Interpolation property : p(xj ) = yj for j = 0, . . . , n
Pn i
Linear system: i=0 ai xj = yj for j = 0, . . . , n, or in matrix form

x20 xn−1 xn0

    
1 x0 ··· 0 a0 y0
1 x1 x21 ··· xn−1
1
n  a 
x1  1   y1 

xn−1
 
1 x2 x22 ··· 2 xn2   a2   y2 
  =   =⇒ Au = y
..  ..   .. 

. .. .. .. ..
 .. . . . . .  .   . 
1 xn x2n ··· xn−1
n xnn an yn
A u y

(n + 1) × (n + 1) matrix A is called the Vandermonde matrix

Distinct xj =⇒ A nonsingular =⇒ unique solution u for any y
Weierstrass Approximation Theorem (1885): every continuous function
on a finite interval can be accurately approximated by a polynomial
Advantages: easy to evaluate (Horner’s method), differentiate and integrate

Example. Linear interpolation: find the polynomial (line) that interpolates the
data (x0 , y0 ) and (x1 , y1 ).
Answer. Let p(x) = a0 + a1 x. Then solve the simultaneous equations
( (
p(x0 ) = y0 a0 + a1 x0 = y0
=⇒
p(x1 ) = y1 a0 + a1 x1 = y1

to obtain a0 and a1 .
Alternatively, a quick informal approach is to equate the ratios
p(x) − y0 y1 − y0
= ,
x − x0 x1 − x0
and then rearrange to obtain
y1 − y0
p(x) = y0 + (x − x0 ).
x1 − x0

Example. See an interpolating polynomial of degree 4 on page 5.

12
 Polynomial interpolation in Lagrange form
n + 1 data points: (xj , yj ) for j = 0, . . . , n
Lagrange polynomials of degree n as basis functions:
n  
Y x − xj
ℓi (x) := for i = 0, . . . , n
xi − xj
j=0
j6=i

ℓi (xj ) = 1 for i = j, and ℓi (xj ) = 0 if i 6= j

n
X
Interpolating polynomial of degree n: p(x) = yi ℓi (x)
i=0
No linear system to solve, but ℓi (x) difficult to evaluate
Theorem. If f is n + 1 times continuously differentiable on [a, b], then the
error in approximating f (x) by p(x) is
n
f (n+1) (ξ) Y
f (x) − p(x) = (x − xj )
(n + 1)! j=0
for some unknown ξ ∈ [a, b] depending on x.
Matlab lagpol.m

Example. Find the interpolating polynomial through the points

(5, 1), (−7, −23), (−6, −54)
using standard basis functions and Lagrange polynomials.
Answer. 3 data points =⇒ polynomial of degree 2
Standard basis functions: Let p(x) = a0 + a1 x + a2 x2 . We solve
        
1 5 25 a0 1 a0 −114
1 −7 49 a1  = −23 =⇒ a1  =  8 
1 −6 36 a2 −54 a2 3
Thus p(x) = −114 + 8x + 3x2 .
Lagrange polynomials:
(x − x1 )(x − x2 ) (x − (−7))(x − (−6)) 1
ℓ0 (x) = = = (x + 7)(x + 6)
(x0 − x1 )(x0 − x2 ) (5 − (−7))(5 − (−6)) 132
(x − x0 )(x − x2 ) (x − 5)(x − (−6)) 1
ℓ1 (x) = = = (x − 5)(x + 6)
(x1 − x0 )(x1 − x2 ) (−7 − 5)(−7 − (−6)) 12
(x − x0 )(x − x1 ) (x − 5)(x − (−7)) 1
ℓ2 (x) = = = − (x − 5)(x + 7)
(x2 − x0 )(x2 − x1 ) (−6 − 5)(−6 − (−7)) 11
Thus p(x) = ℓ0 (x)−23 ℓ1 (x)−54 ℓ2 (x). (This equals p(x) = −114+8x+3x2 .)

13
Exercise. Matlab expolint.m (using horner.m) produces the following plot for
the interpolating polynomial through the 5 data points

(−2, 4), (−1, 2), (0, −3), (1, 5), (3, 4).

Polynomial interpolation
20

−20

−40

−60

−80

−100
−3 −2 −1 0 1 2 3 4

The interpolating polynomial of degree 4 is found to be

5 15 2
p(x) = −3 + x + x2 + x3 − x4 .
6 2 3
Express this interpolating polynomial in terms of Lagrange polynomials.

14
 Equally spaced points (or nodes)
b−a
On [a, b], xj = a + jh for j = 0, . . . , n, with h =
n
1
Carl Runge (1856 - 1927) example: f (x) = on [−1, 1]
1 + 25x2
Large error near end points
Error grows as degree increases!
Chebyshev points
n
Y
Choose xj to minimize max (x − xj )
x∈[−1,1]
j=0
 2n + 1 − 2j 
On [−1, 1], tj = cos π for j = 0, . . . , n
2n + 2
a+b b−a
On [a, b], xj = + tj for j = 0, . . . , n
2 2
Chebyshev nodes are the zeros of the Chebyshev polynomial Tn+1 (x) of the
first kind of degree n + 1 (details omitted)
Interpolation error is minimized by choosing Chebyshev nodes
We can also represent the interpolating polynomial using Chebyshev poly-
nomials Ti (x) for i = 0, . . . , n as basis functions (details omitted)

Example. Matlab excheb.m (using horner.m)

2
Interpolating 1 / (1 + 25 x ) by polynomial of degree 8 at equally spaced points

0.5

−0.5 Interpolation points

Interpolating polynomial
−1 Function

−1 −0.75 −0.5 −0.25 0 0.25 0.5 0.75 1

x
2
Interpolating 1 / (1 + 25 x ) by polynomial of degree 8 at Chebyshev points

0.5

−0.5 Interpolation points

Interpolating polynomial
−1 Function

−1 −0.75 −0.5 −0.25 0 0.25 0.5 0.75 1

x

15
Polynomial of degree n
n
X
p(x) = ak xk = an xn + an−1 xn−1 + · · · + a2 x2 + a1 x + a0 , an 6= 0
k=0

Example: 5x2 + 3x − 7 is a polynomial of degree 2 (quadratic)

Degree n ⇐⇒ n + 1 coefficients
Only requires addition, multiplication, integer powers
Naive implementation requires the evaluation of xk (k − 1 multiplications)
for each k, plus n multiplications and n additions:
n
X n(n − 1) n2 + 3n
(k − 1) + n + n = + 2n = = O(n2 ) flops.
2 2
k=1

Polynomial evaluation by Horner’s method (Matlab horner.m)

Use the equivalent expression

p(x) = an xn + an−1 xn−1 + · · · + a2 x2 + a1 x + a0

= ((· · · (an x + an−1 )x + · · · + a2 )x + a1 )x + a0 ,

which requires only n multiplications and n additions:

2n = O(n) flops.

Matlab p = horner(a,x)
Allows a vector x of inputs
Computes p(x) = a1 + a2 x + · · · + an xn−1 + an+1 xn for each input
Returns a vector p of outputs of the same length as x
Elementwise multiplication .*
Matlab vector/array indices always start with 1
Matlab p = polyval(b,x)
Coefficients in reverse order p(x) = b1 xn + b2 xn−1 + · · · + bn x + bn+1
Degree n implied by length n + 1 of input vector b

16