MAT216: Linear Algebra & Fourier Analysis

Lecture Note: Week 8_Lecture 15_Part 1 and Part 2

Md. Saddam Hossain, Summer 2020
Brac University
Contents:
 Part 1: Orthogonal and Orthonormal Basis

 Part 2: The Gram Schmidt Process

References:
 Introduction to Linear Algebra, 5th Ed. Gilbert Strang

 https://www.youtube.com/watch?v=0MtwqhIwdrI&t=2411s

 https://bux.bracu.ac.bd/courses/course-v1:buX+2020_SummerMAT216+/course/

Part 1: Orthogonal and Orthonormal Basis

Let’s start with the definition of the basis of a vector space one more time: A basis for a vector
space is a sequence of vectors with two properties:

i. The basis vectors are linearly independent and

ii. They span the space.

Usually, we have to check both properties. When the count is right, one property implies the
other.
i. Any n independent vectors in R n must span R n . So, they are a basis.
ii. Any n vectors that span R n must be independent. So, they are a basis.
Therefore, a set of linearly independent vectors that can span the entire space is a basis of a
space.

Definition: A set of linearly independent vectors that can span the entire space, and each of the
vector is perpendicular to all other vectors is an orthogonal basis of the space.

Definition: An orthogonal basis for a subspace 𝑉 of 𝑅 𝑛 is a basis for 𝑉 that is also an orthogonal
set.

i.e. 𝑉 = {𝑣1 , 𝑣2 } is an orthogonal basis if the vectors that form it are perpendicular. In other
words, 𝑣1 and 𝑣2 form an angle of 90° [or, 〈𝑣1 , 𝑣2 〉 = 0].

Definition: A set of vectors 𝑉 = {𝑣1 , 𝑣2 , . . . . . . . , 𝑣𝑛 } are mutually orthogonal if every pair of

vectors is orthogonal. i.e. ⃗⃗𝑣⃗𝑖 . ⃗⃗𝑣⃗𝑗 = 0, for all 𝑖 ≠ 𝑗.

Example 1: The standard basis vectors are orthogonal.

𝑒𝑖 . 𝑒𝑗 = 𝑒𝑖𝑇 𝑒𝑗 = 0 when 𝑖 ≠ 𝑗.

Example 2: Show that {𝑣1 , 𝑣2 , 𝑣3 } is an orthogonal set, where

−1
3 −1 2
𝑣1 = [1] , 𝑣2 = [ 2 ] , 𝑣3 = [−2]
7
1 1
2

Solution:
Here, we have 𝑣1 . 𝑣2 = 𝑣2 . 𝑣3 = 𝑣3 . 𝑣1 = 0.
Thus, each pair of distinct vectors is orthogonal, and so {𝑣1 , 𝑣2 , 𝑣3 } is an
orthogonal set.
  Two subspaces V and W of a vector space are orthogonal if every vector v in V is
perpendicular to every vector w in W. i.e. v T w = 0 for all v in V and all w in 𝑊. [For
example, please find Introduction to Linear Algebra, 5th Edition by Gilbert Strang,
page 195]

 Every vector x in the null space is perpendicular to every row of 𝐴, because 𝐴𝑥 = 0.

The null space 𝑁(𝐴) and the row space 𝐶(𝐴𝑇 ) are orthogonal subspaces of 𝑅 𝑛 .

Definition: When the basis is orthogonal and also the length of the vectors of the basis is 1, then
the basis is called orthonormal.

 A set of vectors 𝑉 = {𝑣1 , 𝑣2 , . . . . . . . 𝑣𝑛 } are orthonormal if:

0 𝑖𝑓 𝑖 ≠ 𝑗
𝑣𝑖𝑇 𝑣𝑗 = { ;
1 𝑖𝑓 𝑖 = 𝑗

In other words, they all have length 1 (𝑖. 𝑒. ‖𝑣

⃗⃗⃗𝑖 ‖ = 1) and are perpendicular to each other.
Orthonormal vectors are always independent.

 A square orthonormal matrix 𝐴 is called an orthogonal matrix. If 𝐴 is square then

𝐴𝑇 𝐴 = 𝐼 tells us that 𝐴𝑇 = 𝐴−1

Example: The standard basis vectors are orthogonal and each of them has unit length, therefore
they form an orthonormal basis.

Example: Determine which of the following sets of vector form orthonormal basis:
3 0
a. 𝑢 = [ ] , 𝑣 = [ ] ;
0 −2
 1 1 1
b. 𝑢 = [ 0 ] , 𝑣 = [√2] , 𝑤 = [−√2];
−1 1 1
1 0
c. 𝑢 = [ ] , 𝑣 = [ ] ;
0 −1
1 0
d. 𝑢 = [2] , 𝑣 = [0];
0 3

When we need a basis to do calculation, it is convenient to use and orthonormal basis. For
instance, the formula for a vector space projection is much simpler with an orthonormal
basis.
They use for making good coordinate system or good coordinate bases.

Part 2: The Gram Schmidt Process

Let’s consider a set of vectors 𝑉 = {𝑣

⃗⃗⃗⃗1 , ⃗⃗⃗⃗⃗
𝑣2 , . . . . . . . ⃗⃗⃗⃗⃗⃗
, 𝑣𝑛 }which may not be orthonormal. Our goal in
this particular section is to produce a set of orthonormal vectors 𝑈 = {𝑢
⃗⃗⃗⃗1 , ⃗⃗⃗⃗⃗
𝑢2 , . . . . . . . ⃗⃗⃗⃗⃗⃗
, 𝑢𝑛 } by Gram
Schmidt process with the same span as the original set of vectors 𝑉.
 Gram Schmidt process always start with:

⃗⃗⃗⃗
𝑣
Step 1: 𝑢1 = ‖𝑣⃗⃗⃗1‖
⃗⃗⃗⃗
𝑖

Remember that dividing a vector by its length always produced a unit vector, so ⃗⃗⃗⃗
𝑢1 has length 1
and points in the same direction as ⃗⃗⃗⃗
𝑣1 [Find video lecture 15, part 2].

2 ⃗⃗⃗⃗⃗
𝑤
Step 2: Compute ⃗⃗⃗⃗⃗
𝑤2 = ⃗⃗⃗⃗
𝑣2 − (𝑣
⃗⃗⃗⃗2 . ⃗⃗⃗⃗
𝑢1 )𝑢⃗⃗⃗⃗1 and find ⃗⃗⃗⃗
𝑢2 = ‖𝑤
⃗⃗⃗⃗⃗ ‖ 2

After doing the first step we could produce a vector ⃗⃗⃗⃗⃗

𝑤2 which is orthogonal to ⃗⃗⃗⃗
𝑢1 and by second
step we get the length 1.

3 ⃗⃗⃗⃗⃗
𝑤
Step 3: Let us first find 𝑤3 = ⃗⃗⃗⃗
⃗⃗⃗⃗⃗ 𝑣3 − (𝑣
⃗⃗⃗⃗3 . ⃗⃗⃗⃗
𝑢2 )𝑢⃗⃗⃗⃗2 − (𝑣
⃗⃗⃗⃗3 . ⃗⃗⃗⃗
𝑢1 )𝑢⃗⃗⃗⃗1 and then set ⃗⃗⃗⃗
𝑢3 = ‖𝑤
⃗⃗⃗⃗⃗ ‖ 3

Thus ⃗⃗⃗⃗⃗
𝑤3 is orthogonal to ⃗⃗⃗⃗
𝑢1 and ⃗⃗⃗⃗
𝑢2 and by dividing by its length, we get something of length one.

. . . . . . . . . . . . . . . . . . . .. . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . .

. . . . . . .. . .. . . . .. . . . . . .. . . . . . . . . . . . . . . . . . . . .. . . . . .. . . . .. .. . .. . .

𝑛 ⃗⃗⃗⃗⃗⃗
𝑤
𝑤
⃗⃗ 𝑛 = 𝑣𝑛 − (𝑣𝑛 . 𝑢 ⃗ 𝑛−1 - …………… - (𝑣𝑛 . 𝑢
⃗ 𝑛−1 )𝑢 ⃗ 1 )𝑢
⃗ 1 and set ⃗⃗⃗⃗
𝑢𝑛 = ‖𝑤
⃗⃗ ‖ 𝑛

𝑢𝑗 ’s constructed
At the final step we take ⃗⃗⃗𝑣𝑖 and subtract off its projections onto all the previous ⃗⃗⃗
thus far, and divide the result by its length.

 Span {𝑢
⃗⃗⃗⃗1 , ⃗⃗⃗⃗⃗
𝑢2 , . . . . . . . ⃗⃗⃗⃗⃗⃗
, 𝑢𝑛 } = span {𝑣
⃗⃗⃗⃗1 , ⃗⃗⃗⃗⃗
𝑣2 , . . . . . . . ⃗⃗⃗⃗⃗⃗
, 𝑣𝑛 } and {𝑢
⃗⃗⃗⃗1 , ⃗⃗⃗⃗⃗
𝑢2 , . . . . . . . ⃗⃗⃗⃗⃗⃗
, 𝑢𝑛 } is an
orthonormal set, thus {𝑢
⃗⃗⃗⃗1 , ⃗⃗⃗⃗⃗
𝑢2 , . . . . . . . ⃗⃗⃗⃗⃗⃗
, 𝑢𝑛 } is an orthonormal basis V.

Example: Find an orthonormal basis spanned by a set of vectors:

2 4
𝑣1 = [−5] and 𝑣2 = [−1].
1 2
Solution:

2
√30
⃗⃗⃗⃗1
𝑣 −5
Step 1: ⃗⃗⃗⃗
𝑢1 = ‖𝑣⃗⃗⃗𝑖 ‖
= √30
1
[√30]

⃗⃗⃗⃗⃗
𝑤2
Step 2: Compute ⃗⃗⃗⃗⃗
𝑤2 = ⃗⃗⃗⃗
𝑣2 − (𝑣
⃗⃗⃗⃗2 . ⃗⃗⃗⃗
𝑢1 )𝑢⃗⃗⃗⃗1 and find ⃗⃗⃗⃗
𝑢2 = ‖𝑤
⃗⃗⃗⃗⃗ ‖
2

1 3
4 4 −5 3
15
⃗⃗⃗⃗⃗2 = [−1] −
𝑤 𝑢1 = [−1] − [ 2 ] = [ 2 ]
⃗⃗⃗⃗
√30 1 3
2 2
2 2

√2
√3
1
𝑢2 =
⃗⃗⃗⃗
√6
1
[√6]

Therefore, according to the definition 𝑈 = {𝑢

⃗⃗⃗⃗1 , ⃗⃗⃗⃗⃗
𝑢2 } form an orthonormal basis

Exercise:

1. Find an orthonormal basis of ℋ: 𝑥 − 2𝑦 − 3𝑧 = 0.

1 1 5
2. Let 𝑣1 = [ 2 ], 𝑣2 = [ 0 ], and 𝑣3 = [2]. Find an orthonormal basis for span (𝑣1 , 𝑣2 , 𝑣3 ).
−2 −4 0
3. Construct an orthonormal basis of ℝ3 by applying Gram-Schmidt orthogonalization to
1 1 1
[1] , [0 ] , [1].
1 1 0