Chapter 5: Inner Product Spaces

Courtesy:- Material for this lecture is selected from Kolman’s book, and
handouts of Virtual University, Lahore, Virtual COMSATS and Imperial
College London.
Objective of this Lecture:-
(A) The prime objective of this chapter is to convert a set of linearly independent vectors
(basis) into a set of orthonormal vectors (basis).
(B) To achieve this goal, first we need to define several bits and pieces like “dot product
(inner product), length (norm or magnitude) of vectors, unit vector, distance between
vectors, projection of vectors, orthogonal vectors and orthonormal vectors”.

After studying this lecture, you are desired to do

Home Work: Do Questions 6, 8, 18 and 24 of Exercise 5.1, Do Questions 1-8 of
Exercise 5.2; Questions 8,9,12,13,29,30 of Exercise 5.3, and Questions 1-4, 9-20
of Exercise 5.4.

Section 5.1:
Definition 1: (Dot or inner or scalar or projection product of vectors in terms of
vector components)

𝒖𝟏 𝒗𝟏
𝒖𝟐 𝒗𝟐
. .
Let 𝒖, 𝒗 ∈ 𝑅𝑛 then 𝒖 = . and 𝒗 = . , consider these vectors as 𝑛 × 1
. .
[𝒖 𝒏 ] [𝒗𝒏 ]
𝑇
matrices. Clearly 𝒖 = [𝒖𝟏 𝒖𝟐 . . . 𝒖𝒏 ].
Now the dot or inner product is a scalar quantity dented by 𝒖• 𝒗 or (𝒖, 𝒗) and is
defined as matrix product
 𝑣1
𝑣2
.
𝒖• 𝒗 = (𝒖, 𝒗) = 𝒖𝑇 𝒗 = [𝑢1 𝑢2 . . . 𝑢𝑛 ] . = 𝑢1 𝑣1 + 𝑢2 𝑣2 + ⋯ + 𝑢𝑛 𝑣𝑛
.
[𝑣𝑛 ]
Note: Although notation 𝒖• 𝒗 is more familiar but we will use this natation
(𝒖, 𝒗) for dot/inner product to maintain the consistency with Kolman’s book.
𝟔 𝟒
Example 1: Let 𝒖 = [𝟒] and 𝒗 = [−𝟏] then
𝟐 𝟖
(𝒖, 𝒗) = 𝟔(𝟒) + 𝟒(−𝟏) + 𝟐(𝟖) = 𝟑𝟔 = (𝒗, 𝒖) (Inner product is commutative)
Definition 2: (Length or norm or magnitude of vector)
𝑣1
𝑣2
.
Let 𝒗 ∈ 𝑅𝑛 then 𝒗 = . , then length of this vector is denoted by ‖𝒗‖ (scalar
.
[𝑣𝑛 ]
quantity) and defined as ‖𝒗‖ = √𝒗𝟏 2 + 𝒗𝟐 2 + . . . +𝒗𝒏 2 . When 𝑛 = 2 𝑜𝑟 3 i.e.,
our vector is from 𝑅2 (2D) or 𝑅3 (3D) then

‖𝒗‖ = √𝒗𝟏 2 + 𝒗𝟐 2 And ‖𝒗‖ = √𝒗𝟏 2 + 𝒗𝟐 2 + 𝒗𝟑 2 , respectively.

Observation: A nice relation between dot product and norm of a vector.
(𝒗, 𝒗) = 𝒗𝟏 𝒗𝟏 + 𝒗𝟐 𝒗𝟐 + ⋯ + 𝒗𝒏 𝒗𝒏 = 𝒗𝟏 𝟐 + 𝒗𝟐 𝟐 +. . . + 𝒗𝒏 𝟐 = ‖𝒗‖𝟐
𝟏
𝟑
Example 2: Let 𝒗 be vector in 𝑹 , such that 𝒗 = [ 𝟐 ] then ‖𝒗‖ = √1 + 4 + 4=3
−𝟐
Definition 3: (Unit vector and normalization)
A vector having length 1 is called a unit vector. We can convert any arbitrary
vector 𝒗 in to a unit vector 𝒖 using formula
𝒗
𝒖=
‖𝒗‖
This way of getting 𝒖 from 𝒗 is called normalization of 𝒗 and 𝒖 is called
normalized vector. Both 𝒖 and 𝒗 are in same direction.
𝟏
𝟑
Example 3: Let 𝒗 be vector in 𝑹 , such that 𝒗 = [ 𝟐 ] then ‖𝒗‖ = √1 + 4 + 4=3
−𝟐
𝟏
𝟏 3
𝑣 𝟏 𝟐
Then 𝒖 = ‖𝑣‖ = [ 𝟐 ] = 3
. Now verify
3
−𝟐 −𝟐
[3]

1 2 2 2 −2 2 1 4 4
√
‖𝒖‖ = ( ) + ( ) + ( ) = √ + + = 1
3 3 3 9 9 9

Definition 1a: (Dot or inner product of vectors in terms of projection)

Here we present an alternative formula for dot/inner product in terms of projection
of vectors.
Let 𝒖, 𝒗 ∈ 𝑅𝑛 then (𝒖, 𝒗) = ‖𝒖‖‖𝒗‖ cos(𝜃), Proof of this formula is simple using
Law of cosines of triangle.
(𝒖, 𝒗) = ‖𝒖‖‖𝒗‖ cos(𝜃) , = ‖𝒗‖ (scalar projection of 𝒖 on 𝒗) , Where 𝜃 is angle
between 𝒖 and 𝒗. (See Figure 1)
 Figure 1
Observation: The formula in Definition 1a is excellent in establishing various
facts about dot/inner product whereas formula in Definition 1 is handy for
calculations.
NOTE: Watch the video given in link below, only for students interested in
proof, for 𝒖, 𝒗 ∈ 𝑹𝟐 ,
(𝒖, 𝒗) = 𝒖𝟏 𝒗𝟏 + 𝒖𝟐 𝒗𝟐 = ‖𝒖‖‖𝒗‖ 𝒄𝒐𝒔(𝜽)
https://www.youtube.com/watch?v=LzP_9jJpR-g
Some facts about dot/inner product:
𝝅
Fact 1: (𝒖, 𝒗) = 0 if 𝜽 = i.e., vectors are orthogonal to each other when their dot
𝟐
product is zero.
Fact 2: (𝟎, 𝒗) = 0 if i.e., Zero vector is orthogonal to every vector.
Fact 3: (𝒖, 𝒗) = ‖𝒖‖‖𝒗‖ if 𝜽 = 0 i.e., vectors are pointing in same direction.
Fact 4: (𝒖, 𝒗) = −‖𝒖‖‖𝒗‖ if 𝜽 = 𝝅 i.e., vectors are in opposite direction.

Definition 4: (Distance between two vectors)

Let 𝒖, 𝒗 ∈ 𝑅𝑛 then distance between 𝒖 and 𝒗 is denoted by 𝒅(𝒖, 𝒗) and is defined
as

𝒅(𝒖, 𝒗) = ‖𝒖 − 𝒗‖ = √(𝒖𝟏 − 𝒗𝟏 )2 + (𝒖𝟐 − 𝒗𝟐 )2 +. . . +(𝒖𝒏 − 𝒗𝒏 )2 .

Note: Next figure is only for interested readers, proof can directly be seen
geometrically from law of parallelogram of vectors.
Now we present some theorems having many applications in Mathematics and
proofs are beyond the scope of this course.
Do questions 6, 8, 18 and 24 of Exercise 5.1
Theorem 1: Cauchy-Schwartz Inequality:-
If 𝒖 and 𝒗 are any two vectors in 𝑹𝒏 , then |(𝒖, 𝒗)| ≤ ‖𝒖‖‖𝒗‖
i.e., Absolute value of dot product is less or equal to product of magnitudes of
vectors.
Example 4:

i.e., |−5| ≤ √14 . √17 ⇒ 5 ≤ 15.43

Theorem 2: Triangle Inequality:-
If 𝒖 and 𝒗 are any two vectors in 𝑹𝒏 , then ‖𝒖 + 𝒗‖ ≤ ‖𝒖‖ + ‖𝒗‖
−2
Example 5: Consider again 𝒖 and 𝒗 given in example 4, then 𝒖 + 𝒗 = [ 4 ] and
−1
‖𝒖 + 𝒗‖ = √21 , Now √21 ≤ √14 + √17 ⇒ 𝟒. 𝟔 ≤ 𝟕. 𝟖𝟔.
Theorem 3: The Pythagorean Theorem:-
Two vectors 𝒖 and 𝒗 in 𝑹𝒏 are orthogonal if and only if
(‖𝒖 + 𝒗‖)𝟐 = ‖𝒖‖𝟐 + ‖𝒗‖𝟐 .
Definition 5: (Orthogonal Set)
Let 𝑆 = {𝑉1 , 𝑉2 , . . . , 𝑉𝑝 } be the set of non-zero vectors in 𝑹𝒏 , then 𝑆 is said to be
orthogonal set if all vectors in 𝑆 are mutually orthogonal. Mathematically
 (𝑉𝑖 , 𝑉𝑗 ) = 0 𝑓𝑜𝑟 𝑎𝑙𝑙 𝑖 ≠ 𝑗

Theorem 4:- Let 𝑆 = {𝑉1 , 𝑉2 , . . . , 𝑉𝑝 } be a finite orthogonal set of vectors of a

Euclidean space 𝑹𝒏 . Then 𝑆 is linearly independent.
Definition 6: (Orthogonal Basis)
Let 𝑆 = {𝑉1 , 𝑉2 , . . . , 𝑉𝑝 } be a basis for subspace 𝑾 in 𝑹𝒏 , (𝒅𝒊𝒎 𝑾 = 𝒑 < 𝒏), then
𝑆 is said to be orthogonal basis if set 𝑆 is an orthogonal set.
Definition 7: (Orthonormal Set)
Let 𝑆 = {𝑈1 , 𝑈2 , . . . , 𝑈𝑝 } be the set of non-zero vectors in 𝑹𝒏 , then 𝑆 is said to be
orthonormal set if 𝑆 is orthogonal set of unit vectors. Mathematically
(𝑈𝑖 , 𝑈𝑗 ) = 0 𝑓𝑜𝑟 𝑎𝑙𝑙 𝑖 ≠ 𝑗 𝑎𝑛𝑑 ‖𝑈𝑖 ‖ = 1 𝑓𝑜𝑟 𝑎𝑙𝑙 𝑈𝑖 ∈ 𝑆.
Definition 8: (Orthonormal Basis)
Let 𝑆 = {𝑈1 , 𝑈2 , . . . , 𝑈𝑝 } be a basis for subspace 𝑾 in 𝑹𝒏 , then 𝑆 is said to be
orthonormal basis if set 𝑆 is an orthonormal set.
Example 6:

Now we have enough tools and finally arrived at the beginning of our core
problem.
 Gram-Schmidt Process to obtain orthonormal basis:-
Our life is very easy while working with orthonormal bases. We observed that
to work with the natural/standard basis for 𝑹𝟐 𝐚𝐧𝐝 𝑹𝟑 the computations are
kept to a minimum. (See my previous recording)
𝟏
𝟑
Recall (1) if 𝑆 = {𝑒1 , 𝑒2 , 𝑒3 } are standard basis of 𝑹 , where 𝑒1 = [𝟎] , 𝑒2 =
𝟎
𝟎 𝟎 𝒂
[𝟏] , 𝑒3 = [𝟎] then any vector 𝒗 = [𝒃] ∈ 𝑹𝟑 can directly be written as linear
𝟎 𝟏 𝒄
combination of basis i.e., 𝒗 = 𝒂𝑒1 + 𝒃𝑒2 + 𝒄𝑒3 . (Coefficients of bases vectors are
coordinates of the vector 𝒗)
Recall (2) if set 𝑆 = {𝑉1 , 𝑉2 , 𝑉3 } is any basis of 𝑹𝟑 , and vector 𝒗 ∈ 𝑹𝟑 then 𝒗 can
be written as 𝒗 = 𝒂𝑉1 + 𝒃𝑉2 + 𝒄𝑉3 . (See Examle 1(LC)), We have to solve a non-
homogeneous system to obtain the coordinates of 𝒗.
Following theorem shows the advantage of orthonormal basis over any basis of
vector space.
Theorem 5:- (In book Theorem 5.5)
Let 𝑆 = {𝑉1 , 𝑉2 , . . . , 𝑉𝑛 } be a finite orthonormal basis of finite vectors of a
Euclidean space 𝑽 and let 𝒗 be any vector in 𝑽. Then
𝒗 = 𝑐1 𝑉1 + 𝑐2 𝑉2 + 𝑐3 𝑉3 + ⋯ + 𝑐𝑛 𝑉𝑛
(𝒗, 𝑉1 ) = 𝑐1 (𝑉1 , 𝑉1 ) = 𝑐1 ||𝑉1 ||2 = 𝑐1
Where 𝑐𝑖 = (𝒗, 𝑉𝑖 ), 𝑖 = 1,2,3, … , 𝑛
Note:- We only need to compute few dot product in presence of orthonormal
basis.
Example 7:-
Solution:-

Theorem 6:- (In book Theorem 5.6)

Let 𝑉 = 𝑅n be a Euclidean space and 𝑊 an m-dimensional subspace of 𝑉, then
there exists an orthonormal basis 𝑇 = {𝒘𝟏 , 𝒘𝟐 , . . . , 𝒘𝒎 }.
OR equivalently, for subspace 𝑊 given any basis set say 𝑆 = {𝒖𝟏 , 𝒖𝟐 , . . . , 𝒖𝒎 }
Obtain/convert (1) set 𝑆 to an orthogonal basis set say 𝑇 ∗ = {𝒗𝟏 , 𝒗𝟐 , . . . , 𝒗𝒎 }
(using Gram-Schmidt process)
(2) Finally normalize this set to get orthonormal basis
𝒗𝟏
𝑇 = {𝒘𝟏 = , 𝒘 , . . . , 𝒘𝒎 }.
||𝒗𝟏 || 𝟐
NOTE:-PROOF OF THIS THEOREM IS BEYOND THE SCOPE OF OUR
COURSE, HOWEVER I WILL GIVE GENERAL OUTLINE OF PROOF.
 𝒗𝟏 𝒗𝟐 𝒗𝟑 𝒗𝟒
𝒘𝟏 = ; 𝒘𝟐 = ; 𝒘𝟑 = ; 𝒘𝟒 =
||𝒗𝟏 || ||𝒗𝟐 || ||𝒗𝟑 || ||𝒗𝟒 ||

Do Example 2 on page (323) of Kolman’s book. For video support of

this please visit following links.
https://www.youtube.com/watch?v=KKffS_U6_34&t=104s
https://www.youtube.com/watch?v=swXcm_vTjWU
Question 9: Another solution:
Hint: Containing (Leading 1)
Merge given vectors with standard basis; {u1, u2, e1,e2,e3}
Do yourself (I am going to use linear Algebra toolkit)
𝑺 = {𝑢1 , 𝑢2 , 𝑢3 }
2/3 2/3 1
Where 𝑢1 = [−2/3]; 𝑢2 = [ 1/3 ] ; 𝑢3 = [0]
1/3 −2/3 0
(1) Use Gram-Schmidt Process 𝑣1 , 𝑣2 , 𝑣3 =?
2/3
𝑣1 = 𝑢1 = [−2/3]
1/3
(𝑢2 , 𝑣1 ) 2/3 2/3
0
𝑣2 = 𝑢2 − 𝑣 = [ 1/3 ] − [−2/3]
(𝑣1 , 𝑣1 ) 1 (𝑣1 , 𝑣1 )
−2/3 1/3
2/3
𝑣2 = [ 1/3 ]
−2/3
2 2
1 2 3 2 3
(𝑢3 , 𝑣1 ) (𝑢3 , 𝑣2 ) 2 1
𝑣3 = 𝑢3 − 𝑣1 − 𝑣2 = [0] − 3 − − 3
(𝑣1 , 𝑣1 ) (𝑣2 , 𝑣2 ) 1 3 1 3
0
1 2
[ 3 ] −
[ 3]
4 4
1− −
9 9 1/9
4 2
= 0 + − = [2/9]
9 9 2/9
2 4
[0 − 9 + 9 ]
1/9
𝑣3 = [2/9]
2/9
 2/3 2/3
𝑣1 𝑣2
𝑤1 = = [−2/3] ; 𝑤2 = = [ 1/3 ] ;
||𝑣1 || ||𝑣2 ||
1/3 −2/3
1/9 1/9 1/3
𝑣3 1
𝑤3 =
||𝑣3 ||
=
1/3
[2/9] = 3 [2/9] = [2/3]
2/9 2/9 2/3

𝑊 = {[𝑏 + 2𝑐 − 𝑑 𝑏 𝑐 𝑑]: 𝑎 = 𝑏 + 2𝑐 − 𝑑}
[𝑏 + 2𝑐 − 𝑑 𝑏 𝑐 𝑑]
= 𝑏[1 1 0 0] + 𝑐[2 0 1 0] + 𝑑[−1 0 0 1]
Solution: Solve using augmented matrix: [A|O] (Do yourself)
𝑥1 −4𝑟 −4
𝑋 = [𝑥2 ] = [ 5𝑟 ] = 𝑟 [ 5 ]
𝑥3 𝑟 1
−4
Basis of solution (Null) space 𝑆 = {[ 5 ]}
1
−4
𝑢1 = [ 5 ]
1
−4
𝑣1 = 𝑢1 = [ 5 ]
1
𝑣1 1 −4
𝑤1 = = [5]
||𝑣1 || √42
1