ENGG 5501: Foundations of Optimization 2019–20 First Term

Handout C: Real Analysis Cheat Sheet

Instructor: Anthony Man–Cho So Updated: August 26, 2019

The purpose of this handout is to give a brief review of some of the basic concepts and results
in analysis. If you are not familiar with the material and/or would like to do some further reading,
you may consult, e.g., the books [1, 4, 3, 2].

1 Basic Topology
In this course, we shall frequently operate on various metric spaces. Recall that a metric space is
a pair (X, d), where X is a set and d : X × X → R is a function satisfying the following properties:

(a) (Non–Negativity) for any x, y ∈ X, d(x, y) > 0 if x 6= y and d(x, x) = 0;

(b) (Symmetry) for any x, y ∈ X, d(x, y) = d(y, x);

(c) (Triangle Inequality) for any x, y, z ∈ X, d(x, z) ≤ d(x, y) + d(y, z).

One of the most important examples of a metric space is (Rn , d), where n ≥ 1 is an integer and
d : Rn × Rn → R is given by
n
!1/2
X
2
d(x, y) = |xi − yi | .
i=1

To facilitate our subsequent discussion, let us begin with some definitions. Let (X, d) be a metric
space. The open ball with center at x̄ ∈ X and radius r > 0 is defined as the set

B ◦ (x̄, r) = {x ∈ X : d(x, x̄) < r} .

In a similar fashion, the closed ball with center at x̄ ∈ X and radius r > 0 is defined as the set

B(x̄, r) = {x ∈ X : d(x, x̄) ≤ r} .

Now, for any point x ∈ X and subset S ⊆ X, we say that

• x is a limit point of S if every open ball centered at x contains a point y 6= x such that
y ∈ S;

• S is closed if every limit point of S belongs to S;

• x is an interior point of S if there is an open ball B centered at x such that B ⊆ S;

• S is open if every point of S is an interior point of S;

• the complement of S (denoted by S c ) is the set of all points y ∈ X such that y 6∈ S;

• S is bounded if there exists an M < ∞ and a point ȳ ∈ X such that d(y, ȳ) ≤ M for all
y ∈ S; i.e., S ⊆ B(ȳ, M );

1
 • S is connected if it cannot be written as the disjoint union of two non–empty open sets.

Given a set S ⊆ X, let S 0 be the set of limit points of S in X. Then, the set S ∪ S 0 is called the
closure of S and is denoted by cl(S).
To illustrate the above concepts, consider the case where S = [0, 1) ⊆ R. The point x = 1 is a
limit point of S, since for any r > 0, the open ball B ◦ (1, r) = {x ∈ R : |x − 1| < r} contains the
point y = 1 − r/2 6= x, which belongs to S. However, the limit point x = 1 does not belong to
S and so S is not closed. Now, any point x̄ ∈ (0, 1) is an interior point of S, since the open ball
B ◦ (x̄, r(x̄)), where r(x̄) = min{x̄, 1 − x̄}, is completely contained in S. On the other hand, x = 0 is
not an interior point of S and hence S is not open. This example also demonstrates an interesting
point: a set can be neither open nor closed! It is clear that S is bounded, as S ⊆ [−1, 1] = B(0, 1).
Finally, we have cl(S) = [0, 1].
Based on the above definitions, we have the following results:

• A set S is open iff its complement is closed.

• A set S is closed iff its complement is open.

• For any collection {Sα }α of open sets, the set ∪α Sα is open.

• For any collection {Sα }α of closed sets, the set ∩α Sα is closed.

• For any finite collection S1 , . . . , Sn of open sets, the set ∩ni=1 Si is open.

• For any finite collection S1 , . . . , Sn of closed sets, the set ∪ni=1 Si is closed.

Note that the finiteness assumption is crucial for the last two statements to hold. Indeed, consider
the collection of sets Si = (−1/i, 1/i) ⊆ R, where i = 1, 2, . . .. It is clear that each Si is an open
set. However, we have ∩i≥1 Si = {0}, which is closed.
Now, let us turn to the concept of compactness. We say that a subset S ⊆ Rn is compact if it
is closed and bounded. One interesting feature of compact sets is the following:

• Every infinite subset S 0 of a compact set S has a limit point in S.

Moreover,

• Every bounded infinite subset of Rn has a limit point in Rn .

2 Sequences
Before we discuss sequences in detail, let us introduce various notions of a bound on a set in R. We
say that a set S ⊆ R is bounded above if there exists a constant β ∈ R such that x ≤ β for all
x ∈ S. The number β is called an upper bound of S. The lower bound of a set in R is defined
analogously.
Now, suppose that S ⊆ R is bounded above and there exists an α ∈ R satisfying (i) α is an
upper bound of S and (ii) if γ < α, then γ is not an upper bound of S. Then, α is called the least
upper bound or supremum of S, which we denote by α = sup S.
In a similar fashion, suppose that S ⊆ R is bounded below and there exists an α ∈ R satisfying
(i) α is a lower bound of S and (ii) if γ > α, then γ is not a lower bound of S. Then, α is called

2
the greatest lower bound or infimum of S, which we denote by α = inf S. Note that both the
supremum and infimum of a set, if exist, are unique.
To illustrate the above concepts, consider the set S = (0, 1) ⊆ R. Clearly, S is bounded above
by 1 and bounded below by 0. Now, we claim that 1 = sup S and 0 = inf S. Indeed, if γ < 1, then
γ cannot be an upper bound of S, for γ < γ +  ∈ S for sufficiently small  > 0. A similar argument
shows that 0 = inf S, as desired. This example also shows that the supremum and infimum of a set
S need not belong to S.
We say that a sequence {an }n≥1 in a metric space (X, d) converges to a ∈ X if for every
 > 0, there exists an N such that d(an , a) <  whenever n ≥ N . In this case, we write an → a or
limn→∞ an = a and call a ∈ X a limit of {an }n≥1 .
Given a sequence {an }n≥1 in (X, d) and a sequence {nj }j≥1 of positive integers with n1 < n2 <
· · · , the sequence {anj }j≥1 is called a subsequence of {an }n≥1 .
Now, we have the following results concerning sequences and subsequences:
• Let {an }n≥1 , {bn }n≥1 be sequences in Rm and {αn }n≥1 be a sequence in R. Suppose that
an → a, bn → b, and αn → α. Then, we have

an + bn → a + b, an bn → ab, αn an → αa,

where an bn ∈ Rm denotes the component–wise product of the vectors an and bn .

• If a, a0 ∈ X are such that an → a and an → a0 , then a = a0 . In other words, the limit of a

sequence, if exists, is unique.

• If {an }n≥1 converges, then the set {an }n≥1 is bounded.

• The sequence {an }n≥1 converges to a iff every subsequence of {an }n≥1 converges to a.

• If {an }n≥1 is a sequence in a compact set S ⊆ Rm , then there exists a subsequence of {an }n≥1
that converges to a point in S.

• Every bounded sequence in Rm contains a convergent subsequence.

• If {an }n≥1 is a monotonic sequence in R (i.e., either an ≤ an+1 or an ≥ an+1 for n = 1, 2, . . .),
then it converges iff it is bounded.
As an illustration, consider the set S = [−1, 1] ⊆ R. Since S is closed and bounded, it is compact.
Now, consider the sequence an = (−1)n , where n = 1, 2, . . .. Note that the sequence {an }n≥1 does
not converge. However, it has two convergence subsequences; namely, {an }n odd and {an }n even .
The former converges to −1 ∈ S, while the latter converges to 1 ∈ S.
Finally, given a sequence {an }n≥1 in R, we define its limit superior and limit inferior by

lim sup {an } = inf sup {aj }

n≥1 j≥n

and
lim inf {an } = sup inf {aj },
n≥1 j≥n

respectively. We then have the following results:

• a = lim sup {an } iff (i) for every  > 0, there exists an N such that an < a +  for all n ≥ N
and (ii) for every  > 0 and integer N ≥ 1, there exists an n ≥ N such that an > a − .

3
 • a = lim inf {an } iff (i) for every  > 0, there exists an N such that an > a −  for all n ≥ N
and (ii) for every  > 0 and integer N ≥ 1, there exists an n ≥ N such that an < a + .

• lim inf {an } ≤ lim sup {an }.

• The sequence {an }n≥1 converges to a iff a = lim inf {an } = lim sup {an }.

As an example, consider again the sequence an = (−1)n , where n = 1, 2, . . .. For any n ≥ 1, we

have supj≥n {aj } = 1 and inf j≥n {aj } = −1, whence lim sup {an } = 1 and lim inf {an } = −1.

3 Functions
In this section, let (X, dX ), (Y, dY ) be metric spaces and S ⊆ X be a subset of X. Furthermore,
let f : S → Y be a function.
We begin with the notion of a limit of a function. Let x̄ ∈ X be a limit point of S (note that
x̄ need not be in S). Then, we write limx→x̄ f (x) = ȳ if there exists a ȳ ∈ Y with the following
property: for every  > 0, there exists a δ > 0 such that dY (f (x), ȳ) <  for all x ∈ S with
0 < d(x, x̄) < δ.
We can also rephrase the above definition in terms of limits of sequences. Specifically, we have
limx→x̄ f (x) = ȳ iff limn→∞ f (xn ) = ȳ for every sequence {xn }n≥1 in S such that xn 6= x̄ and
limn→∞ xn = x̄. From this definition, we see that the limit of a function, if exists, is unique.

3.1 Continuity
Let x ∈ S. We say that f is continuous at x if for every  > 0, there exists a δ > 0 such that
dY (f (x), f (x0 )) <  for all x0 ∈ S with dX (x, x0 ) < δ. If f is continuous at every x ∈ S, then we say
that f is continuous on S.
The notion of continuity has many characterizations. We list some of them below.

• Let x̄ ∈ S be a limit point of S. Then, f is continuous at x̄ iff limx→x̄ f (x) = f (x̄).

• f is continuous on S iff the set f −1 (T ) ≡ {x ∈ S : f (x) ∈ T } is open for every open set
T ⊆Y.

• f is continuous on S iff the set f −1 (T ) is closed for every closed set T ⊆ Y .

Moreover, the following hold:

• Let f : S → R and g : S → R be continuous on S. Then, f + g and f g are continuous on S.

If g 6= 0 on S, then f /g is also continuous on S.

• Let X, Y, Z be metric spaces and S ⊆ X be a subset of X. Consider the functions f : S →

f (S) ⊆ Y and g : f (S) → Z. Suppose that f is continuous at x ∈ S and g is continuous at
f (x) ∈ f (S) ⊆ Y . Then, the function h : S → Z, which is given by h(x) = g ◦ f (x) = g(f (x)),
is continuous at x ∈ S.

• (Intermediate Value Theorem) If S is connected in X and f is continuous on S, then

f (S) is connected in Y . In particular, if S is connected, f : S → R is continuous, and there
exist x, x0 ∈ S such that f (x) < r < f (x0 ) for some r ∈ R, then there exists an x̄ ∈ S such
that f (x̄) = r.

4
 An important property of continuous functions is that they behave well on compact sets. Before
we make this statement precise, let us state a definition. Let X be any set. We say that a function
f : X → Rn is bounded if there exists an M < ∞ such that kf (x)k2 ≤ M for all x ∈ X.
Now, we have the following results concerning continuous functions on compact sets in Rl :
• Let S ⊆ Rl be a compact set and f : S → Rn be a continuous function on S. Then, the
set f (S) ≡ {y ∈ Rn : y = f (x) for some x ∈ S} is closed and bounded. In particular, f is
bounded.
• Let S ⊆ Rl be a compact set and f : S → R be a continuous function on S. Define
m = sup f (x) = sup f (S) and m = inf f (x) = inf f (S).
x∈S x∈S

Then, there exist x0 , x00 ∈ S such that m = f (x0 ) and m = f (x00 ); i.e., the function f attains
its maximum and minimum over S at x0 ∈ S and x00 ∈ S, respectively.
The last result is very important in the context of optimization, as it gives a sufficient condition
under which an optimization problem has an optimal solution. As an illustration, let S = (0, ∞)
and consider the function f : S → R defined by f (x) = 1/x. Clearly, S is not compact, as it is
neither closed nor bounded. Now, observe that inf x∈S f (x) = 0. However, there does not exist an
x0 ∈ S such that f (x0 ) = 0.
A stronger notion of continuity is that of Lipschitz continuity. Let S ⊆ X and f : S → Y .
We say that f is Lipschitz continuous on S with parameter L ∈ (0, ∞) if for any x, x0 ∈ S, we have
dY (f (x), f (x0 )) ≤ L · dX (x, x0 ).
It is clear that f is continuous on S whenever it is Lipschitz continuous on S. However, the converse
does not hold in general. For instance, consider the function f : R → R+ defined by f (x) = |x|1/2 .
It can be easily verified that f is continuous but not Lipschitz continuous on R.

3.2 Differentiation
3.2.1 Univariate Functions
In this course, we will frequently need to compute the derivatives of functions. Before we discuss
multivariate differentiation, let us first review some of the elements in the theory of univariate
differentiation. To begin, let f : [a, b] → R be a function. For any x ∈ (a, b), define
f (t) − f (x)
φ(t) = where a < t < b, t 6= x
t−x
and set f 0 (x) = limt→x φ(t), provided that the limit exists in accordance with the definition given
at the beginning of Section 3. The function f 0 is called the derivative of f , and whenever f 0 is
defined at x ∈ (a, b), we say that f is differentiable at x. If f 0 is defined at every point of a set
S ⊆ (a, b), then we say that f is differentiable on S. Note that if f is differentiable at x ∈ (a, b),
then it is continuous at x.
Before we proceed further, let us recall three fundamental results in the theory of differentiation.
• Let f : [a, b] → R be continuous. Suppose that f 0 (x) exists at some x ∈ (a, b). Furthermore,
suppose that g is defined on an interval I that contains the range of f (i.e., f ([a, b]) ⊆ I) and
g is differentiable at f (x). If we define a function h : [a, b] → R by h(t) = g(f (t)), then the
Chain Rule asserts that h is differentiable at x and h0 (x) = g 0 (f (x))f 0 (x).

5
 • Let f, g : [a, b] → R be continuous functions that are differentiable on (a, b). Then, the Mean
Value Theorem asserts the existence of an x ∈ (a, b) at which

(f (b) − f (a))g 0 (x) = (g(b) − g(a))f 0 (x).

By taking g(x) = x, we see that if f : [a, b] → R is a continuous function that is differentiable

on (a, b), then there exists an x ∈ (a, b) such that

f (b) − f (a) = (b − a)f 0 (x).

• Let f : [a, b] → R be continuous and define F : [a, b] → R by

Z x
F (x) = f (t) dt.
a

Then, the Fundamental Theorem of Calculus asserts that F is continuous on [a, b] and
differentiable on (a, b). Moreover, we have F 0 (x) = f (x) for all x ∈ (a, b).
Now, if f has a derivative f 0 on an interval and if f 0 is itself differentiable, then we can speak of
the derivative of f 0 , which we denote by f 00 , and call it the second derivative of f . By continuing
in this manner, we obtain functions f, f 0 , f 00 , f (3) , . . . , f (n) , . . ., each being the derivative of the
preceding one. We call f (n) the n–th derivative of f .
In many occassions, we will need to approximate a given function by a simpler function, say, a
polynomial. One such approximation is given by Taylor’s Theorem, which asserts the following.
Let f : [a, b] → R be a function that satisfies the following properties:
(i) f (n−1) is continuous on [a, b].

(ii) f (n) (t) exists for every t ∈ (a, b).

Let a ≤ t1 < t2 ≤ b and define
n−1
X f (j) (t1 )
P (t) = (t − t1 )j .
j!
j=0

Then, there exists a t0 ∈ [t1 , t2 ] such that

f (n) (t0 )
f (t2 ) = P (t2 ) + (t2 − t1 )n .
n!
In particular, we see that f can be approximated by a polynomial of degree n − 1, and the error
can be estimated if we have bounds on |f (n) (x)|.

3.2.2 Multivariate Functions

Let us now turn our attention to multivariate functions. Let S ⊆ Rm and f : S → Rn . Suppose
that S contains an open ball centered at x ∈ Rm . Given u ∈ Rm \{0}, define
f (x + tu) − f (x)
f 0 (x, u) = lim ,
t&0 t
provided that the limit exists. The limit is called the directional derivative of f at x with respect
to u. Note that f 0 (x, u) ∈ Rn .

6
 As it turns out, the notion of directional derivative is not the most appropriate generalization
of the notion of derivative (see, e.g., [2, Chapter 2]). Thus, we need another definition. Towards
that end, let S ⊆ Rm and f : S → Rn . Suppose that S contains an open ball centered at x ∈ Rm .
We say that f is differentiable at x if there exists an n × m matrix A such that

f (x + u) − f (x) − Au
→0 as u → 0. (1)
kuk2

The matrix A, which is unique, is called the derivative of f at x and is denoted by Df (x).
As an example, consider f : Rm → Rn given by f (x) = Bx + b, where B ∈ Rn×m and b ∈ Rn .
For any u ∈ Rm , we have f (x + u) − f (x) = Bu. Thus, by taking A = B in (1), we see that
Df (x) = B.
The following result establishes the relationship between the derivative and directional deriva-
tives of a function:

• Let S ⊆ Rm and f : S → Rn . If f is differentiable at x, then all the directional derivatives of

f at x exist. Moreover, we have f 0 (x, u) = Df (x)u for any u ∈ Rm \{0}.

So far it is a bit awkward to apply the definition directly to compute the derivative of a multivariate
function. To simplify the computation, let us introduce another definition. Let S ⊆ Rm and
f : S → R. We define the i–th partial derivative of f at x, denoted by Di f (x), to be the
directional derivative of f at x with respect to the i–th basis vector ei , provided that it exists. In
other words, we define

f (x + tei ) − f (x)
Di f (x) = lim for i = 1, 2, . . . , m.
t→0 t
Note that Di f (x) ∈ R, as f is a real–valued function. Now, it can be shown that whenever
f : S → R is differentiable at x, we have

Df (x) = D1 f (x) D2 f (x) · · · Dm f (x) ∈ Rm .

 

We remark that the column vector Df (x)T is also known as the gradient of f and is denoted by
∇f (x).
To generalize the above result to vector–valued functions (i.e., functions of the form f : S → Rn ),
we first write f (x) = (f1 (x), f2 (x), . . . , fn (x)), where f1 , . . . , fn : S → R. Then, the following hold:

• f is differentiable at x iff each fi is differentiable at x.

• If f is differentiable at x, then we have

 
Df1 (x)
.. n×m
Df (x) =  ∈R .
 
.
Dfn (x)

The matrix Df (x) is sometimes known as the Jacobian matrix of f .

By viewing Df as a function from Rm to Rn×m ∼ = Rmn , we may define the derivative of Df , denoted
by D2 f , using the above results, provided that the derivative exists. For instance, when n = 1, we

7
have  
D1 D1 f (x) D2 D1 f (x) ··· Dm D1 f (x)
 
 D1 D2 f (x) D2 D2 f (x) · · · Dm D2 f (x) 
2  ∈ Rm×m .
 
D f (x) =  .. .. ..
 .. 
 . . . . 
 
D1 Dm f (x) D2 Dm f (x) · · · Dm Dm f (x)
In this case, the matrix (D2 f (x))T is also known as the Hessian of f and is denoted by ∇2 f (x).
Finally, we can formulate the Multivariate Chain Rule as follows. Let S ⊆ Rm and T ⊆ Rn .
Let f : S → Rn and g : T → Rp be such that f (S) ⊆ T . Let x ∈ S and suppose that y = f (x).
If f is differentiable at x and g is differentiable at y, then the function h : S → Rp defined by
h(u) = g(f (u)) is differentiable at x. Moreover, we have

Dh(x) = Dg(f (x))Df (x),

where the product on the right–hand side is simply matrix multiplication.

References
[1] A. N. Kolmogorov and S. V. Fomin. Introductory Real Analysis. Dover Publications, Inc., New
York, 1975.

[2] J. R. Munkres. Analysis on Manifolds. Westview Press, Boulder, Colorado, 1991.

[3] H. L. Royden. Real Analysis. Macmillan Publishing Company, New York, third edition, 1988.

[4] W. Rudin. Principles of Mathematical Analysis. International Series in Pure and Applied
Mathematics. McGraw–Hill Book Co., Singapore, third edition, 1976.