Math 5311 – Gateaux differentials and Frechet derivatives

Kevin Long

January 26, 2009

1 Differentiation in vector spaces

Thus far, we’ve developed the theory of minimization without reference to derivatives. We’ve even been able to
compute minimizers of quadratic forms without using derivatives, by proving that the minimizer of a positive
definite quadratic form must be the solution to the algebraic equation Kx = f . However, in computations it’s
convenient and efficient to use derivatives.
We’re developing the theory of minimization of functions set in arbitrary vector spaces, so we need to develop
differential calculus in that setting. In multivariable calculus, you learned three related concepts: directional
derivatives, partial derivatives, and gradients. In arbitrary vector spaces, we will be able to develop a gener-
alization of the directional derivative (called the Gateaux differential) and of the gradient (called the Frechet
derivative). We won’t go deeply into the theory of these derivatives in this course, but we’ll establish the basic
differentiation rules.
A reference for differentiation in infinite-dimensional vector spaces, done at the level of this course, is D. R.
Smith, Variational Methods in Optimization, Dover, 1998. A careful development of the finite-dimensional case
can be found in M. Spivak, Calculus on Manifolds, J. Munkres, Analysis on Manifolds, or Rudin, Principles of
Mathematical Analysis. If you’re interested, E. W. Cheney’s Analysis for Applied Mathematics gives a treatment of
Gateaux and Frechet derivatives at a level one notch above the level of this course.

1.1 The Gateaux differential

The Gateaux differential generalizes the idea of a directional derivative.

Definition 1. Let f : V → U be a function and let h 6= 0 and x be vectors in V. The Gateaux differential dh f is
defined
f ( x + eh) − f ( x )
dh f = lim .
e →0 e

Some things to notice about the Gateaux differential:

• There is not a single Gateaux differential at each point. Rather, at each point x there is a Gateaux differ-
ential for each direction h. In one dimension, there are two Gateaux differentials for every x: one directed
“forward,” one “backward.” In two of more dimensions, there are infinitely many Gateaux differentials
at each point!
• The Gateaux differential is a one-dimensional calculation along a specified direction h. Because it’s one-
dimensional, you can use ordinary one-dimensional calculus to compute it. Your old friends such as the
chain rule work for Gateaux differentials. Thus, it’s usually easy to compute a Gateaux differential even
when the space V is infinite dimensional.

1
1.2 Examples of Gateaux differentials

1.2.1 Linear and quadratic functions

Let f and x be vectors in an inner product space, and define p( x ) = x T f . Then

x T f + eh T f − x T f
dh p = lim
e →0 e

dh ( x T f ) = hT f .

Next, let K be a symmetric matrix, and define p( x ) = 2x T f + x T Kx. Compute the Gateaux differential

2x T f + eh T f + x T Kx + 2eh T Kx + e2 h T Kh − 2x T f − x T Kx
dh p = lim
e →0 e
= h T f + 2h T Kx.

1.2.2 The exponential function

There are a number of ways to define the exponential function; most convenient for present purposes is to
define it as the infinite series

x2 xk
ex = 1 + x + +···+ +··· .
2 k!

This makes sense even for vector-valued arguments provided multiplication is understood as “elementwise,”
is in the Matlab “dot-star” operation. In the case of an infinite-dimensional vector space V (for example H 1 )
whose elements are real-valued functions, the exponential function eu : V → V simply maps u( x ) pointwise
to its exponential eu( x) .
With e x defined as above, it can be shown (provided multiplication of members of V is commutative!) by
straightforward calculation and rearrangement that e a+b = e a eb (see, e.g., blue or green Rudin. If you’re
interested, see a reference on Lie algebras for information on the exponentiation of non-commutative objects;
that turns out to be useful in fields such as aircraft control and quantum physics.)
With the identity e a+b = e a eb and the series defining e x , we can compute the Gateaux derivative

eu eeh − eu
dh (eu ) = lim
e →0 e
eeh − 1
= eu lim
e →0 e
= heu .

1.2.3 The absolute value function in R

Let f ( x ) = | x |. Calculation of the limit gives

(
h | xx| x 6= 0
dh f =
|h| x = 0.

Notice that the Gateaux differentials of | x | do exist at zero; however, at zero, the Gateaux differentials depend
on h in a nonlinear way.

2
1.2.4 A spatial derivative

u x + eh x − u x
 
du
dh = lim
dx e →0 e

dh
=
dx

1.2.5 A functional

Let J : H 1 (Ω) → R be   
1 2 1 2
J [u] = u + u dx.
Ω 2 x 2
Then

h1 2 1 2 1 2 2 1
i
2 − 1 u2 − 1 u2 dx
Ω 2 u x + 2 u + euh + eu x h x + 2 e h x + 2 eh 2 x 2
dh J = lim
e →0 e

dh J = [uh + u x h x ] dx
Ω

Note: it’s routine in infinite-dimensional optimization problems to exchange integration and Gateaux differ-
entiation without comment. The integration variable x and the differentiation variable u are different. If the
limit e → 0 can be taken by means of a sequence {en } for which the difference quotient of the integrand is
dominated by a Lebesgue-integrable function, the conditions of the Lebesgue dominated convergence theorem
hold and the limit and integral can be interchanged safely. For the section on optimization, we’ll assume this to
be true throughout. However, later in the semester we’ll see some examples with Fourier series and transforms
where differentiation of a convergent series or integral results in divergence at one or more points.
There is, however, an important real-world case where this process requires a little more care: the case where
the limits of integration depend on the variable u. This arises in the field of shape optimization, where the “design
variable” u determines the size and shape of a domain Ω.

1.3 Rules for Gateaux differentials

Computing from the definition quickly gets dull, so as with ordinary calculus, let’s work out rules for Gateaux
differentials.

1.3.1 Differential of a constant

The Gateaux differential of a constant is zero: dh c = 0. The proof follows immediately from the definition.

1.3.2 Sum rule

Gateaux differentiation distributes over sums: dh ( f ± g) = dh f ± dh g. The proof follows immediately from
the definition.

1.3.3 Product rule and quotient rule

The Gateaux differential of an elementwise product f g is dh ( f g) = (dh f ) g + f (dh g).

The Gateaux differential of an inner product h f , gi (or f T g) is dh h f , gi = h f , dh gi + hdh f , gi . With transpose
notation, this is dh ( f T g) = f T dh g + (dh f ) T g.
The proofs are similar to what you would do in R, and are left as exercises; the little-o notation below will be
useful.