2 Notation, definitions, and mathematical

foundation

2.1 Basic notation and definitions

Definition 1: Standard CNN architecture

A standard CNN architecture consists of an M × N rectangular array of cells (C(i, j))
with Cartesian coordinates (i, j), i = 1, 2, . . . , M, j = 1, 2, . . . , N (Fig. 2.1).

Column
1 2 3 j N
1

3
C(i, j)
Row i

Fig. 2.1.

Remark:
There are applications where M = N . For example, a 5 × 512 CNN would be more
appropriate for a scanner, fax machine, or copy machine.

Definition 2: Sphere of influence of cell C(i, j)

The sphere of influence, Sr (i, j), of the radius r of cell C(i, j) is defined to be the set
of all the neighborhood cells satisfying the following property

Sr (i, j) = {C(k, l)| max {|k − i|, |l − j|} ≤ r } (2.1)

1≤k≤M,1≤l≤N

where r is a positive integer.

7
8 Notation, definitions, and mathematical foundation

C(i, j ) C(i, j )

(a) (b)

Fig. 2.2. (a) r = 1 (3 × 3 neighborhood), (b) r = 2 (5 × 5 neighborhood).

We will sometimes refer to Sr (i, j) as a (2r + 1) × (2r + 1) neighborhood. For

example, Fig. 2.2(a) shows an r = 1 (3 × 3) neighborhood. Fig. 2.2(b) shows an r = 2
(5 × 5) neighborhood.

Remarks:
1 In IC implementations, every cell is connected to all its neighbors in Sr (i, j) via
“synaptic” circuits.
2 When r > N /2, and M = N , we have a fully connected CNN where every cell
is connected to every other cell and Sr (i, j) is the entire array. This extreme case
corresponds to the classic Hopfield Net. It is impractical to build any reasonable size
(several thousand cells) Hopfield Net in a VLSI chip. There exists a “commercial”
Hopfield-like chip by INTEL called “ETANN,” type 80170 ($500 in 1995). This
chip has 64 cells which makes it more of an expensive “toy.”

Definition 3: Regular and boundary cells

A cell C(i, j) is called a regular cell with respect to Sr (i, j) if and only if all
neighborhood cells C(k, l) ∈ Sr (i, j) exist. Otherwise, C(i, j) is called a boundary
cell (Fig. 2.3).

Remark:
The outermost boundary cells are called edge cells. Not all boundary cells are edge
cells if r > 1.

Definition 4: Standard CNN

A class 1 M × N standard CNN is defined by an M × N rectangular array of cells
C(i, j) located at site (i, j), i = 1, 2, . . . , M, j = 1, 2, . . . , N . Each cell C(i, j) is
defined mathematically by:

1 State equation


ẋi j = −xi j + A(i, j; k, l)ykl + B(i, j; k, l)u kl + z i j (2.2)
C(k,l)∈Sr (i, j) C(k,l)∈Sr (i, j)
9 2.1 Basic notation and definitions

Boundary cell (if r = 1)

Corner cell

Fig. 2.3.

where xi j ∈ R, ykl ∈ R, u kl ∈ R, and z i j ∈ R are called state, output, input,

and threshold of cell C(i, j), respectively. A(i, j; k, l) and B(i, j; k, l) are called the
feedback and the input synaptic operators to be defined below.

2 Output equation
1 1
yi j = f (xi j ) = |xi j + 1| − |xi j − 1| (2.3)
2 2
This is called standard nonlinearity (Fig. 2.4).

yij

–1 0 1 xij

–1

Fig. 2.4.

3 Boundary conditions
The boundary conditions are those specifying ykl and u kl for cells belonging to Sr (i, j)
of edge cells but lying outside of the M × N array.
10 Notation, definitions, and mathematical foundation

4 Initial state
xi j (0), i = 1, . . . , M, j = 1, . . . , N (2.4)

Remarks:
1 The input u kl is usually the pixel intensity of an M × N gray-scale image or picture
P, normalized without loss of generality, to have the range −1 ≤ u kl ≤ +1 where
“white” is coded by −1 and “black” is coded by +1. For a still image, u kl is a
constant for all time, for a moving image (video) u kl will be a function of time.
Other variables (x(0), y, z) can also be specified as images.
2 In the most general case, A(i, j; k, l), B(i, j; k, l), and z i j may vary with position
(i, j) and time t. Unless otherwise stated, however, we will assume they are space
and time invariant.
3 In the most general case both A(i, j; k, l) and B(i, j; k, l) are nonlinear operators1
which operate on xkl (t), ykl (t), u kl (t), xi j (t), yi j (t), and u i j (t), 0 ≤ t ≤ t0 , to
produce a scalar (A(i, j; k, l) ◦ ykl )(t0 ) and (B(i, j; k, l) ◦ u kl )(t0 ), 0 ≤ t ≤ t0 .
4 We may also introduce synaptic laws depending on the states (C template) and on
mixed variables (D template), respectively.
That is (C(i, j; k, l) ◦ xkl )(t0 ) and (D(i, j; k, l) ◦ (u kl , xkl , ykl )(t0 ).

Unless otherwise stated, however, A(i, j; k, l)ykl and B(i, j; k, l)u kl will denote
ordinary multiplication with real coefficients where they may be nonlinear functions of
states, inputs, and outputs of cells C(i, j), C(k, l) and may involve some time delays
(i.e., they may contain a finite time history, as in the case of having a time delay).
The following are some space and time invariant nonlinear examples chosen from
the CNN catalog of applications (CNN Software Library). See some of them in
TEMLIB (Appendix A).

EXAMPLE 2.1:

a( yij )
1

– 0.025 0.025 yij

–1

Fig. 2.5.
11 2.1 Basic notation and definitions

A(i, j; k, l) = a(yi j ): depends on output (from TEMPLATE MajorityVoteTaker)

(Fig. 2.5).

EXAMPLE 2.2:

C(i, j; k, l) = c(xi j ): depends on state (from TEMPLATE LGTHTUNE) (Fig. 2.6).

c(xij)

1
0 0.2 xij

–3

Fig. 2.6.

EXAMPLE 2.3:

A(i, j; k, l) = a(u i j , u kl ) and B(i, j; k, l) = b(u i j , u kl ): depends on two inputs (from

TEMPLATE GrayscaleLineDetector) (Fig. 2.7).

a(uij, ukl ) b(uij, ukl)

1 1

–0.15 0.15 uij – ukl 0.25 uij – ukl

Fig. 2.7.

EXAMPLE 2.4:

A(i, j; k, l) = a(yi j , ykl ): depends on two outputs (from TEMPLATE GlobalMaxi-

mumFinder) (Fig. 2.8).
12 Notation, definitions, and mathematical foundation

0.25 a( yij, ykl )

0.125

–2 –1 yij – ykl

Fig. 2.8.

EXAMPLE 2.5:
C(i, j; k, l) = c(xi j , xkl ): depends on two states (from TEMPLATE EXTREME)
(Fig. 2.9).

c(xij , xkl )

– 0.2 0.2

xij – xkl
–1

–2
Fig. 2.9.

EXAMPLE 2.6:

D(i, j; k, l) = d(u kl , yi j ): depends on input and output (from TEMPLATE ERO-

SION) (Fig. 2.10).

d (ukl , yij )

0 ukl – yij
–1

Fig. 2.10.

Some examples of time-delay operators:

13 2.1 Basic notation and definitions

EXAMPLE 2.7:

(A(i, j; k, l)ykl )(t) = 0.68ykl (t − 10): depends on the past of the output (from
TEMPLATE SpeedDetection).

EXAMPLE 2.8:

(B(i, j; k, l)u kl )(t) = −0.25u kl (t − 10), (k, l) = (i, j): depends on the past of the
input (from TEMPLATE ImageDifferenceComputation).

Remarks:
1 A gray-scale image can be represented pixelwise using a one-to-one map between a
picture element (pixel) and a cell. The luminance value of the pixel would be coded
as: black → +1, white → −1, gray → (−1, +1).
2 It is useful and correct to think of the triple {A(i, j; k, l), B(i, j; k, l), z i j } as an
elementary CNN instruction (macro), because they specify how an M × N input
image U at t = 0 will be transformed to produce an M × N output image Y(t) for
all t ≥ 0.

Definition 5: Space-invariant or isotropic CNN

A CNN is space-invariant or isotropic if and only if both the synaptic operators
A(i, j; k, l), B(i, j; k, l) and the threshold z i j do not vary with space. In this case,
we can write




A(i, j; k, l)ykl = A(i − k, j − l)ykl
C(k,l)∈Sr (i, j) |k−i|≤r |l− j|≤r




B(i, j; k, l)u kl = B(i − k, j − l)u kl
C(k,l)∈Sr (i, j) |k−i|≤r |l− j|≤r
zi j = z (2.5)

A standard CNN (with linear synaptic operators) has the following state equation
(using the same notation as in (2.2)):



ẋi j =−xi j + A(i, j; k, l)ykl + B(i, j; k, l)u kl + z i j
C(k,l)∈Sr (i, j) C(k,l)∈Sr (i, j)



+ C(i, j; k, l)xkl + D(i, j; k, l)(u kl , xkl , ykl ) (2.2∗ )
C(k,l)∈Sr (i, j) C(k,l)∈Sr (i, j)
14 Notation, definitions, and mathematical foundation

2.2 Mathematical foundations

2.2.1 Vector and matrix representation and boundary conditions

The system of n = M N ordinary differential equations (ODE) defining a standard (not
necessarily space-invariant) CNN can be recast in the form

ẋi j = h i j (x̃i j , t), i = 1, 2, . . . , M, j = 1, 2, . . . , N (2.6)

where


h i j (x̃i j , t) = −xi j (t) + A(i, j; k, l)ykl (t) + si j (t)
C(k,l)∈Sr (i, j)

where

ykl = f (xkl )


si j (t) = B(i, j; k, l)u kl (t) + z i j (t)
C(k,l)∈Sr (i, j)

x̃i j is a vector of length (2r + 1)2 whose components include all variables xkl ∈
Sr (i, j), i.e.

{xkl : |k − i| ≤ r, |l − j| ≤ r }

We can cast Eq. (2.6) into the following M × N matrix differential equation which
exhibits a one-to-one correspondence with the CNN architecture
 
ẋ11 ẋ12 ... ẋ1N
 ẋ ... ẋ2N 
 21 ẋ22 
 .. .. .. 
 
 . . . 
 
 ẋ M−1,1 ẋ M−1,2 . . . ẋ M−1,N 
ẋ M1 ẋ M2 ... ẋ M N
 
h 11 (x̃11 ) h 12 (x̃12 ) ... h 1N (x̃1N )
 h 21 (x̃21 ) h 22 (x̃22 ) ... h 2N (x̃2N ) 
 
 . . . 
=  .. .. ..  (2.7)

 
 h M−1,1 (x̃ M−1,1 ) h M−1,2 (x̃ M−1,2 ) . . . h M−1,N (x̃ M−1,N ) 
h M1 (x̃ M1 ) h M2 (x̃ M2 ) ... h M N (x̃ M N )

Definition 6: Virtual cells

Any cell C(k, l), with |k − i| ≤ r, |l − j| ≤ r , and k ∈ / {1, 2, . . . , M} and/or l ∈ /
{1, 2, . . . , N } is called a virtual cell, and the associated xkl , ykl , u kl , and z kl are called
virtual state, virtual output, virtual input, and virtual threshold, respectively.
15 2.2 Mathematical foundations

Boundary conditions
Any virtual variable in xi j of Eq. (2.6) must be specified via various boundary
conditions which are the most commonly used for a 3 × 3 neighborhood.

1 Fixed (Dirichlet) boundary conditions

Left virtual cells: yi,0 = α1 , u i,0 = β1 , i = 1, 2, . . . , M
Right virtual cells: yi,N +1 = α2 , u i,N +1 = β2 , i = 1, 2, . . . , M
Top virtual cells: y0, j = α3 , u 0, j = β3 , j = 1, 2, . . . , N
Bottom virtual cells: y M+1, j = α4 , u M+1, j = β4 , j = 1, 2, . . . , N
where αi and βi are user prescribed constants (usually equal to zero).
Circuit interpretation: Add one row or column along the boundary and force each
cell to have a fixed input and output by batteries (Fig. 2.11).

M×N
CNN

Fig. 2.11. The circuit interpretation of the fixed (Dirichlet) boundary condition.

2 Zero-flux (Neumann) boundary conditions (Fig. 2.12)

Left virtual cells: yi,0 = yi,1 , u i,0 = u i,1 , i = 1, 2, . . . , M
Right virtual cells: yi,N +1 = yi,N , u i,N +1 = u i,N , i = 1, 2, . . . , M
Top virtual cells: y0, j = y1, j , u 0, j = u 1, j , j = 1, 2, . . . , N
Bottom virtual cells: y M+1, j = y M, j , u M+1, j = u M, j , j = 1, 2, . . . , N .

Remark:
This boundary condition usually applies to the case where there is no input, i.e., u i j =
0 for all (i, j). Because any input would cause energy and/or material flow from the
outside making the system an “open system” in the sense of thermodynamics, CNN
16 Notation, definitions, and mathematical foundation

x11 x1N

x11 x1N

M×N
CNN
xM1 xMN

xM1 xMN

Fig. 2.12. The circuit interpretation of the Neumann boundary condition.

with zero input is called autonomous CNN, and constitutes an important class with
widespread applications in pattern formation and “autowave” generation.

3 Periodic (Toroidal) boundary conditions

Left virtual cells: yi,0 = yi,N , u i,0 = u i,N , i = 1, 2, . . . , M
Right virtual cells: yi,N +1 = yi,1 , u i,N +1 = u i,1 , i = 1, 2, . . . , M
Top virtual cells: y0, j = y M, j , u 0, j = u M, j , j = 1, 2, . . . , N
Bottom virtual cells: y M+1, j = y1, j , u M+1, j = u 1, j , j = 1, 2, . . . , N .

A B
A B

M N
D
C
CNN

D C

Fig. 2.13. The circuit interpretation of the Periodic (Toroidal) boundary condition.

Identify each cell from the top row with the corresponding cell in the bottom row,
identify each cell from the left column with the corresponding cell in the right column.
17 2.2 Mathematical foundations

This is equivalent to fabricating a CNN chip on a silicon torus as its substrate.

Vector differential equation

Since virtually all theorems and numerical techniques for solving systems of ODE
are formulated in vector form, we must repack the M × N matrix ODE (2.6) into an
M N × 1 vector ODE. There are many ways to order the variables. We consider three
typical orders:

1 Row-wise packing scheme

2 Diagonal packing scheme
3 Column-wise packing scheme

These packing schemes are shown in Figs 2.14(a), (b), and (c), respectively.

11 12 13 14 1N 11 12 13 1N
11 1N
22 23
21 2N 21 2N

31 3N
31 3N
41

51

M1 MN M1 MN
M1 M2 M3 MN
(a) (b) (c)

Fig. 2.14. Three (among many others) possible packing schemes.

After repacking, we obtain a system of n = M N vector systems

           
x̂˙ 1 x̂1 ŷ1 û 1 ẑ 1
 
˙x̂ 2   x̂2     ŷ2     û 2   ẑ 2 
          
 .. 
  = − ..  +   . +
.
 . + .
.

 .   .     .     .   .. 
x̂˙ n x̂n ŷn û n ẑ n
      
ẋ x  y B̂ u z

or in vector form

ẋ = −x + Ây + B̂u(t) + z(t)

yi = f (xi ) (2.8)

where x = [x̂1 , x̂2 , . . . , x̂n ]T is the state vector with the same order of state variables.
The two matrices  and B̂ are n × n matrices whose nonzero entries are the synaptic
weights A(i, j; k, l) and B(i, j; k, l), respectively, corresponding to the above three
18 Notation, definitions, and mathematical foundation

0
q q

0
Fig. 2.15. The band structure of  and B̂.

packing schemes. Each matrix is quite sparse (most entries are zero) with the band
structure shown in Fig. 2.15.
For M = N , the band has a bandwidth

w = 2q + 1

where
q = N + 1 for row-wise packing schemes,
q = 2N − 2 for diagonal packing schemes,
q = 2N − 1 for column-wise packing schemes.

The band corresponding to the above three packing schemes can be divided into two
or more diagonal sub-bands, each of which is a sparse matrix.

Remarks:
 and B̂ are very large matrices, e.g., for M = N = 1000 (for HDTV applications),
n = 1,000,000, q = 1001, w = 2003 for row-wise packing scheme, which is only
0.2% of L = 106 (L is the bandwidth of the full matrix). This shows  and B̂ are very
sparse matrices.

2.2.2 Existence and uniqueness of solutions

To motivate the importance of the questions of “existence and uniqueness of solutions”
for a CNN, a question which has never been an issue in linear circuit and system theory,
consider the following three simple nonlinear circuits.

Example 1
Consider the circuit shown in Fig. 2.16(a) whose state equation is given by
1
ẋ = − , t ≥0 (2.9)
2x
19 2.2 Mathematical foundations

The characteristics of the right-hand side are shown in Fig. 2.16(b). The solution of
Eq. (2.9) with initial condition x(0) = x0 > 0 is given by

x(t) = x02 − t, t ≥0 (2.10)

and sketched in Fig. 2.16(c). Observe that this circuit has no solution with any initial
state x0 > 0 for t ≥ T = x02 .

1 1 x
i= x= –
+ + 2v 2x
x0
x x0
v
x
1F 0
t
– – 0 T = x02

(a) (b) (c)

Fig. 2.16. Example of a circuit which has no solution after a finite time T .

Example 2
Consider the circuit shown in Fig. 2.17(a) whose state equation is given by
3 1/3
ẋ = x (2.11)
2
The characteristics of the right-hand side are shown in Fig. 2.17(b). The solution of
Eq. (2.11) with initial condition x(0) = 0 is given by
0, 0≤t ≤T
x(t) = (2.12)
(t − T )3/2 , t>T
for any T ∈ R. This solution is shown in Fig. 2.17(c) for different choices of
T = T1 , T2 , . . . , TN . Since T is arbitrary, this circuit has an infinite number of distinct
solutions.

Example 3
Consider the circuit shown in Fig. 2.18(a) whose state equation is given by
ẋ = x 2 (2.13)
The characteristics of the right-hand side are shown in Fig. 2.18(b). The solution of
Eq. (2.13) with initial state x(0) = 1 is given by
1
x= (2.14)
1−t
20 Notation, definitions, and mathematical foundation

3 3 3 1/3 x
i=– v x= x
+ + 2 2

x v
1F x
0
t
– – 0 T1 T2 TN

(a) (b) (c)

Fig. 2.17. Example of a circuit having infinitely many distinct solutions, all with the same initial
state x(0) = 0.

As shown in Fig. 2.18(c), this circuit has a solution which cannot be continuous
beyond t ≥ 1 because it blows up at t = 1. This phenomenon is called a finite escape
time.

x
i = –v2 x= x2
+ +
x0
x v
1F 1
x
0 1 t
– – 0 1
(a) (b) (c)

Fig. 2.18. Example of a circuit having a finite escape time.

The examples in Fig 2.18 show that even a simple two-element nonlinear circuit
may not have a solution, and even if a solution exists it may not be unique, or may not
be continued for all times t ≥ 0. How do we determine whether a nonlinear circuit has
a unique solution for all times t ≥ 0. This is of fundamental importance for us since
we will be concerned with asymptotic behavior as t → ∞. There is no general method
to answer this fundamental question since any method or theorem must exclude such
simple equations as (2.9), (2.11), and (2.13)! Fortunately, for CNN, we can prove the
following:

Theorem 1: Global existence and uniqueness theorem

Assume the standard CNN described by Eq. (2.2) satisfies the following three
hypotheses:
21 2.2 Mathematical foundations

H1: The synaptic operators are linear and memoryless, i.e., A(i, j; k, l)ykl and
B(i, j; k, l)u kl are scalar multiplications, where A(i, j; k, l) and B(i, j; k, l) are
real numbers.
H2: The input u i j (t) and threshold z i j (t) are continuous functions of time.
H3: The nonlinearity f (x) is Lipschitz continuous in the sense that there exists a
constant L such that for all x  , x  ∈ R

 f (x  ) − f (x  ) ≤ Lx  − x   (2.15)

(Note: for the scalar case, the norm is equivalent to the absolute value.)

Then for any initial state xi j (0) ∈ R, the CNN has a unique solution for any t ≥ 0 (see
Fig. 2.19).

f (x)

–L
L

x
0 x

Fig. 2.19.

Remark:
The standard CNN satisfies all three hypotheses H1, H2, and H3.

Proof:
Hypotheses H1 implies we can recast Eq. (2.1) into the vector form

ẋ = −x + Ây + B̂u(t) + z(t) = h(x, t) yi = f (xi ) (2.16)

We show first that h(x, t) is Lipschitz continuous with respect to x. Choose any x ,
x ∈ Rn , and let y = [ f (x1 ), . . . , f (xn )]T , so that, y = f(x ) and y = f(x ).

h(x , t) − h(x , t) = −x + Ây + x − Ây 

= x − x + Â(y − y )
≤ x − x  + Ây − y  (2.17)
22 Notation, definitions, and mathematical foundation

Now
   

 f (x 1 ) f (x 1 )


 f (x 2 )   
f (x 2 )
     
y − y  =  .. − .. 
 .   . 
 
 f (x n ) f (x n ) 



= | f (x 1 ) − f (x 1 )|2 + · · · + | f (x n ) − f (x n )|2

≤ L 2 |x1 − x1 |2 + L 2 |x2 − x2 |2 + · · · + L 2 |xn − xn |2

(in view of hypothesis H3)
  
 x1 − x1 
  
 x − x  
 2 2 
= L .. 
 . 
 
 x  − x  
n n
 
= L x − x  (2.18)
Substituting equation (2.18) into equation (2.17), we obtain
 
h(x , t) − h(x , t) ≤ 1 + L Â x − x 
= L̂x − x  (2.19)

where L̂ = 1 + LÂ is a global Lipschitz constant, independent of x and t.

Hence, h(x, t) is uniformly Lipschitz continuous with respect to x. Moreover, since
for each x0 ∈ Rn , h(x0 , t) is continuous for all t, in view of H2, we can find a bound
Mx0 t0 a (which in general depends on x0 , t0 , and a) such that
h(x 0 , t) ≤ Mx0 ,t0 ,a ∀ t ∈ [t0 , t0 + a]
Then for all x, such that
x − x0  ≤ b, t ∈ [t0, t0 + a]
h(x, t) ≤ Mx0 ,t0 ,a + L̂b
in view of the Lipschitz continuity of h(x, t).
It follows from the Picard–Lindelof Theorem2 that a unique solution exists for a
time duration of
 
b
min a, second
Mx0 ,t0 ,a + L̂b
By choosing b large enough and a > 1/ L̂, one can see that a solution exists for
approximately 1/ L̂ seconds where L̂ is independent of x0 , t0 , a, and b. We can use the
same procedure to show a unique solution exists for the next 1/ L̂ seconds, etc. So, a
unique solution exists for any time.

23 2.2 Mathematical foundations

2.2.3 Boundedness of solutions

Theorem: Explicit solution bound

For all initial states, inputs, and thresholds satisfying

|xi j (0)| ≤ 1, |u i j (t)| ≤ 1, |z i j (t)| ≤ z max

the solution xi j (t) of the standard CNN with linear memoryless synaptic operators
A(i, j; k, l) and B(i, j; k, l) is uniformly bounded in the sense that there exists a
constant xmax such that

|xi j (t)| ≤ xmax ∀t ≥ 0, 1 ≤ i ≤ M, 1≤ j ≤N

where


xmax = 1 + z max + max (|A(i, j; k, l)| + |B(i, j; k, l)|) (2.20)
1≤i≤M
1≤ j≤N C(k,l)∈Sr (i, j)

Lemma 1 (Appendix 1)
The complete solution of

ẋ = −x + f (t)
x(0) = x0

is given by
 t
x(t) = x0 e−t + e−(t−τ ) f (τ ) dτ, t ≥0
0

In the special case where f (τ ) = f 0 where f 0 is a constant, we have

x(t) = x0 e−t + f 0 (1 − e−t ), t ≥0

The equivalent electrical circuit is as shown in Fig. 2.20.

+
x(t) 1Ω f (t)

Fig. 2.20.
24 Notation, definitions, and mathematical foundation

Proof of Theorem 2:



ẋi j = −xi j + A(i, j; k, l)ykl + B(i, j; k, l)u kl +z i j (2.21)
C(k,l)∈Sr (i, j) C(k,l)∈Sr (i, j)
 
αi j (t) βi j (u(t))

f (t)

where αi j (t) and βi j (t) are defined for all t ≥ 0 in view of Theorem 1. Applying
Lemma 1,
 t
x(t) = xi j (0) e−t + e−(t−τ ) [αi j (τ ) + βi j (u(τ )) + z i j (τ )] dτ (2.22)
0

Applying triangular inequality

 t
x(t) ≤ |xi j (0) e−t | + e−(t−τ ) [|αi j (τ )| + |βi j (u(τ ))| + |z i j (τ )|] dτ
0
 t
−t
≤ |xi j (0)e | + (αmax + βmax + z max ) e−(t−τ ) dτ (2.23)
0

where

αmax = max |αi j (t)| ≤ |A(i, j; k, l)| max ykl (t) (2.24)
t≥0 t≥0
C(k,l)∈Sr (i, j)


βmax = max |βi j (u)| ≤ |B(i, j; k, l)| max u kl (t) (2.25)
u u
C(k,l)∈Sr (i, j)

But
 t
e−(t−τ ) dτ = 1 − e−t < 1, t ≥0 (2.26)
0
|u kl (t)| ≤ 1 (2.27)
|xi j (0)| ≤ 1 (2.28)

Hence, (2.21)–(2.28) imply

|xi j (t)| ≤ |xi j (0) e−t | + αmax + βmax + z max

≤ 1 + z max + |A(i, j; k, l)| + |B(i, j; k, l)|
C(k,l)∈Sr (i, j) C(k,l)∈Sr (i, j)


≤ 1 + z max + max (|A(i, j; k, l)| + |B(i, j; k, l)|)
1≤i≤M
1≤ j≤N C(k,l)∈Sr (i, j)
= xmax

independent of (i, j).

25 2.2 Mathematical foundations

Remarks:
1 For space-invariant CNN, |xi j (t)| is bounded by


 
x̂max = 1 + z max + |Akl | + |B kl | (2.29)
1≤k≤M 1≤l≤N

2 Theorem 2 imposes a minimum power supply voltage for any CNN circuit imple-
mentation, and is fundamental for designing a CNN circuit.

2.2.4 Space-invariant CNN

Cloning template representation
Since the majority of all CNN applications use only space-invariant standard CNNs
with a 3 × 3 neighborhood (sphere of influence r = 1), it will be extremely convenient
to introduce some concise notations and terminologies for future analysis.
Consider a typical cell C(i, j) ∈ Sr (i, j) as follows

C(i − 1, j − 1) C(i − 1, j) C(i − 1, j + 1)

C(i, j − 1) C(i, j) C(i, j + 1)
C(i + 1, j − 1) C(i + 1, j) C(i + 1, j + 1)

Let us examine the contributions from the two synaptic operators and the threshold
items in (2.21).

1 Contributions from the feedback synaptic operator A(i, j; k, l)

In view of space-invariance, we can write




A(i, j; k, l) ykl = A(k − i, l − j)y kl
C(k,l)∈Sr (i, j) |k−i|≤1 |l− j|≤1
= a−1,−1 yi−1, j−1 + a−1,0 yi−1, j + a−1,1 yi−1, j+1
a0,−1 yi, j−1 + a0,0 yi, j + a0,1 yi, j+1
a1,−1 yi+1, j−1 + a1,0 yi+1, j + a1,1 yi+1, j+1


1

1
= ak,l yi+k, j+l (2.30)
k=−1 l=−1

where amn = A(m, n)

a−1,−1 a−1,0 a−1,1 yi−1, j−1 yi−1, j yi−1, j+1

= a0,−1 a0,0 a0,1  yi, j−1 yi, j yi, j+1 = A  Yi j (2.31)
a1,−1 a1,0 a1,1 yi+1, j−1 yi+1, j yi+1, j+1

where the 3 × 3 matrix A is called the feedback cloning template, and the symbol “”
denotes the summation of dot products, henceforth called a template dot product. In
26 Notation, definitions, and mathematical foundation

discrete mathematics, this operation is called “spatial convolution.” The 3 × 3 matrix

Yi j in (2.31) can be obtained by moving an opaque mask with a 3 × 3 window to
position (i, j) of the M × N output image Y , henceforth called the output image at
C(i, j).
An element akl is called a center (resp., surround) element, weight, or coefficient,
of the feedback template A, if and only if (k, l) = (0, 0) (resp., (k, l) = (0, 0)).
It is sometimes convenient to decompose the A template as follows
A = A0 + Ā (2.32)
0 0 0
A = 0 a0,0
0
0
0 0 0
a−1,−1 a−1,0 a−1,1
Ā = a0,−1 0 a0,1
a1,−1 a1,0 a1,1
where A0 and Ā are called the center and surround component templates, respectively.

2 Contributions from the input synaptic operator B(i, j; k, l)

Following the above notation, we can write




B(i, j; k, l)y kl = B(k − i, l − j) u kl
C(k,l)∈Sr (i, j) |k−i|≤1 |l− j|≤1


1

1
= bkl u i+k, j+l (2.33)
k=−1 l=−1

b−1,−1 b−1,0 b−1,1 u i−1, j−1 u i−1, j u i−1, j+1

= b0,−1 b0,0 b0,1  u i, j−1 u i, j u i, j+1 = B  Ui j (2.34)
b1,−1 b1,0 b1,1 u i+1, j−1 u i+1, j u i+1, j+1
where the 3 × 3 matrix B is called the feedforward or input cloning template, and Ui j
is the translated masked input image.
Similarly, we can write
B = B0 + B̄ (2.35)
0 0 0
B = 0 b0,0
0
0
0 0 0
b−1,−1 b−1,0 b−1,1
B̄ = b0,−1 0 b0,1
b1,−1 b1,0 b1,1
where B0 and B̄ are called the center and surround feedforward template, respectively.
27 2.2 Mathematical foundations

3 Contribution from the threshold terms

zi j = z

Using the above notations, a space-invariant CNN is completely described by

ẋi j = −xi j + A  Yi j + B  Ui j + z (2.36)

We will usually decompose (2.36) as follows

ẋi j = −xi j + a00 f (xi j ) + Ā ⊗ Yi j + B ⊗ Ui j + z (2.37)

 
g(xi j ) wi j (t)

where

h i j (xi j ; wi j ) = g(xi j ) + wi j (xi j , t) (2.38)

is called the rate function, g(xi j ) is called the driving-point (DP) component because
it is closely related to a central concept from nonlinear circuit theory, and

wi j (xi j , t) = Ā  Yi j + B  Ui j + z

is called the offset level.

2.2.5 Three simple CNN classes

Each CNN is uniquely defined by three terms of the cloning templates {A, B, z}, which
consist of 19 real numbers for a 3 × 3 neighborhood (r = 1). Since real numbers are
uncountable, there are infinitely many distinct CNN templates, of which the following
three subclasses are the simplest and hence mathematically tractable.

Definition 7: Excitatory and Inhibitory synaptic weights (Fig. 2.21)

A feedback synaptic weight akl is said to be excitatory (resp., inhibitory) if and only if
it is positive (resp., negative).
A synaptic weight is “excitatory” (resp., inhibitory) because it makes the rate
function h i j (xi j , wi j ) more positive (less positive) for a positive input, and hence
increases (resp., decreases) ẋi j , namely the rate of growth of xi j (t).

Definition 8: Zero-feedback (feedforward) class C (0, B, z) (Fig. 2.22)

A CNN belongs to the zero-feedback class C (0, B, z) if and only if all feedback
template elements are zero, i.e., A ≡ 0.
Each cell of a zero-feedback CNN is described by

ẋi j = −xi j + B  Ui j + z (2.39)

28 Notation, definitions, and mathematical foundation

(a) (A, B, z)

B A

Input U State X Output Y

(b) z
U x ij x ij y ij
B + ∫ dt f( )
uij –

A
Y
Fig. 2.21. A space-invariant CNN C(A, B, z) with a 3 × 3 neighborhood S1 (i, j). (a) Signal flow
structure of a CNN with a 3 × 3 neighborhood. The two shaded cones symbolize the weighted
contributions of input and output voltages of cell C(k, l) ∈ S1 (i, j) to the state voltage of the center
cell C(i, j). (b) System structure of a cell C(i, j). Arrows printed in bold mark parallel data paths
from the input and the output of the surround cells u kl and ykl , respectively. Arrows with thinner
lines denote the threshold, input, state, and output, z, u i j , xi j , and yi j , respectively.

Definition 9: Zero-input (Autonomous) class C (A, 0, z) (Fig. 2.23)

A CNN belongs to the zero-input class C (A, 0, z) if and only if all feedforward
template elements are zero, i.e., B ≡ 0.
Each cell of a zero-input CNN is described by

ẋi j = −xi j + A  Yi j + z (2.40)

Definition 10: Uncoupled (scalar) class C (A0 , B, z) (Fig. 2.24)

A CNN belongs to the uncoupled class C (A0 , B, z) if and only if ai j = 0 except i = j,
i.e., Ā ≡ 0.
Each cell of an uncoupled CNN is described by a scalar nonlinear ODE which is
not coupled to its neighbors:

ẋi j = −xi j + a00 f (xi j ) + B  Ui j + z (2.41)

29 2.2 Mathematical foundations

Fig. 2.22. Zero-feedback (feedforward) CNN ∈ C(0, B, z). (a) Signal flow structure of a
zero-feedback CNN with a 3 × 3 neighborhood. The cone symbolizes the weighted contributions of
input voltages of cells C(k, l) ∈ S1 (i, j) to the center cell C(i, j). (b) System structure of a cell
C(i, j). Arrows printed in bold denotes the input signal from the surround cells. In this case, there is
no self-feedback, and no couplings from the outputs of the surround cells.

Fig. 2.23. Zero-input (Autonomous) CNN ∈ C(0, B, z). (a) Signal flow structure of a zero-input
CNN with a 3 × 3 neighborhood. The cone symbolizes the weighted contributions of the output
voltage of cells C(k, l) ∈ S1 (i, j) to the center cell C(i, j). (b) System structure of a center cell
C(i, j). Arrow printed in bold denotes the signal fed-back from the outputs of the surround cells.
In this case, there are no input signals.
30 Notation, definitions, and mathematical foundation

Fig. 2.24. Uncoupled CNN ∈ C(0, B, z). (a) Signal flow structure of an uncoupled CNN with a
3 × 3 neighborhood. The cone symbolizes the weighted contributions of the input voltages of cells
C(k, l) ∈ S1 (i, j) to the center cell C(i, j). (b) System structure of a center cell C(i, j). Arrow
printed in bold denotes the input signals from the surround cells. In this case, the data streams
simplified into simple streams marked by thinner arrows, indicating only a “scalar” self-feedback,
but no couplings from the outputs of the surround cells.

So far, we have not mentioned the details of the physical or biological meaning of
the various terms in the CNN equations. To highlight some of these issues, next we
show a possible electronic circuit model of a cell. In Fig. 2.25, voltage-controlled cur-
rent sources are used to implement various coupling terms. These trans-conductances
can be easily constructed on CMOS integrated circuits. The details will be discussed in
Chapter 15. A very rough sketch of a typical living neuron with interacting neighbors
is shown in Fig. 2.26. A more detailed discussion can be found in Chapter 16.

2.2.6 Synaptic signal flow graph representation

Sometimes it will be more convenient to use a synaptic signal flow graph

representation3 for both the A and the B templates as shown in Figs 2.27 and 2.28,
respectively. These two flow graphs show explicitly the directions of the signal flows
from neighboring cells and their associated synaptic weights akl and bkl , respectively.
Except for the symbols {a00 , akl } for the synaptic signal flow graph A, and {b00 , bkl }
31 2.2 Mathematical foundations

synaptic current sources controlled

by the inputs of surround cells

current summing synaptic current sources controlled

b ,j
–1 u i–1 j+
1 node ij by the outputs of surround cells
,– 0 ,
1 u b –1, ,1u i–1 of cell C(ij)
i–
1,
j–
1
b –1
b0,–1 bi, j–1 b0,1ui, j+1
total
fe ,j
curr edback y i–1

1
a

, j+
ent 1, 0

–1
A Y –1 y a –1,

yi
1
j–

1,1
total i+
1,

1,

a–
fe ij
i+
u

curr edforwa
j–
b 1,1

1
b1,0ui+1, j
–1

ent rd
1,

u i+1,
b

uij a0,–1yi, j–1 a0,1 yi, j+1

j+1

a1

1
j–
1,
,1 yi
i–
y
,–
1
+1
,j
–1

a1,0 yi+1, j
+1
a

uij
bij uij xij
– zij xij +
1 xij
input voltage +
aij y ij
of cell C(ij) f (xij )
– 1 y ij
threshold current
of cell C(ij) –
internal core
of cell C(ij) state voltage
of cell C(ij)
output voltage
of cell C(ij)
Fig. 2.25. Cell realization of a standard CNN cell C(i, j). All diamond-shape symbols denote a
voltage-controlled current source which injects a current proportional to the indicated controlling
voltage u kl or ykl , weighted by bkl or akl , respectively, except for the rightmost diamond f (xi j ) in
the internal core which is a nonlinear voltage controlled current source, resulting in an output
voltage yi j = f (xi j ).
32 Notation, definitions, and mathematical foundation

sensory
neuron
C(mn)
axon ymn

cell body enclosed

by a membrane
neuron
de put

C(ij)
nd b m
in

rit
ic nu mn

axon yij

a synapse from the axon re

of a sensory (input) fe curr
ed en
neuron (say from the ba t
ck
“olfactory bulb”) to a a
dendrite of neuron C(ij) dendrite kl y
kl

neuron
C(kl)
a synapse from the axon
(output) of neuron C(kl)
axon to a dendrite of neuron
ykl C(ij)

Fig. 2.26. A caricature of a typical neuron C(i, j) receiving an input from a sensory neuron on the
left and a neighbor neuron below through respective “synapses.”

for the synaptic signal flow graph B, these two signal flow graphs are identical and
hence easy to remember. Observe that the bold heavy edges indicate signal flowing
into cell C(i, j), whereas the light edges indicate signal flowing out from cell C(i, j).
Observe that each synaptic weight occurs exactly twice in each flow graph, and that
they are always associated with a pair of edges, one bold edge and one light edge, with
arrows pointing in the same directions. For example, the coefficient a−10 is associated
with the two edges originating from the North (N) cell and terminating in the South (S)
33 2.2 Mathematical foundations

a−1,−1 a−1,0 a−1,1

(a) a0,−1 a00 a0,1
a1,−1 a1,0 a1,1

NW N NE

yi–1, j–1 yi–1, j yi–1, j+1

a1,1 a–1,0 a1,0 a–1,1

a1,–1
a–1,–1

a0,1 a 0,1

W a00 E
yi, j–1 yi, j+1
a0,–1 a0,–1

a–1,1 a1,1

a–1,–1
a–1,0
a1,0
a1,–1

yi+1, j–1 yi+1, j yi+1, j+1

SW S SE
(b)

Fig. 2.27. Feedback synaptic signal flow graph A associated with the A template.3 (a) The A
template with self-feedback synaptic weight a00 , (b) Signal flow graph A.

cell. This is because the “top” cell C(i − 1, j) is North of the center cell C(i, j), which
is itself North of the “bottom” cell C(i + 1, j). The same observation applies to the
horizontal pairs, and all diagonal pairs of similarly-directed bold-light edges. Observe
also that for each zero coefficient akl = 0, or bkl = 0, two corresponding edges will
disappear from the corresponding signal flow graph. Hence, for templates with only
a few non-zero entries, their associated synaptic signal flow graphs are particularly
simple. It is in such situations where useful insights can be obtained, specially when
two or more such synaptic signal flow graphs are interconnected to form a composite
synaptic system graph.
34 Notation, definitions, and mathematical foundation

b−1,−1 b−1,0 b−1,1

(a) b0,−1 b00 b0,1
b1,−1 b1,0 b1,1

NW N NE

ui–1, j–1 ui–1, j ui–1, j+1

b1,1 b–1,0 b1,0 b–1,1

b1,–1
b–1,–1

b0,1 b 0,1

W b00 E
ui, j–1 ui, j+1
b0,–1 b0,–1

b–1,1 b1,1

b–1,–1
b–1,0
b1,0
b1,–1

ui+1, j–1 ui+1, j ui+1, j+1

SW S SE
(b)
Fig. 2.28. Input synaptic signal flow graph B associated with the B template. (a) B template with
self-input synaptic weight b00 . (b) Signal flow graph B.