Hopfield Network Example

We have a 5 node Hopfield network and we want it to recognize the pattern (0 1 1 0 1). Since there
are 5 nodes, we need a matrix of 5 x 5 weights, where the weights from a node back to itsef
are 0. !he weight matrix wi ook ike this"
0
W
12
W
13
W
14
W
15
W
21
0
W
23
W
24
W
25
W
31
W
32
0
W
34
W
35
W
41
W
42
W
43
0
W
45
W
51
W
52
W
53
W
54
0
Where W
ij
= W
ji

Since the weights are s#mmetric, we on# have to cac$ate the $pper diagona of weights, and then we
can cop# each weight to its inverse weight. %n this case, & is the vector (0 1 1 0 1), so &
1
' 0, &
(
' 1,
&
)
' 1, &
*
' 0, and &
5
' 1. !h$s the comp$tation of the weights is as foows"
W
12
= (2V
1
1!(2V
2
1! = (0 1!(2 1! = (1!(1! = 1
W
13
= (2V
1
1!(2V
3
1! = (0 1!(2 1! = (1!(1! = 1
W
14
= (2V
1
1!(2V
4
1! = (0 1!(0 1! = (1!(1! = 1
W
15
= (2V
1
1!(2V
5
1! = (0 1!(2 1! = (1!(1! = 1
W
23
= (2V
2
1!(2V
3
1! = (2 1!(2 1! = (1!(1! = 1
W
24
= (2V
2
1!(2V
4
1! = (2 1!(0 1! = (1!(1! = 1
W
25
= (2V
2
1!(2V
5
1! = (2 1!(2 1! = (1!(1! = 1
W
34
= (2V
3
1!(2V
4
1! = (2 1!(0 1! = (1!(1! = 1
W
35
= (2V
3
1!(2V
5
1! = (2 1!(2 1! = (1!(1! = 1
W
45
= (2V
4
1!(2V
5
1! = (0 1!(2 1! = (1!(1! = 1
1
So now o$r weight matrix ooks ike this"
0 1 1 1 1
W
21
0 1 1 1
W
31
W
32
0 1 1
W
41
W
42
W
43
0 1
W
51
W
52
W
53
W
54
0
+# refecting abo$t the diagona, we get the f$ weight matrix"
0 1 1 1 1
1 0 1 1 1
1 1 0 1 1
1 1 1 0 1
1 1 1 1 0
,or competeness- sake, #o$- remember that the origina form$a was set $p to aow #o$ to have n
patterns. So et-s consider the case where we want o$r 5 node .opfied net to store both the
pattern &
1
' (0 1 1 0 1) and another pattern &
(
' (1 0 1 0 1). /ne wa# #o$ co$d go abo$t
cac$ating the weights is on a weight b# weight basis. ,or exampe, W
1(
co$d be cac$ated
as"
W
12
=

(2V
"
1
1!(2V
"
2
1!
= (2V
1
1
1!(2V
1
2
1! # (2V
2
1
1!(2V
2
2
1!
= (2$0 1!(2$1 1! # (2$1 1!(2$0 1!
(
= (0 1!(2 1! # (2 1!(0 1!
= (1!(1! # (1!(1!
= 1 # 1
= 2
0o$ co$d go thro$gh each weight ike this and cac$ate the new weight matrix. !his is ike# the wa#
#o$ wo$d do it b# comp$ter, b$t % find if %-m cac$ating them b# hand, it-s easier to foc$s on
one pattern at a time and then s$m the res$ting matrices. !o do this, #o$ wo$d cac$ate the
matrix for the first pattern (which we did above), then cac$ate the va$e for the second matrix
and fina# add the two matrices together. .ere-s the weight matrix for the pattern (1 0 1 0 1)"
0 1 1 1 1
1 0 1 1 1
1 1 0 1 1
1 1 1 0 1
1 1 1 1 0
1ow add this to the previo$s weight matrix and we get"
0 2 0 0 0
2 0 0 0 0
0 0 0 2 2
0 0 2 0 2
0 0 2 2 0
)
How to %pdate a node in a Hopfield network
So now we have a weight matrix for a 5 node .opfied network that-s meant to recognize the patterns
(0 1 1 0 1) and (1 0 1 0 1). /ne thing #o$ might notice is that these patterns on# differ b# (
bits. 2s #o$ might imagine, patterns this cose together might be diffic$t to te apart. +#
anaog#, #o$ might have tro$be discriminating a ower case 3c3 from 3e3 or an $pper case 3/3
from 343 if the# were manged bad# eno$gh. 5et-s start from the pattern (1 1 1 1 1), which
on# differs from each of these patterns b# ( bits, and see what happens.
6pdating a node in a .opfied network is ver# m$ch ike $pdating a perceptron. ,irst #o$ do a
weighted s$m of the inp$ts from the other nodes, then if that va$e is greater than or e7$a to 0,
#o$ o$tp$t 1. /therwise, #o$ o$tp$t 0. %n form$a form"
V
i
in =

W
ji
V
j

V
i
& 1 if V
i
in &= 0
el"e V
i
& 0
So ass$ming we have the fina weight matrix from the previo$s section, and we start from the state (1
1 1 1 1), for the )rd node, we have"
V
3
in =

W
j3
V
j

=
W
13
V
1
# W
23
V
2
# W
43
V
4
# W
53
V
5

= 0$1 # 0$1 # 2$1 # 2$1
= 0
"in'e 0 &= 0(
V
3
= 1
*
%n this case, the va$e of &
)
doesn-t change. %t-s worth noticing that since the weight from node ) to
itsef is 0, we co$d have 8$st cac$ated the dot prod$ct of the )rd co$mn o$t of the weight
matrix and the c$rrent state to cac$ate the weighted s$m"
V
3
in = (0 0 0 2 2! $ (1 1 1 1 1! = 2 # 2 = 0
)e*%en'in+ of node %pdate" in a Hopfield network
0o$ might have noticed b# now that se7$encing the $pdates of the nodes in a .opfied network is
somewhat trick#. .ow can #o$ $pdate a ne$ron if the va$es of its inp$ts are changing9 We,
there are two approaches. !he first is s#nchono$s $pdating, which means a the nodes get
$pdated at the same time, based on the existing state (i.e. not on the va$es the nodes are
changing to). !o $pdate the nodes in this method, #o$ can 8$st m$tip# the weight matrix b#
the vector of the c$rrent state.
!his isn-t ver# reaistic in a ne$ra sense, as ne$rons don-t a $pdate at the same rate. !he# have
var#ing propagation dea#s, var#ing firing times, etc., so a more reaistic ass$mption wo$d be
to $pdate them in random order. !his was the method described b# .opfied, in fact. 0o$
random# seect a ne$ron, and $pdate it. !hen #o$ random# seect another ne$ron and $pdate
it. 0o$ keep doing this $nti the s#stem is in a stabe state (which we- tak abo$t ater).
%n practice, peope code .opfied nets in a semi:random order. !he# $pdate a of the nodes in one
step, b$t within that step the# are $pdated in random order. So it might go ), (, 1, 5, *, (, ), 1,
5, *, etc. !his is 8$st to avoid a bad pse$do:random generator from favoring one of the nodes,
which co$d happen if it was p$re# random" ), (, 1, (, (, (, 5, 1, (, (, *, (, 1, etc.
5
How to tell when ,o% 'an "top %pdatin+ the network
;
!he main reason to want to c#ce thro$gh a the nodes each step is that it-s the on# wa# #o$ can te
when to stop. +asica#, if #o$ go thro$gh a the nodes and none of them changes, #o$ can
stop. %f #o$-re $pdating them in a fixed se7$ence (e.g. 1, (, ), *, 5, 1, (, ), *, 5, 1, (, ), etc.),
this means that if #o$ go thro$gh 5 consec$tive ne$rons witho$t changing an# of their va$es,
then #o$-re at an attractor so #o$ can stop.
<
-ini"hin+ %p the example
=
1ow et-s finish the exampe % started. %n other words, given the weight matrix for a 5 node network
with (0 1 1 0 1) and (1 0 1 0 1) as attractors, start at the state (1 1 1 1 1) and see where it goes.
!o keep it simpe, %-m going to $pdate the nodes in the fixed order ), 1, 5, (, *, ), 1, 5, (, *, etc.
% aread# did the first $pdate of node ), and it didn-t change, so contin$ing"
%pdate node 3 did it( no 'han+e
%pdate node 1
V
1
in = (0 2 0 0 0! . (1 1 1 1 1! = 2
"in'e 2 / 0( V
1
= 0 (it changed)
%pdate node 5
V
5
in = (0 0 2 2 0! . (0 1 1 1 1! = 0
"in'e 0 &= 0( V
5
= 1 (it didn't change)
%pdate node 2
V
2
in = (2 0 0 0 0! . (0 1 1 1 1! = 0
"in'e 0 &= 0( V
2
= 1 (it didn't change)
%pdate node 4
V
4
in = (0 0 2 0 2! . (0 1 1 1 1! = 4
"in'e 4 / 0( V
4
= 0 (it changed)
%pdate node 3
V
3
in = (0 0 0 2 2! . (0 1 1 0 1! = 2
"in'e 2 &= 0( V
3
= 1 (it didn't change)
%pdate node 1
V
1
in = (0 2 0 0 0! . (0 1 1 0 1! = 2
"in'e 2 / 0( V
1
= 0 (it didn't change)
%pdate node 5
V
5
in = (0 0 2 2 0! . (0 1 1 0 1! = 2
"in'e 2 &= 0( V
5
= 1 (it didn't change)
%pdate node 2
V
2
in = (2 0 0 0 0! . (0 1 1 0 1! = 0
"in'e 0 &= 0( V
2
= 1 (it didn't change)
>
10