2806 Neural Computation

Committee Machines
Lecture 7
2005 Ari Visa

Agenda
Some historical notes
Some theory
Committee Machines
Conclusions

Some Historical Notes

Boosting by filtering: Schapire 1990
AdaBoost:(Schapire, Freund: A decision-theoretic
generalization of on-line learning and an
application to boosting, 1995)
Bagging: (Leo Breiman: Bagging Predictors 1996)
General arching: Leo Breiman: Arcing Classifiers,
1998)
Kittler, Hatef, Duin, On Combining Classifiers, 1998

Some Theory

The principle of divide and conquer: a complex computational task is solved

by dividing it into a number of computationally simple tasks and then
combining the solutions to the tasks.
Committee machine = In supervised learning, computational simplicity is
achieved by distributing the learning task among a number of experts, which
in turn divides the input space into a set of subspaces. The combination of
experts is said to constitute a committee machine.
Modularity (Osherson & al 1990): A neural network is said to be modular if
the computation performed by the network can be decomposed into two or
more modules that operate on distinct inputs without communicating with
each other. The outputs of the modules are mediated by an integrating unit that
is not permitted to feed information back to the modules. In particular , the
integrating unit both decides how the outputs of the modules should be
combined to form the final output of the system, and decides which modules
should learn which training pattern.

Some Theory

Committee machines are universal approximators

a) Static structures: the responses of several experts are combined by
means of a mechanism that does not involve the input signal
Ensemble averaging: outputs are linearly combined
Boosting: a weak learning algorithm is converted into one that
achieves arbitarily high accuracy
b) Dynamic structures: the input signal is directly involved in
actuating the mechanism that integrates the outputs of the individual
experts into an overall output.
Mixture of experts: the responses of the experts are nonlinearly
combined by means of a single gating network.
Hierarchical mixture of experts: the responses of the experts are
nonlinearly combined by means of several gating network arranged
in a hierarchical fashion.

Some Theory
Ensamble Averaging: A
number of differently
trained neural networks,
which share a common
input and whose
individual outputs are
somehow combines to
produce an overall output.

Note, all the experts are

trained on the same data
set, they may differ from
each other in the choice of
initial conditions used in
network training.
The output is sum of the
individual outputs of the
experts and then
computing the probability
of correct classification.

Some Theory
The bias of the ensemble-averaged function
FI(x), pertaining to the commitee machine is
exactly the same as that of the function F(x)
pertaining to a single neural network.
The variance of the ensemble-averaged
function FI(x) is less than that of the
function F(x).

Some Theory

Boosting: the experts

are trained on data sets
with entirely different
distributions.
Boosting by filtering
Boosting by
subsampling
Boosting by
reweighting

Some Theory

1.
2.
-

3.
-

The original idea of boosting is rooted in a distribution-free or probably approximately correct

model of learning (PAC).
Boosting by filtering: the committee machine consists of three experts (requires a large data set).
1. The first expert is trained on a set consisting of N 1 examples.
2. The trained first expert is used to filter another set of examples by proceeding in the following
manner:
Flip a fair coin( ~ simulates a random guess)
If the result is heads, pass a new pattern through the first expert and discard correctly classified
patterns until a pattern is missclassified. That missclassified pattern is added to the training set for
the second expert.
If the result is tails, pass new patterns through the first expert and discard incorrectly classified
patterns until apattern is classified correctly. That correctly classified pattern is added to the
training set for the second expert.
Continue this process until a total of N 1 examples has been filtered by the first expert. This set of
filtered examples constitutes the training set for the second expert.
3. Once the second expert has been trained in the usual way, a third training set is formed for the
third expert by proceeding in the following manner:
Pass a new pattern through both the first and the second experts. If the two experts agree in their
decisions, discard that pattern. If they disagree, the pattern is added to the training set for the third
expert.
Continue with this process until total N 1 examples has been filtered jointly by the first and second
experts. This set of jointly filtered examples constitutes the training set for the third expert.

Some Theory

Classification: If the
first and the second
experts in the
committee agree in
their respective
decisions, that class
label is used.
Otherwise, the class
label discovered by the
third expert is used.

The three experts have

an error rate of < 1/2
g() = 32 -23

Some Theory

Some Theory

AdaBoost (boosting by
resampling -> batch
learning)
AdaBoost adjusts
adaptively to the error of
the weak hypothesis
returned by the weak
learning model.
When the number of
possible classes (labels) is
M>2, the boosting
problem becomes more
intricate.

Some Theory

Error performance due

to AdaBoost is
peculiar.
The shape of the error
rate debends on the
definition of
confidence.

Some Theory

Dynamic is used here in the sense that integration of knowledge

acquired by the experts is accomplished under the action of the input
signal.
Probabilistic generative model
1. An input vector x is picked at random from some prior distribution.
2. A perticular rule, say kth rule, is selected in accordance with the
conditional probability P(k|x,a(0)), given x and some parameter vector
a(0).
3. for rule k, k<01,2,...,K, the model response d is linear in x, with an
additive error k modeled as a Gaussian distributed random variable
with zero mean and unit variance:
E[k] = 0 for all k and var[k] = 1 for all k.
P(D=d|x, (0) ) = Kk=1 P(D=d|x,wk(0) ) P(k|x,a(0)),

Some Theory
Mixture of Experts Model:
yk = wkTx k=1,2,...,K
The gating network consists
of a single layer of K
neurons, with each neuron
assigned to a specific
expert.

Some Theory
The neurons of the gating
network are nonlinear.
gk = exp(uk)/Kj=1exp(uj)
k = 1,2,...,K and uk = akTx ->
Softmax
The gating network is a
classifier that maps the
input x into multinomial
probabilities -> The
different experts will be
able to match the desired
response.

Some Theory
fD(d|x,) = Kk=1 gk fD(d|x,k,)
= 1/2 Kk=1 gk exp(-1/2(d-yk)2)
associative Gaussian mixture model
Given the training sample {(xi ,di)}Ni=1, the problem
is to learn the conditional means k = yk and the
mixing parameters gk,k = 1,2,...,K in an optimum
manner, so that fD(d|x,) provides a good estimate
of the underlying probability density function of
the environment responsible for generating the
training data.

Some Theory
Hierarchical Mixture of
Experts Model
HME is a natural extension of the
ME model.
The HME model differs from the
ME model in that the input
space is divided into a nested
set of subspaces, with the
information combined and
redistributed among the experts
under the control of several
gating networks arranged in a
hierarchical manner.

Some Theory

The formulation of the HME model can be viewed

in two ways.
1) The HME model is a product of the divide and
conquer strategy
2) The HME model is a soft-decision tree
Standard decision trees suffer from a greediness
problem, once a decision is made in such a tree, it
is frozen and never changes thereafter.

Some Theory
Classification and decision tree
(CART Breiman 1984)
1) selection of splits: let a node t
denotea subset of the current tree
T. Let d-(t) denote the average of
di for all cases (x,di) falling into t,
that is, d-(t) = 1/N(t) xjt di where
the sum is over all di such that xit
and N(t) is the total number of
cases t.
2) Define E(t) = 1/N xjt (di d-(t))2
and E(T) = tT E(t)
The best split s* is then taken to be
the particular split for which we
have E(s*,t) = max sS E(t,s)

Some Theory
Determination of a terminal node: A node t is declared a
terminal node if this condition is satisfied:
max sS E(s,t) < th, th is a prescribed threshold.
Least-square estimation of a terminal nodes parameters:
Let t denote a terminal node in the final binary tree T, and let X(t)
denote the matrix composed of xi t. Let d(t) denote the
corresponding vector composed of all the d i in t. Define w(t) =
X+(t)d(t) where X+(t) is the pseudoinverse of matrix X(t).
Using the weights calculated above , the split selection problem is
solved by looking for the least sum of squared residuals with
respect to the regression surface.

Some Theory
g = 1 / {1+exp(-(aTx +b))}
Using CART to initialize the HME model:
1)
1) Apply CART to the training data
2)
2) Set the synaptic weight vectors of the experts in the HME model
equal to the least-squares estimates of the parameter vectors at the
corresponding terminal nodes of the binary tree resulting from the
application of CART.
3)
3) For the gating networks:
4)
a) set the synaptic weight vectors to point in direction that are
orthogonal to the corresponding splits in the binary tree obtained
from CART and
5)
b) set the lengths of the synaptic weight vectors equal to small
random vectors.

Some Theory
A posteriori probabilities at the nonterminal nodes of the
tree:
hk= gk2j=1 gj|k exp(-1/2(d-yjk)2) / {2k=1 gk 2j=1 gj|k exp(-1/2(dyjk)2)}
hj|k= gj|k exp(-1/2(d-yjk)2) / { 2j=1 gj|k exp(-1/2(d-yjk)2)}
the joint a posteriori probability that expert(j,k) produces an
output yjk :
hjk = hk hj|k
= gk gj|k exp(-1/2(d-yjk)2) / {2k=1 gk 2j=1 gj|k exp(-1/2(d-yjk)2)}

0 hjk 1 for all (j,k); 2j=1 2k=1 hjk = 1

Some Theory

The parameter estimation for the HME model:

Maximum likelihood estimation: y = 2k=1 gk 2j=1
gj|k yjk

fD(d|x,) = 1/2 2k=1 gk 2j=1 gj|k exp(-1/2(d-yk)2)

likelihood function l()= fD(d|x,)
log-likelihood function L()= log[fD(d|x,)]
L() / = 0 the maximum likelihood
estimate

Some Theory

Learning strategies for the

HME model
1. Stochastic gradient
approach
This approach yields an on-line
algorithm for the maximization
of L().
L / wjk = hj|k(n) hk (n)(d(n)
yj|k(n))x(n)
L / ak = (hk(n) gk(n))x(n)
L / ajk = hk(n)(hj|k(n) gj|
k(n))x(n)

Some Theory

2. Expectation-maximization approach
Expectation step (E-step), which uses the observed
data set of an incomplete data problem and the
current value of the parameter vector to
manufacture data so as to postulate an augmented
or so-called complete data set.
Maximization step (M-step), which consists of
deriving a new estimate of the parameter vector by
maximizing the log-likelihood function of the
complete data manufactured in the E-step.

Some Theory

The EM algorithm is directed at finding a value of that

maximizes the incomplete-data log-likelihood function
L()= log[fD(d|)]
This problem is solved indirectly by working iteratively
with the complete-data log-likelihood function Lc()=
logfc(r|), which is a random variable, because the
missing data vector z is unknown.
E-step: Q(,^(n)) = E[Lc()]
M-step maximize Q(,^(n)) with respect to n
^(n+1) = arg max Q(,^(n)); continue until the
difference between L(^(n+1)) and L(^(n)) drops to
some arbitrary small value

Some Theory
fD(di|xi,) = 1/2 2k=1 g(i)k 2j=1 g(i)j|k exp(-1/2(di-y(i)k)2)
L()= log[N i=1 fD(di|xi,)]
Lc()= log[N i=1 fc(di,z(i)jk|xi,)]
Q(,^(n)) = E[Lc()]
= Ni=1 2j=1 2k=1 h(i)jk (log g(i)k + log g(i)j|k -1/2(d(i) y(i)jk)2
wjk(n+1) = arg minwjk Ni=1 h(i)jk (di y(i)jk)2
aj(n+1) = arg maxaj Ni=1 2k=1 h(i)k log g(i)k
ajk(n+1) = arg maxajk Ni=1 2l=1 h(i)l 2m=1 h(i)m|l log g(i)m|l

Summary
Ensemble averaging improves error
performance by combining two effects:
a) overfitting the individual experts
b) using different initial conditions in the
training of the individual experts
Boosting improves error performance by
filtering and resampling .

Summary

Simple models provide insight into the problem but

lack accuracy
Complex models provide accurate results but lack
insight.
The architecture of HME is similar to that of
CART, but differs from it in soft partitioning the
input space.
The HME uses a nested form of nonlinearity
similar to MLP, not for the purpose of input-output
mapping, but rather for partitioning the input space.