Unit-2

SUPERVISED LEARNING
NETWORK
BY
Dr Deepa S
.
• SYLLABUS
• Shallow neural networks- Perceptron Networks-Theory-Perceptron
Learning Rule Architecture-Flowchart for training Process-Perceptron
Training Algorithm for Single and Multiple Output Classes.
• Back Propagation Network- Theory-Architecture-Flowchart for
training process. Training Algorithm-Learning Factors for
Back-Propagation Network.
• Radial Basis Function Network RBFN: Theory, Architecture, Flowchart
and Algorithm.
Shallow neural networks
• Shallow neural networks, also known as single-layer neural networks or
perceptrons, are a type of neural network architecture that consists of a
single layer of artificial neurons.
• Unlike deep neural networks, which have multiple hidden layers between
the input and output layers, shallow neural networks have just one hidden
layer (if any) and directly connect the input to the output layer.
• Shallow neural networks were among the earliest neural network models
developed, with the perceptron, proposed by Frank Rosenblatt in 1957,
being one of the first artificial neural networks. However, their limited
depth makes them less capable of learning complex representations
compared to deep neural networks.
• Perceptionnetworks : Perceptionnetworks come under single-layer feed-forward
networks and are also called simple perceptrons.
• Original Perception network
.
• A Perception network with its three units is shown in Figure below:
• 1. Sensory Unit:
• The sensory unit (also known as the input unit) is responsible for receiving the input data and
passing it into the network. In the context of the perceptron model, the input data is typically
represented as a feature vector, where each element of the vector corresponds to a specific input
feature.
• For example, in a simple binary classification problem, the sensory unit may receive a
two-dimensional input vector (x1, x2) representing two features of the input data.
• 2. Response Unit:
• The response unit (also known as the hidden unit) is the processing unit within the perceptron
network. It takes the input from the sensory unit and applies a weighted sum of the inputs,
followed by an activation function.
• Mathematically, the response unit computes the net input (z) as the weighted sum of the input
features (x) and their associated weights (w), along with an optional bias term (b):
• z = (w1 * x1) + (w2 * x2) + ... + (wn * xn) + b
• After calculating the net input, the response unit applies an activation function (often a nonlinear
function) to the net input to produce an output signal.
Original Perception network
• 3. Output Unit:
• The output unit receives the output signal from the response unit and generates the final
output of the perceptron network. In binary classification tasks, the output unit typically
applies a threshold function to convert the continuous output from the response unit into
a binary output (0 or 1).
• For example, a common threshold function used in binary classification is the step
function, which can be defined as follows:
• output = 1, if z >= 0
• output = 0, if z < 0
• The output of the output unit represents the prediction or classification made by the
perceptron network based on the input data and the learned weights.
• It's important to note that the terms "sensory unit," "response unit," and "output unit"
are more commonly used in the historical context of the original perceptron model and
are less frequently used in modern deep learning terminologies.
Original Perception network
• Learning rule
• In case of the perceptron learning rule, the learning signal is the difference between the desired
.
and actual response of a neuron.
• The perceptron learning rule is explained as follows:
• Consider a finite "n" number of input training vectors, with their associated target values x(n)and
t{n), where "n" ranges from 1 to N.
• The target is either+ 1 or -1.
• The output ''y" is obtained on the basis of the net input calculated and activation function being
applied over the net input.

• The weight updation in case of perceptron learning is as shown.

Architecture
• A simple perceptron network
architecture is shown in Figure
• In Figure, there are n input
neurons, 1 output neuron and a
bias.
• The input-layer and output layer
neurons are connected through a
directed communication link, which
is associated with weights.
• The goal of the perceptron net is to
classify the input pattern as a
member or not a member to a
particular class.
Training algorithm/process
Flowchart for Training Process

•.
Flowchart for Training Process
• The flowchart depicted here presents the flow of the training
process.
• As depicted in the flowchart, first the basic initialization
required for the training process is performed.
• The entire loop of the training process continues until the training
input pair is presented to the network.
The training (weight updation) is done on the basis of the comparison
between the calculated and desired output.
The loop is terminated if there is no change in weight
And example
Input Net Out Target Weights
(1,X1,X2) (B,W1,W2)
Epic 1 ( 0 , 0 , 0)
(1 , 1 , 1 ) 0 0 1 (1 , 1 , 1)
(1 , 1 , -1) 1 1 -1 (0 , 2 , 0 )
(1 , -1 , 1) 2 1 -1 (-1 , 1 , 1)
(1 , -1 , -1 ) -3 -1 -1 (-1 , 1 , 1)
Epoc 2
( 1 , 1 , 1) 1 1 1 ( -1 , 1 , 1)
( 1 , 1 , -1) -1 -1 -1 ( -1 , 1, 1)
( 1 , -1 , 1 ) -1 -1 -1 ( -1 , 1 , 1)
( 1 , -1 , -1) -1 -1 -1 (-1 , 1 , 1)
• What is the learning algorithm?
• It is an adaptive method that self-arranges a network of computing
units to implement the required behavior. Some of these algorithms
do this by bringing in front of the network a few examples of the
required input-output mapping.
• The perceptron algorithm, in its most basic form, finds its use in the
binary classification of data. Perceptron takes its name from the basic
unit of a neuron, which also goes by the same name.
Perceptron Training Algorithm for Single
Output Classes
• The perceptron algorithm can be used for either binary or bipolar
input vectors, having bipolar targets, threshold being fixed and
variable bias.
• In the algorithm below, initially the inputs are assigned. Then the net
input is calculated. The output of the network is obtained by applying
the activation function over the calculated net input.
• On performing comparison over the calculated and the desired
output, the weight updation process is carried out. The entire
network is trained based on the mentioned stopping criterion.
•:
Perceptron Training Algorithm for Single
Output Classes
• The algorithm of a perceptron network is as follows
• Step 0:Initialize the weights and the bias. Also initialize the learning
• rate α(O <α ≤1). For simplicity α is set to 1.
• Step 1:Perform Steps 2-6 until the final stopping condition is false.
• Step 2: Perform Steps 3-5 for each training pair indicated by s:t.
• Step 3: The input layer containing input units is applied with identity
• activation functions:
• Xi = Si
• Step 4: Calculate the output of the network. To do so, first obtain the net
• input:

.
• Where "n" is the number of input neurons in the input layer. Then apply activations over
the netinput calculated to obtain the output:

• Step 5:Weight and bias adjustment: Compare the value of the actual (calculated) output
and desired(target) output.

• Step 6: Train the network until there is no weight change. This is the stopping condition
for the network. If this condition is not met, then start again from Step 2.
Single output class
Perceptron Training Algorithm for Multiple
Output Classes
• Step 0:Initialize the weights, biases and learning rate suitably.
• Step 1: Check for stopping condition; if it is false, perform Steps 2-6.
• Step 2: Perform Steps 3--5 for each bipolar or binary training vector
pair s:t.
• Step 3: Set activation (identity) of each input unit i= 1 ton:
xi=si
• Step 4:Calculate output response of each output unit j=1 to m; First
the net input is calculated as:
• Then activations are applied over the net input to calculate the output
response:
.
• Step 5: Make adjustment in weights and bias for j =1 to m and i= 1to n

• Step 6: Test for the stopping condition, i.e., if there is no change in

• weights then stop the training process, else start again from Step 2.
• Perceptron Network Testing Algorithm
• The testing algorithm is asfollows:
• Step 0: The initial weights to be used here are taken from the training
• algorithms.
• Step 1: For each input vector X to be classified, perform Steps 2-3.
• Step 2: Set activations of the input unit.
• Step 3: Obtain the response of output unit.
Back propagation Network
• Theory
• The back propagation learning algorithm is one of the most important developments in neural
networks. The networks associated with back-propagation learning algorithm are called back
propagation networks(BPNs).
• For a given set of training input-output pair, this algorithm provides a procedure for changing the
weights in a BPN to classify the given input patterns correctly.
• The basic concept for this weight update algorithm is simply the gradient-descent method. This is
a method where the error is propagated back to the hidden unit.
• The aim of the neural network is to train the net to achieve a balance between the net's ability to
respond and its ability to give reasonable responses to the input that is similar but not identical to
the one that is used in training.
• The back-propagation algorithm is different from other networks in respect to the process by
which weights are calculated during the learning period of the network.
• The general difficulty with the multilayer perceptrons is calculating
the weights of the hidden layers in an efficient way that would result in a
very smallor zero output error.
• When the hidden layers are increased the network training becomes more
complex. To update weights, the error must be calculated.
• The error, which is the difference between the actual (calculated)and the
• desired (target) output, is easily measured at the output layer.
• The training of the BPN is done in three stages:
•  the feed-forward of the input training pattern
•  thecalculation and back-propagation of the error
•  updation of weights.
Architecture
• A back-propagation neural network is a
multilayer, feed forward neural network
consisting of an input layer,a hidden layer and
an output layer.
• The neurons present in the hidden and output
layers have biases, whichare the connections
from the units whose activation is always 1.
• The bias terms also acts as weights.
• The Figure below shows the architecture of a
BPN, depicting only the direction of
information flow for the feed-forward phase.
During the back propagation phase of learning
signals are sent in the reverse direction.
• The inputs sent to the BPN and the output
obtained from the net could be either binary
(0, 1) orbipolar ( -1, + 1). The activation
function could be any function which
increases monotonically and is Also
• The back-propagation algorithm is different from other networks in
• respect to the process by which the weights are calculated during the
• learning period of the network.
•  The training of the BPN is done in three stages – the feed-forward of the
• input training pattern, the calculation and back-propagation of the error,
• and updation of weights.
• The testing of the BPN involves the computation  of feed-forward phase
• only.
•  There can be more than one hidden layer (more beneficial ) but one
• hidden layer is sufficient.
•  Even though the training is very slow, once the network is trained it can
• produce its outputs very rapidly
Backpropagation Algorithm:
• Step 1: Inputs X, arrive through the pre connected path.
• Step 2: The input is modeled using true weights W. Weights are usually chosen
randomly.
• Step 3: Calculate the output of each neuron from the input layer to the hidden
layer to the output layer.
• Step 4: Calculate the error in the outputs
• Backpropagation Error= Actual Output – Desired Output
• Step 5: From the output layer, go back to the hidden layer to adjust the weights to
reduce the error.
• Step 6: Repeat the process until the desired output is achieved.

•
BPN Flowchart for Training Process
Training algorithm

• Parameters :
• x = inputs training vector x=(x1,x2,…………xn).
• t = target vector t=(t1,t2……………tn).
• δk = error at output unit.
• δj = error at hidden layer.
• α = learning rate.
• V0j = bias of hidden unit j.
• Wok=bias of output unit k.
• Zj=hidden unit j. The net input to zj is
• Zinj=Voj+∑xiVij, Zj=f(Zinj)
• Yk is output unit k The netinput to Yk is
• Yink =Wok+∑ZjWjk, Yk=f(Yink)
BPN
• Training Algorithm :
• Step 1: Initialize weight to small random values.
• Step 2: While the stepsstopping condition is to be false do step 3 to
10.
• Feed forward phase(Phase1)
• Step 3: For each training pair do step 4 to 9 (Feed-Forward).
• Step 4: Each input unit receives the signal unit and transmits the
signal xi signal to all the units.
• Step 5 : Each hidden unit Zj (z=1 to a) sums its weighted input signal to
calculate its net input
• zinj = v0j + Σxivij ( i=1 to n)
• Applying activation function zj = f(zinj) and sends this signals to all
units in the layer about i.e output units
• For each output l=unit yk = (k=1 to m) sums its weighted input signals.
• yink = w0k + Σ ziwjk (j=1 to a)
• and applies its activation function to calculate the output signals.
• yk = f(yink)
.
• Back propagation Error :
• Step 6: Each output unit yk (k=1 to n) receives a target pattern
corresponding to an input pattern then error is calculated as:
• δk = ( tk – yk ) f’(yink )
• Step 7: Each hidden unit Zj (j=1 to a) sums its input from all units in
the layer above
• δinj = Σ δj wjk
• The error information term is calculated as :
• δj = δinj f’(Zinj )
• Updation of weight and bias :
• Step 8: Each output unit yk (k=1 to m) updates its bias and weight (j=1 to a). The
. weight correction term is given by :
• Δ wjk = α δk zj
• and the bias correction term is given by Δwk = α δk.
• therefore wjk(new) = wjk(old) + Δ wjk
• w0k(new) = wok(old) + Δ wok
• for each hidden unit zj (j=1 to a) update its bias and weights (i=0 to n) the
weight connection term
• Δ vij = α δj xi
• and the bias connection on term
• Δ v0j = α δ
j
• Therefore vij(new) = vij(old) + Δvij
• v0j(new) = v0j(old) + Δv0j
• Step 9: Test the stopping condition. The stopping condition can be the
minimization of error, number of epochs.
Numerical example-BPN
Backpropagation vs. Gradient Descent
Learning Factors
• feedforward Process: Backpropagation relies on the feedforward process, where
input data is passed through the neural network to generate predictions. The
feedforward process establishes the foundation for the subsequent backward pass
during which gradients are calculated.
• Loss Function: The choice of an appropriate loss function is crucial in
backpropagation. The loss function measures the difference between the
predicted output and the actual target output. The network's objective is to
minimize this loss during training.
• Backward Pass (Backpropagation): This step involves calculating the gradients of
the loss function with respect to each network parameter (weights and biases).
The gradients indicate the direction and magnitude of the adjustments required
for the network's parameters to minimize the loss.
• Gradient Descent: Backpropagation utilizes gradient descent (or its variants) to
update the network's parameters based on the calculated gradients. Gradient
descent determines the step size and direction in the parameter space to reach
the optimal set of weights and biases that minimize the loss function.
• Learning Rate: The learning rate is a hyperparameter that plays a significant role in the
gradient descent process. It determines the step size for parameter updates. A higher
learning rate can result in faster convergence but may risk overshooting the optimal
values, while a lower learning rate can slow down convergence.
• Iterations/Epochs: Backpropagation involves iterating over the training dataset for
multiple epochs. Each epoch corresponds to one complete pass through the entire
dataset. The number of iterations and epochs influence how well the network learns from
the data.
• Activation Functions: Activation functions introduce non-linearity to the neural network,
enabling it to learn complex relationships in the data. The choice of activation functions
can impact the network's ability to model the data effectively.
• Network Architecture: The architecture of the neural network, including the number of
layers, the number of neurons in each layer, and the connections between neurons, is a
crucial factor in determining the network's representational capacity and its ability to
learn complex patterns.
• Mini-batch Size: Instead of updating the weights after processing each individual data
point, backpropagation often uses mini-batches of data for more efficient computation of
gradients. The mini-batch size can affect the stability of training and the utilization of
computational resources.
• Regularization: Regularization techniques, such as L1 or L2 regularization, can be applied
to prevent overfitting during training. Regularization adds penalty terms to the loss
function, discouraging overly complex models with excessively large weights.
• Weight Initialization: Proper initialization of the network's weights is essential to facilitate
convergence during training. Poor weight initialization can lead to vanishing or exploding
gradients, hampering the learning process.
• Momentum and Adaptive Learning Rates: Techniques like momentum and adaptive
learning rates (e.g., RMSprop, Adam) can be employed to enhance gradient descent by
incorporating historical gradient information. These techniques help improve convergence
and overcome certain challenges in traditional gradient descent.
What Are Radial Basis Functions?
• What Are Radial Basis Functions?
• In simpler terms, Radial Basis Function
(RBF) is a function whose output changes
based on how far the input is from the
center point. RBFs are symmetric around
their center, meaning the output remains
the same if the distance from the center is
the same, regardless of the direction.
Activation functions
Radial Basis Function Network (RBFN
• A Radial Basis Function Network (RBFN) is
a type of feedforward neural network that
uses RBFs in its hidden layer. The network
consists of three layers: the input layer,
the hidden/kernel layer, and the output
layer.
• The input layer receives an n-dimensional
input vector, and each node in the hidden
layer represents RBF neurons that are
centered at a specific point in the input
space. The output layer combines the
outputs of the hidden layer nodes through
a linear combination of weights to
generate the final output values.
• Regularization theory-
• Regularization technique is a way of controlling the smoothness properties of a
mapping function. It involves adding to the error function an extra term which is
designed to penalize mappings which are not smooth. Instead of restricting the
number of hidden units, an alternative approach for preventing over fitting in RBF
networks comes from the regularization theory.
• Regularization network-
• RBF network can be seen as special case of regularization network. RBF network
have sound theoretical foundation in regularization theory. RBF network fit
naturally into the framework of the regularization of interpolation/approximation
task. For these problems, regularization means the smoothing of the
interpolation/approximation curve, surface. This approach to RBF network, is also
known as regularization network.
Radial Basis Function Network(RBFN)
• Architecture of RBF
• RBF network is an artificial neural network with an input layer, a
hidden layer, and an output layer. The Hidden layer of RBF consists of
hidden neurons, and activation function of these neurons is a
Gaussian function. Hidden layer generates a signal corresponding to
an input vector in the input layer, and corresponding to this signal,
network generates a response.
RBNN ALGORITHM