EXPERIMENT – 4

OBJECTIVE: Build an Artificial Neural Network by implementing the Backpropagation

algorithm and test the same using appropriate data sets.
PURPOSE: Artificial neural networks (ANNs) are powerful tools for machine learning with
applications in many areas including speech recognition, image classification, medical
diagnosis, and spam filtering. It has been shown that ANNs can approximate any function to
agu
nyardaengtereee of accuracy given enough neurons and training time. However, there is no
on the number of neurons required or the time it will take to train them. These are the main
disadvantages of using ANNs. Here we develop the BackPropogation algorithm which learns
the weights for a multilayer network, given a network with a fixed set of units and
ibnettewrceoennnections. It employs gradient descent to attempt to minimize the squared error
the network output values and target values for these outputs.
BACK PROPAGATION ALGORITHM:

Multiple layer perceptron are effectively applied to handle tricky problems if trained with a vastly
accepted algorithm identified as the back-propagation algorithm (error) in a supervised manner.
It functions on learning law with error-correction. It is also a simplified version for the least mean
square (LMS) filtering algorithm which is equally popular to error back-propagation algorithm. In Error
back- propagation training there are two computational passes via several network layers:
In forward pass, vector input is applied to the nodes of the system propagating each layer„s outcome

to
the next layer via network. To get the accurate response of the network, these outputs pass on from
several layers and arrive at a set of outputs. In forward pass network weights are permanent. On other
hand in the backward pass, weights are adjusted according to rule for error correction. Error signal is
the actual response of the network minus the desired response. The propagation of this error signal
through the network is towards backward in direction opposite to the connections of synaptic. The
move
the real response of network closer to the favored response, tuning of weights is to be done. There are
three unique features of a multilayer perception:
1. For each neuron in any system, its illustration has an activation function that is non-linear. The
logistical function is used to define a function which is sigmoid.
2. There are layer(s) of hidden neurons not contained in the input or the output present in the
neural network. The study over complex tasks is facilitated by these hidden neurons.
3. Connectivity degree is high in network. Weight's population should be changed if there is a
requirement to alter the connectivity of the network.
The stochastic gradient descent version of the BACKPROPAGATION algorithm for feed forward
networks containing two layers of sigmoid units.
Step 1: begins by constructing a network with the desired number of hidden and output units and
initializing all network weights to small random values. . For each training example, it applies the
network to the example, calculates the error of the network output for this example, computes the
gradient with respect to the error on this example, then updates all weights in the network. This
gradient
descent step is iterated (often thousands of times, using the same training examples multiple times)
until
the network performs acceptably well.
Step 2: The gradient descent weight-update rule is similar to the delta training rule The only difference
is that the error (t - o) in the delta rule is replaced by a more complex error term aj.

Nikhil Shukla
21003215301112
Step 3: updates weights incrementally, following the Presentation of each training example. This
corresponds to a stochastic approxi- mation to gradient descent. To obtain the true gradient of E one
would sum the Sj, xji values over all training examples before altering weight values.
Step 4: The weight-update loop in BACKPROPAGATION may be iterated thousands of times in a
typical application. A variety of termination conditions can be used to halt the procedure.
One may choose to halt after a fixed number of iterations through the loop, or once the error on the
training examples falls below some threshold.
PROGRAM:

import random

from math import exp

from random import seed
# Initialize a network
def initialize_network(n_inputs, n_hidden, n_outputs):

network = list()
hidden_layer = [{'weights':[random.uniform(-0.5,0.5) for i in range(n_inputs + 1)]} for i in range(n_hidden)]

network.append(hidden_layer)

output_layer = [{'weights':[random.uniform(-0.5,0.5) for i in range(n_hidden + 1)]} for i in range(n_outputs)]

network.append(output_layer)
i= 1

print("\n The initialised Neural Network:\n")

for layer in network:

j=1

for sub in layer:

print("\n Layer[%d] Node[%d]:\n" %(i,j),sub)

j=j+1
i=i+1

return network

# Calculate neuron activation (net) for an input

def activate(weights, inputs):

activation = weights[-1]

for i in range(len(weights)-1):

activation += weights[i] * inputs[i]

return activation

# Transfer neuron activation to sigmoid function

def transfer(activation):

Nikhil Shukla
2100321530112
return 1.0 / (1.0 + exp(-activation))

# Forward propagate input to a network output

def forward_propagate(network, row):

inputs = row

for layer in network:

new_inputs = []

for neuron in layer:

activation = activate(neuron['weights'], inputs)

neuron['output'] = transfer(activation)

new_inputs.append(neuron['output'])

inputs = new_inputs

return inputs

# Calculate the derivative of an neuron output

def transfer_derivative(output):
return output * (1.0 - output)

# Backpropagate error and store in neurons

def backward_propagate_error(network, expected):

for i in reversed(range(len(network))):

layer = network[i]

errors = list()
if i != len(network)-1:

for j in range(len(layer)):

error = 0.0

for neuron in network[i + 1]:

error += (neuron['weights'][j] * neuron['delta'])

errors.append(error)
else:

for j in range(len(layer)):

neuron = layer[j]
errors.append(expected[j] - neuron['output'])

for j in range(len(layer)):
neuron = layer[j]

neuron['delta'] = errors[j] * transfer_derivative(neuron['output']) #

Update network weights with error

Nikhil Shukla
2100321530112
def update_weights(network, row, l_rate):

for i in range(len(network)):
inputs = row[:-1]

if i != 0:

inputs = [neuron['output'] for neuron in network[i - 1]]

for neuron in network[i]:

for j in range(len(inputs)):

neuron['weights'][j] += l_rate * neuron['delta'] * inputs[j]

neuron['weights'][-1] += l_rate * neuron['delta']

# Train a network for a fixed number of epochs

def train_network(network, train, l_rate, n_epoch, n_outputs):

print("\n Network Training Begins:\n")

for epoch in range(n_epoch):

sum_error = 0

for row in train:

outputs = forward_propagate(network, row)

expected = [0 for i in range(n_outputs)]

expected[row[-1]] = 1

sum_error += sum([(expected[i]-outputs[i])**2 for i in range(len(expected))])

backward_propagate_error(network, expected)
update_weights(network, row, l_rate)

print('>epoch=%d, lrate=%.3f, error=%.3f' % (epoch, l_rate, sum_error))

print("\n Network Training Ends:\n")

#Test training backprop algorithm

seed(2)
dataset = [[2.7810836,2.550537003,0],

[1.465489372,2.362125076,0],

[3.396561688,4.400293529,0],

[1.38807019,1.850220317,0],
[3.06407232,3.005305973,0],

[7.627531214,2.759262235,1],

[5.332441248,2.088626775,1],

[6.922596716,1.77106367,1],
[8.675418651,-0.242068655,1],

Nikhil Shukla
2100321530112
[7.673756466,3.508563011,1]]

print("\n The input Data Set :\n",dataset)

n_inputs = len(dataset[0]) - 1

print("\n Number of Inputs :\n",n_inputs)

n_outputs = len(set([row[-1] for row in dataset]))

print("\n Number of Outputs :\n",n_outputs)

#Network Initialization

network = initialize_network(n_inputs, 2, n_outputs)

# Training the Network

train_network(network, dataset, 0.5, 20, n_outputs)

print("\n Final Neural Network :")

i= 1

for layer in network:

j=1

for sub in layer:

print("\n Layer[%d] Node[%d]:\n" %(i,j),sub)

j=j+1
i=i+1

OUTPUT:
The input Data Set :

[[2.7810836, 2.550537003, 0], [1.465489372, 2.362125076, 0], [3.396561688, 4.400293529, 0], [1.38807019,
1.850220317, 0], [3.06407232, 3.005305973, 0], [7.627531214, 2.759262235, 1], [5.332441248, 2.088626775,
1], [6.922596716, 1.77106367, 1], [8.675418651, -0.242068655, 1], [7.673756466, 3.508563011, 1]]

Number of Inputs :

Number of Outputs :

2
The initialised Neural Network:

Layer[1] Node[1]:

{'weights': [0.4560342718892494, 0.4478274870593494, -0.4434486322731913]}

Layer[1] Node[2]:

{'weights': [-0.41512800484107837, 0.33549887812944956, 0.2359699890685233]}

Layer[2] Node[1]:

{'weights': [0.1697304014402209, -0.1918635424108558, 0.10594416567846243]}

Layer[2] Node[2]:

Nikhil Shukla
2100321530112
{'weights': [0.10680173364083789, 0.08120401711200309, -0.3416171297451944]}

Network Training Begins:

>epoch=0, lrate=0.500, error=5.278

>epoch=1, lrate=0.500, error=5.122

>epoch=2, lrate=0.500, error=5.006

>epoch=3, lrate=0.500, error=4.875

>epoch=4, lrate=0.500, error=4.700

>epoch=5, lrate=0.500, error=4.466

>epoch=6, lrate=0.500, error=4.176

>epoch=7, lrate=0.500, error=3.838

>epoch=8, lrate=0.500, error=3.469

>epoch=9, lrate=0.500, error=3.089

>epoch=10, lrate=0.500, error=2.716

>epoch=11, lrate=0.500, error=2.367

>epoch=12, lrate=0.500, error=2.054

>epoch=13, lrate=0.500, error=1.780

>epoch=14, lrate=0.500, error=1.546

>epoch=15, lrate=0.500, error=1.349

>epoch=16, lrate=0.500, error=1.184

>epoch=17, lrate=0.500, error=1.045

>epoch=18, lrate=0.500, error=0.929

>epoch=19, lrate=0.500, error=0.831

Network Training Ends:

Final Neural Network :

Layer[1] Node[1]:
{'weights': [0.8642508164347665, -0.8497601716670763, -0.8668929014392035], 'output':
0.9295587965836384, 'delta': 0.005645382825629247}
Layer[1] Node[2]:

{'weights': [-1.2934302410111025, 1.7109363237151507, 0.7125327507327329], 'output':

0.04760703296164151, 'delta': -0.005928559978815076}
Layer[2] Node[1]:

{'weights': [-1.3098359335096292, 2.16462207144596, -0.3079052288835876], 'output':

0.19895563952058462, 'delta': -0.03170801648036037}
Layer[2] Node[2]:

{'weights': [1.5506793402414165, -2.11315950446121, 0.1333585709422027], 'output': 0.8095042653312078,

'delta': 0.029375796661413225}

Nikhil Shukla
2100321530112
 EXPERIMENT – 5
OBJECTIVE: Write a program to implement the naïve Bayesian classifier for a sample training data
set stored as a .CSV file. Compute the accuracy of the classifier, considering few test data sets.
PURPOSE: It is a classification technique based on Bayes? Theorem with an assumption of
independence among predictors. In simple terms, a Naive Bayes classifier assumes that the presence of
a particular feature in a class is unrelated to the presence of any other feature. For example, a fruit may
be considered to be an apple if it is red, round, and about 3 inches in diameter. Even if these features
depend on each other or upon the existence of the other features, all of these properties independently
contribute to the probability that this fruit is an apple and that is why it is known as „Naive?.
Bayes theorem provides a way of calculating posterior probability P(c|x) from P(c), P(x) and P(x|c).
Look at the equation below:
Handling Of Data:
load the data from the CSV file and split in to training and test data set.

Training data set can be used to by Naïve Bayes to make predictions.

And Test data set can be used to evaluate the accuracy of the model.
Summarize Data:
The summary of the training data collected involves the mean and the standard deviation for
each attribute, by class value.
These are required when making predictions to calculate the probability of specific attribute
values belonging to each class value.
summary data can be break down into the following sub-tasks:
a. Separate Data By Class: The first task is to separate the training dataset instances by class value
so that we can calculate statistics for each class. We can do that by creating a map of each class
value to a list of instances that belong to that class and sort the entire dataset of instances into
the appropriate lists.
b. Calculate Mean: We need to calculate the mean of each attribute for a class value. The mean is
the central middle or central tendency of the data, and we will use it as the middle of our
Gaussian distribution when calculating probabilities.
c. Calculate Standard Deviation: We also need to calculate the standard deviation of each attribute
for a class value. The standard deviation describes the variation of spread of the data, and we
will use it to characterize the expected spread of each attribute in our Gaussian distribution
when calculating probabilities.
d. Summarize Dataset: For a given list of instances (for a class value) we can calculate the mean
and the standard deviation for each attribute. The zip function groups the values for each
attribute across our data instances into their own lists so that we can compute the mean and
standard deviation values for the attribute.
e. Summarize Attributes By Class: We can pull it all together by first separating our training
dataset into instances grouped by class. Then calculate the summaries for each attribute.
Make Predictions:
Making predictions involves calculating the probability that a given data instance belongs to
each class,
then selecting the class with the largest probability as the prediction.
Finally, estimation of the accuracy of the model by making predictions for each data instance
in the test dataset.
Evaluate Accuracy:
The predictions can be compared to the class values in the test dataset and a classification\ accuracy can
be calculated as an accuracy ratio between 0& and 100%.
PROGRAM:
import csv
import random
import math

Nikhil Shukla
210032153012
# 1.Data Handling
# 1.1 Loading the Data from csv file of Pima indians diabetes dataset.
def loadcsv(filename):
lines = csv.reader(open(filename, "r"))
dataset = list(lines)
for i in range(len(dataset)):
# converting the attributes from string to floating point numbers
dataset[i] = [float(x) for x in dataset[i]]
return dataset
#1.2 Splitting the Data set into Training Set
def splitDataset(dataset, splitRatio):
trainSize = int(len(dataset) * splitRatio)
trainSet = []
copy = list(dataset)
while len(trainSet) < trainSize:
index = random.randrange(len(copy)) # random index
trainSet.append(copy.pop(index))
return [trainSet, copy]
def separateByClass(dataset):
separated = {}
for i in range(len(dataset)):
vector = dataset[i]
if (vector[-1] not in separated):
separated[vector[-1]] = []
separated[vector[-1]].append(vector)
return separated
def mean(numbers):
return sum(numbers)/float(len(numbers))
def stdev(numbers):
avg = mean(numbers)
variance = sum([pow(x-avg,2) for x in numbers])/float(len(numbers)-1)
return math.sqrt(variance)
def summarize(dataset):
summaries = [(mean(attribute), stdev(attribute)) for attribute in zip(*dataset)]
del summaries[-1]
return summaries
def summarizeByClass(dataset):
separated = separateByClass(dataset)
summaries = {}
for classValue, instances in separated.items():
summaries[classValue] = summarize(instances)
return summaries
def calculateProbability(x, mean, stdev):
exponent = math.exp(-(math.pow(x-mean,2)/(2*math.pow(stdev,2))))
return (1 / (math.sqrt(2*math.pi) * stdev)) * exponent
#3.2 Calculate Class Probabilities
def calculateClassProbabilities(summaries, inputVector):
probabilities = {}
for classValue, classSummaries in summaries.items():
probabilities[classValue] = 1
for i in range(len(classSummaries)):
mean, stdev = classSummaries[i]
x = inputVector[i]
probabilities[classValue] *= calculateProbability(x, mean, stdev)
return probabilities

Nikhil Shukla
2100321530112
def predict(summaries, inputVector):
probabilities = calculateClassProbabilities(summaries, inputVector)
bestLabel, bestProb = None, -1
for classValue, probability in probabilities.items():
if bestLabel is None or probability > bestProb:
bestProb = probability
bestLabel = classValue
return bestLabel
def getPredictions(summaries, testSet):
predictions = []
for i in range(len(testSet)):
result = predict(summaries, testSet[i])
predictions.append(result)
return predictions
#5. Computing Accuracy
def getAccuracy(testSet, predictions):
correct = 0
for i in range(len(testSet)):
if testSet[i][-1] == predictions[i]:
correct += 1
return (correct/float(len(testSet))) * 100.0
#Main Function
def main():
filename = 'ConceptLearning.csv'
splitRatio = 0.67
dataset = loadcsv(filename)
print("\n The length of the Data Set : ",len(dataset))
print("\n The Data Set Splitting into Training and Testing \n")
trainingSet, testSet = splitDataset(dataset, splitRatio)
print('\n Number of Rows in Training Set:{0} rows'.format(len(trainingSet)))
print('\n Number of Rows in Testing Set:{0} rows'.format(len(testSet)))
print("\n First Five Rows of Training Set:\n")
for i in range(0,5):
print(trainingSet[i],"\n")
print("\n First Five Rows of Testing Set:\n")
for i in range(0,5):
print(testSet[i],"\n")

# prepare model
summaries = summarizeByClass(trainingSet)
print("\n Model Summaries:\n",summaries)
# test model
predictions = getPredictions(summaries, testSet)
print("\nPredictions:\n",predictions)
accuracy = getAccuracy(testSet, predictions)
print('\n Accuracy: {0}%'.format(accuracy))
main()
OUTPUT:
The length of the Data Set : 768

The Data Set Splitting into Training and Testing

Number of Rows in Training Set:514 rows

Nikhil Shukla
2100321530112
Number of Rows in Testing Set:254 rows
First Five Rows of Training Set:

[4.0, 116.0, 72.0, 12.0, 87.0, 22.1, 0.463, 37.0, 0.0]

[0.0, 84.0, 64.0, 22.0, 66.0, 35.8, 0.545, 21.0, 0.0]

[0.0, 162.0, 76.0, 36.0, 0.0, 49.6, 0.364, 26.0, 1.0]

[10.0, 101.0, 86.0, 37.0, 0.0, 45.6, 1.136, 38.0, 1.0]

[5.0, 78.0, 48.0, 0.0, 0.0, 33.7, 0.654, 25.0, 0.0]

First Five Rows of Testing Set:

[1.0, 85.0, 66.0, 29.0, 0.0, 26.6, 0.351, 31.0, 0.0]

[8.0, 183.0, 64.0, 0.0, 0.0, 23.3, 0.672, 32.0, 1.0]

[4.0, 110.0, 92.0, 0.0, 0.0, 37.6, 0.191, 30.0, 0.0]

[10.0, 139.0, 80.0, 0.0, 0.0, 27.1, 1.441, 57.0, 0.0]

[7.0, 100.0, 0.0, 0.0, 0.0, 30.0, 0.484, 32.0, 1.0]

Model Summaries:
{0.0: [(3.3474320241691844, 3.045635385378286), (111.54380664652568, 26.040069054720693),
(68.45921450151057, 18.15540652389224), (19.94561933534743, 14.709615608767137),
(71.50151057401813, 101.04863439385403), (30.863141993957708, 7.207208162103949),
(0.4341842900302116, 0.2960911906946818), (31.613293051359516, 12.100651311117689)], 1.0:
[(4.469945355191257, 3.7369440851983082), (139.3879781420765, 33.733070931373234),
(71.14754098360656, 20.694403393963842), (22.92896174863388, 18.151995092528765),
(107.97814207650273, 146.92526156736633), (35.28633879781422, 7.783342260348583),
(0.5569726775956286, 0.3942245334398509), (36.78688524590164, 11.174610282702282)]}
Predictions:
[0.0, 1.0, 0.0, 1.0, 0.0, 1.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 0.0, 1.0, 0.0,
1.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0,
1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 1.0,
0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0,
0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 1.0, 1.0,
1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0,
0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0,
1.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0,
0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0,
0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0,
0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 1.0, 0.0, 0.0, 1.0,
0.0]

Accuracy: 80.31496062992126%

Nikhil Shukla
2100321530112