PRANVEER SINGH INSTITUTE OF TECHNOLOGY, KANPUR

DEPARTMENT OF COMPUTER SCIENCE & ENGINEERING

Even Semester 2023-24

B. Tech.- Third Year

Semester- VI

Lab File
Machine Learning Lab
(KAI651)

Submitted To : Submitted By :
Faculty Name : Kshmta Chauhan Name : Kamal Nayan Yagik
Designation : Assistant Professor Roll No. : 2101641520077
Section : CS-AI-3A
 Table of Contents
• Vision and Mission Statements of the Institute

• Vision and Mission Statements of the Department

• PEOs, POs, PSOs of the Department

• Course Objective and Outcomes

• List of Experiments

• Index
 Department Vision Statement
To be a recognized Department of Computer Science & Engineering that produces versatile
computer engineers, capable of adapting to the changing needs of computer and related industry.

Department Mission Statements

The mission of the Department of Computer Science and Engineering is:

i. To provide broad based quality education with knowledge and attitude to succeed in Computer
Science & Engineering careers.

ii. To prepare students for emerging trends in computer and related industry.

iii. To develop competence in students by providing them skills and aptitude to foster culture of
continuous and lifelong learning.

iv. To develop practicing engineers who investigate research, design, and find workable solutions to
complex engineering problems with awareness & concern for society as well as environment.

Program Educational Objectives (PEOs)

i. The graduates will be efficient leading professionals with knowledge of computer science &
engineering discipline that enables them to pursue higher education and/or successful careers in various
domains.

ii. Graduates will possess capability of designing successful innovative solutions to real life problems
that are technically sound, economically viable and socially acceptable.

iii. Graduates will be competent team leaders, effective communicators and capable of working in
multidisciplinary teams following ethical values.

iv. The graduates will be capable of adapting to new technologies/tools and constantly upgrading their
knowledge and skills with an attitude for lifelong learning
 Department Program Outcomes (POs)
The students of Computer Science and Engineering Department will be able:

1. Engineering knowledge: Apply the knowledge of mathematics, science, Computer Science &
Engineering fundamentals, and an engineering specialization to the solution of complex engineering
problems.

2. Problem analysis: Identify, formulate, review research literature, and analyze complex
engineering problems reaching substantiated conclusions using first principles of mathematics,
natural sciences, and Computer Science & Engineering sciences.

3. Design/development of solutions: Design solutions for complex Computer Science &

Engineering problems and design system components or processes that meet the specified needs
with appropriate consideration for the public health and safety, and the cultural, societal, and
environmental considerations.

4. Investigation: Use research-based knowledge and research methods including design of

experiments, analysis and interpretation of data, and synthesis of the information to provide valid
conclusions.

5. Modern tool usage: Create, select, and apply appropriate techniques, resources, and modern
engineering and IT tools including prediction and modelling to complex Computer Science &
Engineering activities with an understanding of the limitations.

6. The Engineering and Society: Apply reasoning informed by the contextual knowledge to assess
societal, health, safety, legal and cultural issues and the consequent responsibilities relevant to the
professional engineering practice in the field of Computer Science and Engineering.

7. Environment and sustainability: Understand the impact of the professional Computer Science
& Engineering solutions in societal and environmental contexts, and demonstrate the knowledge of,
and need for sustainable development.

8. Ethics: Apply ethical principles and commit to professional ethics and responsibilities and norms
of the Computer Science & Engineering practice.

9. Individual and team work: Function effectively as an individual, and as a member or leader in
diverse teams, and in multidisciplinary settings.

10. Communication: Communicate effectively on complex Computer Science & Engineering

activities with the engineering community and with society at large, such as, being able to
comprehend and write effective reports and design documentation, make effective presentations,
and give and receive clear instructions.

11. Project management and finance: Demonstrate knowledge and understanding of the
Computer Science & Engineering and management principles and apply these to one’s own work,
as a member and leader in a team, to manage projects and in multidisciplinary environments.

12. Life-long learning: Recognize the need for, and have the preparation and ability to engage in
independent and life-long learning in the broadest context of technological change.
 Department Program Specific Outcomes (PSOs)
The students will be able to:

1. Use algorithms, data structures/management, software design, concepts of programming

languages and computer organization and architecture.

2. Understand the processes that support the delivery and management of information systems
within a specific application environment.

Course Outcomes

*Level of Bloom’s Level to be

*Level of Bloom’s Taxonomy Level to be met
Taxonomy Met
L1: Remember 1 L2: Understand 2
L3: Apply 3 L4: Analyze 4
L5: Evaluate 5 L6: Create 6

CO Number Course Outcomes

Remember the mathematical and statistical prospective of machine learning
KAI-651.1 algorithms through python programming.
KAI-651.2 Understand machine learning algorithms to solve real world problems.
Apply appropriate data sets to the machine learning algorithms and predict
KAI-651.3 unknown target data class or form clusters.
Design Java/Python programs for various learning algorithms and analyze various
KAI-651.4 performance metrics result by cross validation testing method.
 List of Experiments

Lab Lab Experiment Correspo

No. nding CO
Implement and demonstrate the FIND-S algorithm for finding the most specific
1 hypothesis based on a given set of training data samples. Read the training data
from a .CSV file. 1
For a given set of training data examples stored in a .CSV file, implement and
2 demonstrate the Candidate- Elimination algorithm to output a description of the
set of all hypotheses consistent with the training examples. 2
Write a program to demonstrate the working of the decision tree based ID3
3 algorithm. Use an appropriate data set for building the decision tree and apply this
knowledge to classify a new sample. 2
Build an Artificial Neural Network by implementing the Backpropagation
4
algorithm and test the same using appropriate data sets. 4
Write a program to implement the naïve Bayesian classifier for a sample training
5 data set stored as a .CSV file. Compute the accuracy of the classifier, considering
few test data sets. 2
Assuming a set of documents that need to be classified, use the naïve Bayesian
6 Classifier model to perform this task. Built-in Java classes/API can be used to
write the program. Calculate the accuracy, precision, and recall for your data set. 3
Write a program to construct a Bayesian network considering medical data. Use
7 this model to demonstrate the diagnosis of heart patients using standard Heart
Disease Data Set. You can use Java/Python ML library classes/API. 4
Apply EM algorithm to cluster a set of data stored in a .CSV file. Use the same
data set for clustering using k-Means algorithm. Compare the results of these two
8
algorithms and comment on the quality of clustering. You can add Java/Python
ML library classes/API in the program. 3
Write a program to implement k-Nearest Neighbour algorithm to classify the iris
9 data set. Print both correct and wrong predictions. Java/Python ML library classes
can be used for this problem. 2
Implement the non-parametric Locally Weighted Regression algorithm in order to
10
fit data points. Select appropriate data set for your experiment and draw graphs. 3
 INDEX

S Lab Experiment Date of Date of Marks Faculty

No Experiment Submission Signature
Implement and demonstrate the FIND-S algorithm
1 for finding the most specific hypothesis based on a
given set of training data samples. Read the training
data from a .CSV file.
For a given set of training data examples stored in a
2 .CSV file, implement and demonstrate the
Candidate- Elimination algorithm to output a
description of the set of all hypotheses consistent
with the training examples.
Write a program to demonstrate the working of the
3 decision tree based ID3 algorithm. Use an
appropriate data set for building the decision tree
and apply this knowledge to classify a new sample.
Build an Artificial Neural Network by
4 implementing the Backpropagation algorithm and
test the same using appropriate data sets.
Write a program to implement the naïve Bayesian
5 classifier for a sample training data set stored as a
.CSV file. Compute the accuracy of the classifier,
considering few test data sets.
Assuming a set of documents that need to be
6 classified, use the naïve Bayesian Classifier model
to perform this task. Built-in Java classes/API can
be used to write the program. Calculate the
accuracy, precision, and recall for your data set.
Write a program to construct a Bayesian network
7 considering medical data. Use this model to
demonstrate the diagnosis of heart patients using
standard Heart Disease Data Set. You can use
Java/Python ML library classes/API.
Apply EM algorithm to cluster a set of data stored
8 in a .CSV file. Use the same data set for clustering
using k-Means algorithm. Compare the results of
these two algorithms and comment on the quality of
clustering. You can add Java/Python ML library
classes/API in the program.
Write a program to implement k-Nearest Neighbour
9 algorithm to classify the iris data set. Print both
correct and wrong predictions. Java/Python ML
library classes can be used for this problem.
Implement the non-parametric Locally Weighted
10 Regression algorithm in order to fit data points.
Select appropriate data set for your experiment and
draw graphs.
 Experiment-1
Implement and demonstrate the FIND-S algorithm for finding the most specific hypothesis based on
a given set of training data samples. Read the training data from a .CSV file.
FIND-S Algorithm
1. Initialize h to the most specific hypothesis in H
2. for each positive training instance x
for each attribute constraint ai in h
if the constraint ai is satisfied by x then
do nothing
else
replace ai in h by the next more general constraint that is satisfied by x
3. Output hypothesis h

Training Examples:

sky airtemp humidity wind water forcast enjoysport

sunny warm normal strong warm same yes
sunny warm high strong warm same yes
rainy cold high strong warm change no
sunny warm high strong cool change yes

Program:
import csv
num_attributes = 6
a = []
print("\n The Given Training Data Set \n")
with open('enjoysport.csv', 'r') as csvfile:
reader = csv.reader(csvfile)
for row in reader:
a.append (row)
print(row)
print("\n The initial value of hypothesis: ")
hypothesis = ['0'] * num_attributes
print(hypothesis)
for j in range(0,num_attributes):
hypothesis[j] = a[0][j];
print("\n Find S: Finding a Maximally Specific Hypothesis\n")
for i in range(0,len(a)):
if a[i][num_attributes]=='yes':
for j in range(0,num_attributes):
if a[i][j]!=hypothesis[j]:
hypothesis[j]='?'
else :
hypothesis[j]= a[i][j]
print(" For Training instance No:{0} the hypothesis is ".format(i),hypothesis)
print("\n The Maximally Specific Hypothesis for a given Training Examples :\n")
print(hypothesis)
Data Set:

sunny warm normal strong warm same yes

sunny warm high strong warm same yes
rainy cold high strong warm change no
sunny warm high strong cool change yes

Output:

The Given Training Data Set

['sunny', 'warm', 'normal', 'strong', 'warm', 'same', 'yes']
['sunny', 'warm', 'high', 'strong', 'warm', 'same', 'yes']
['rainy', 'cold', 'high', 'strong', 'warm', 'change', 'no']
['sunny', 'warm', 'high', 'strong', 'cool', 'change', 'yes']

The initial value of hypothesis:

['0', '0', '0', '0', '0', '0']

Find S: Finding a Maximally Specific Hypothesis

For Training Example No:0 the hypothesis is
['sunny', 'warm', 'normal', 'strong', 'warm', 'same']
For Training Example No:1 the hypothesis is
['sunny', 'warm', '?', 'strong', 'warm', 'same']
For Training Example No:2 the hypothesis is
['sunny', 'warm', '?', 'strong', 'warm', 'same']
For Training Example No:3 the hypothesis is
['sunny', 'warm', '?', 'strong', '?', '?']
The Maximally Specific Hypothesis for a given Training Examples:
['sunny', 'warm', '?', 'strong', '?', '?']
 Experiment-2
For a given set of training data examples stored in a .CSV file, implement and demonstrate the
Candidate- Elimination algorithm to output a description of the set of all hypotheses consistent with
the training examples.

sky airtemp humidity wind water forcast enjoysport

sunny warm normal strong warm same yes
sunny warm high strong warm same yes
rainy cold high strong warm change no
sunny warm high strong cool change yes

Program:-
import numpy as np
import pandas as pd

data = pd.read_csv('enjoysport.csv')
concepts = np.array(data.iloc[:,0:-1])
print(concepts)
target = np.array(data.iloc[:,-1])
print(target)
def learn(concepts, target):
specific_h = concepts[0].copy()
print("initialization of specific_h and general_h")
print(specific_h)
general_h = [["?" for i in range(len(specific_h))] for i in range(len(specific_h))]
print(general_h)

for i, h in enumerate(concepts):
print("For Loop Starts")
if target[i] == "yes":
print("If instance is Positive ")
for x in range(len(specific_h)):
if h[x]!= specific_h[x]:
specific_h[x] ='?'
general_h[x][x] ='?'

if target[i] == "no":
print("If instance is Negative ")
for x in range(len(specific_h)):
if h[x]!= specific_h[x]:
general_h[x][x] = specific_h[x]
else:
general_h[x][x] = '?'

print(" steps of Candidate Elimination Algorithm",i+1)

print(specific_h)
print(general_h)
print("\n")
print("\n")

indices = [i for i, val in enumerate(general_h) if val == ['?', '?', '?', '?', '?', '?']]
for i in indices:
 general_h.remove(['?', '?', '?', '?', '?', '?'])
return specific_h, general_h

s_final, g_final = learn(concepts, target)

print("Final Specific_h:", s_final, sep="\n")

print("Final General_h:", g_final, sep="\n")

OUTPUT:

Initialization of specific_h and general_h

['sunny' 'warm' 'normal' 'strong' 'warm' 'same']
[['?', '?', '?', '?', '?', '?'], ['?', '?', '?', '?', '?', '?'], ['?', '?', '?', '?', '?', '?'], ['?', '?', '?', '?', '?', '?'], ['?', '?', '?', '?', '?', '?'],
['?', '?', '?', '?', '?', '?']]

Steps of Candidate Elimination Algorithm 1

['sunny' 'warm' 'normal' 'strong' 'warm' 'same']
[['?', '?', '?', '?', '?', '?'], ['?', '?', '?', '?', '?', '?'], ['?', '?', '?', '?', '?', '?'], ['?', '?', '?', '?', '?', '?'], ['?', '?', '?', '?', '?', '?'],
['?', '?', '?', '?', '?', '?']]

Steps of Candidate Elimination Algorithm 2

['sunny' 'warm' '?' 'strong' 'warm' 'same']
[['?', '?', '?', '?', '?', '?'], ['?', '?', '?', '?', '?', '?'], ['?', '?', '?', '?', '?', '?'], ['?', '?', '?', '?', '?', '?'], ['?', '?', '?', '?', '?', '?'],
['?', '?', '?', '?', '?', '?']]
Steps of Candidate Elimination Algorithm 3
['sunny' 'warm' '?' 'strong' 'warm' 'same']
[['sunny', '?', '?', '?', '?', '?'], ['?', 'warm', '?', '?', '?', '?'], ['?', '?', '?', '?', '?', '?'], ['?', '?', '?', '?', '?', '?'], ['?', '?', '?', '?',
'?', '?'], ['?', '?', '?', '?', '?', 'same']]

Steps of Candidate Elimination Algorithm 4

['sunny' 'warm' '?' 'strong' '?' '?']
[['sunny', '?', '?', '?', '?', '?'], ['?', 'warm', '?', '?', '?', '?'], ['?', '?', '?', '?', '?', '?'], ['?', '?', '?', '?', '?', '?'], ['?', '?', '?', '?',
'?', '?'], ['?', '?', '?', '?', '?', '?']]

Final Specific_h:
['sunny' 'warm' '?' 'strong' '?' '?']

Final General_h:
[['sunny', '?', '?', '?', '?', '?'], ['?', 'warm', '?', '?', '?', '?']]
 Experiment-3
Write a program to demonstrate the working of the decision tree based ID3 algorithm. Use an
appropriate data set for building the decision tree and apply this knowledge to classify a new
sample.

ID3 Steps

1. Calculate the Information Gain of each feature.

2. Considering that all rows don’t belong to the same class, split the dataset S into subsets using the feature

for which the Information Gain is maximum.

3. Make a decision tree node using the feature with the maximum Information gain.

4. If all rows belong to the same class, make the current node as a leaf node with the class as its label.

5. Repeat for the remaining features until we run out of all features, or the decision tree has all leaf nodes.

We denote our dataset as S, entropy is calculated as:

Entropy(S) = - ∑ pᵢ * log₂(pᵢ) ; i = 1 to n

where,

n is the total number of classes in the target column (in our case n = 2 i.e YES and NO)

pᵢ is the probability of class ‘i’ or the ratio of “number of rows with class i in the target column” to the “total

number of rows” in the dataset.Information Gain for a feature column A is calculated as:

IG(S, A) = Entropy(S) - ∑((|Sᵥ| / |S|) * Entropy(Sᵥ))

where Sᵥ is the set of rows in S for which the feature column A has value v, |Sᵥ| is the number of rows

in Sᵥ and likewise |S| is the number of rows in S.

Programs:-
import math
import csv
def load_csv(filename):
lines=csv.reader(open(filename,"r"));
dataset = list(lines)
headers = dataset.pop(0)
return dataset,headers

class Node:
def __init__(self,attribute):
self.attribute=attribute
self.children=[]
 self.answer=""

def subtables(data,col,delete):
dic={}
coldata=[row[col] for row in data]
attr=list(set(coldata))

counts=[0]*len(attr)
r=len(data)
c=len(data[0])
for x in range(len(attr)):
for y in range(r):
if data[y][col]==attr[x]:
counts[x]+=1

for x in range(len(attr)):
dic[attr[x]]=[[0 for i in range(c)] for j in range(counts[x])]
pos=0
for y in range(r):
if data[y][col]==attr[x]:
if delete:
del data[y][col]
dic[attr[x]][pos]=data[y]
pos+=1
return attr,dic

def entropy(S):
attr=list(set(S))
if len(attr)==1:
return 0

counts=[0,0]
for i in range(2):
counts[i]=sum([1 for x in S if attr[i]==x])/(len(S)*1.0)

sums=0
for cnt in counts:
sums+=-1*cnt*math.log(cnt,2)
return sums

def compute_gain(data,col):
attr,dic = subtables(data,col,delete=False)

total_size=len(data)
entropies=[0]*len(attr)
ratio=[0]*len(attr)

total_entropy=entropy([row[-1] for row in data])

 for x in range(len(attr)):
ratio[x]=len(dic[attr[x]])/(total_size*1.0)
entropies[x]=entropy([row[-1] for row in dic[attr[x]]])
total_entropy-=ratio[x]*entropies[x]
return total_entropy

def build_tree(data,features):
lastcol=[row[-1] for row in data]
if(len(set(lastcol)))==1:
node=Node("")
node.answer=lastcol[0]
return node

n=len(data[0])-1
gains=[0]*n
for col in range(n):
gains[col]=compute_gain(data,col)
split=gains.index(max(gains))
node=Node(features[split])
fea = features[:split]+features[split+1:]

attr,dic=subtables(data,split,delete=True)

for x in range(len(attr)):
child=build_tree(dic[attr[x]],fea)
node.children.append((attr[x],child))
return node

def print_tree(node,level):
if node.answer!="":
print(" "*level,node.answer)
return

print(" "*level,node.attribute)
for value,n in node.children:
print(" "*(level+1),value)
print_tree(n,level+2)

def classify(node,x_test,features):
if node.answer!="":
print(node.answer)
return
pos=features.index(node.attribute)
for value, n in node.children:
if x_test[pos]==value:
classify(n,x_test,features)

'''Main program'''
dataset,features=load_csv(r'C:\Users\anil1\Desktop\ML KAI601 Notes\ML Lab\id3.csv')
node1=build_tree(dataset,features)
print("The decision tree for the dataset using ID3 algorithm is")
print_tree(node1,0)
testdata,features=load_csv(r'C:\Users\anil1\Desktop\ML KAI601 Notes\ML Lab\id3_test_1.csv')

for xtest in testdata:

print("The test instance:",xtest)
print("The label for test instance:",end=" ")
classify(node1,xtest,features)

Training Data:
Outlook Temperature Humidity Wind PlayTennis
Sunny hot high weak no
Sunny hot high strong no
Overcast hot high weak yes
Rain mild high weak yes
Rain cool normal weak yes
Rain cool normal strong no
Overcast cool normal strong yes
Sunny mild high weak no
Sunny cool normal weak yes
Rain mild normal weak yes
Sunny mild normal strong yes
Overcast mild high strong yes
Overcast hot normal weak yes
Rain mild high strong no

Test Data:
Outlook Temperature Humidity Wind
Rain cool normal strong
Sunny mild normal strong

OUTPUT:
ID3 Tree:

The decision tree for the dataset using ID3 algorithm is

Outlook
overcast
yes
rain
Wind
weak
yes
strong
no
sunny
Humidity
normal
yes
high
no
The test instance: ['rain', 'cool', 'normal', 'strong']
The label for test instance: no
The test instance: ['sunny', 'mild', 'normal', 'strong']
The label for test instance: yes
 Experiment-4
Build an Artificial Neural Network by implementing the Backpropagation algorithm and test the
same using appropriate data sets.

Programs:-

import numpy as np
X = np.array(([2, 9], [1, 5], [3, 6]), dtype=float) # two inputs [sleep,study]
y = np.array(([92], [86], [89]), dtype=float) # one output [Expected % in Exams]
X = X/np.amax(X,axis=0) # maximum of X array longitudinally
y = y/100

#Sigmoid Function
def sigmoid (x):
return 1/(1 + np.exp(-x))

#Derivative of Sigmoid Function

def derivatives_sigmoid(x):
return x * (1 - x)

#Variable initialization
epoch=5000 #Setting training iterations
lr=0.1 #Setting learning rate
inputlayer_neurons = 2 #number of features in data set
hiddenlayer_neurons = 3 #number of hidden layers neurons
output_neurons = 1 #number of neurons at output layer

#weight and bias initialization

wh=np.random.uniform(size=(inputlayer_neurons,hiddenlayer_neurons)) #weight of the link from input
node to hidden node
bh=np.random.uniform(size=(1,hiddenlayer_neurons)) # bias of the link from input node to hidden node
wout=np.random.uniform(size=(hiddenlayer_neurons,output_neurons)) #weight of the link from hidden
node to output node
bout=np.random.uniform(size=(1,output_neurons)) #bias of the link from hidden node to output node

#draws a random range of numbers uniformly of dim x*y

for i in range(epoch):

#Forward Propogation
hinp1=np.dot(X,wh)
hinp=hinp1 + bh
hlayer_act = sigmoid(hinp)
outinp1=np.dot(hlayer_act,wout)
outinp= outinp1+ bout
output = sigmoid(outinp)

#Backpropagation
EO = y-output
outgrad = derivatives_sigmoid(output)
d_output = EO* outgrad
EH = d_output.dot(wout.T)
#how much hidden layer weights contributed to error
hiddengrad = derivatives_sigmoid(hlayer_act)
d_hiddenlayer = EH * hiddengrad

# dotproduct of nextlayererror and currentlayerop

wout += hlayer_act.T.dot(d_output) *lr
wh += X.T.dot(d_hiddenlayer) *lr

print("Input: \n" + str(X))

print("Actual Output: \n" + str(y))
print("Predicted Output: \n" ,output)

Input:
[[0.66666667 1. ]
[0.33333333 0.55555556]
[1. 0.66666667]]
Actual Output:
[[0.92]
[0.86]
[0.89]]
Predicted Output:
[[0.85726252]
[0.83545767]
[0.85611575]]
 Experiment-5
Write a program to implement the naïve Bayesian classifier for a sample training data set stored as a
.CSV file. Compute the accuracy of the classifier, considering few test data sets.

Data set:
Examples Pregnancies Glucose BloodPressure SkinThickness Insulin BMI Diabetic Age Outcome
Pedigree
Function
1 6 148 72 35 0 33.6 0.627 50 1
2 1 85 66 29 0 26.6 0.351 31 0
3 8 183 64 0 0 23.3 0.672 32 1
4 1 89 66 23 94 28.1 0.167 21 0
5 0 137 40 35 168 43.1 2.288 33 1
6 5 116 74 0 0 25.6 0.201 30 0
7 3 78 50 32 88 31 0.248 26 1
8 10 115 0 0 0 35.3 0.134 29 0
9 2 197 70 45 543 30.5 0.158 53 1
10 8 125 96 0 0 0 0.232 54 1

Program:

import csv
import random
import math

def load_csv(filename):
lines = csv.reader(open(filename, "r"))
dataset = list(lines)
for i in range(len(dataset)):
# converting strings into numbers for processing
dataset[i] = [float(x) for x in dataset[i]]
return dataset

def split_dataset(dataset, split_ratio):

# 67% training size
train_size = int(len(dataset) * split_ratio)
train_set = []
copy = list(dataset)
while len(train_set) < train_size:
# generate indices for the dataset list randomly to pick elements for training data
index = random.randrange(len(copy))
train_set.append(copy.pop(index))
return [train_set, copy]

def separate_by_class(dataset):
separated = {} # dictionary of classes 1 and 0
# creates a dictionary of classes 1 and 0 where the values are the instances belonging to each class
for i in range(len(dataset)):
vector = dataset[i]
if (vector[-1] not in separated):
separated[vector[-1]] = []
 separated[vector[-1]].append(vector)
return separated

def mean(numbers):
return sum(numbers) / float(len(numbers))

def stdev(numbers):
avg = mean(numbers)
variance = sum([pow(x - avg, 2) for x in numbers]) / float(len(numbers) - 1)
return math.sqrt(variance)

def summarize(dataset):
# creates a list of tuples [(mean(attribute), stdev(attribute)) for attribute in zip(*dataset)]
summaries = [(mean(attribute), stdev(attribute)) for attribute in zip(*dataset)]
del summaries[-1] # excluding labels +ve or -ve
return summaries

def summarize_by_class(dataset):
separated = separate_by_class(dataset)
summaries = {}
for class_value, instances in separated.items():
summaries[class_value] = summarize(instances) # summarize is used to calculate mean and #std
return summaries

def calculate_probability(x, mean, stdev):

exponent = math.exp(-(math.pow(x - mean, 2) / (2 * math.pow(stdev, 2))))
return (1 / (math.sqrt(2 * math.pi) * stdev)) * exponent

def calculate_class_probabilities(summaries, input_vector):

# probabilities contains the probabilities of all classes for test data
probabilities = {}
for class_value, class_summaries in summaries.items():
# class and attribute information as mean and std
probabilities[class_value] = 1
for i in range(len(class_summaries)):
mean, stdev = class_summaries[i] # take mean and std of every attribute for class 0 and 1
#separately
x = input_vector[i] # test vector's first attribute
probabilities[class_value] *= calculate_probability(x, mean, stdev) # use normal distribution
return probabilities

def predict(summaries, input_vector):

# training and test data are passed
probabilities = calculate_class_probabilities(summaries, input_vector)
best_label, best_prob = None, -1
for class_value, probability in probabilities.items():
# assigns that class which has the highest probability
if best_label is None or probability > best_prob:
best_prob = probability
best_label = class_value
return best_label

def get_predictions(summaries, test_set):

predictions = []
for i in range(len(test_set)):
 result = predict(summaries, test_set[i])
predictions.append(result)
return predictions

def get_accuracy(test_set, predictions):

correct = 0
for i in range(len(test_set)):
if test_set[i][-1] == predictions[i]:
correct += 1
return (correct / float(len(test_set))) * 100.0

def main():
filename = 'naivedata.csv'
split_ratio = 0.67
dataset = load_csv(filename)
training_set, test_set = split_dataset(dataset, split_ratio)
print('Split {0} rows into train={1} and test={2} rows'.format(len(dataset), len(training_set),
len(test_set)))

# prepare model
summaries = summarize_by_class(training_set)

# test model
predictions = get_predictions(summaries, test_set) # find the predictions of test data with the #training
data
accuracy = get_accuracy(test_set, predictions)
print('Accuracy of the classifier is: {0}%'.format(accuracy))

main()

Output:

Split 768 rows into train=514 and test=254 rows

Accuracy of the classifier is : 71.65354330708661%
 Experiment-6
Assuming a set of documents that need to be classified, use the naïve Bayesian Classifier model to
perform this task. Built-in Java classes/API can be used to write the program. Calculate the
accuracy, precision, and recall for your data set.

Data set:-
Text Documents Label
1 I love this sandwich pos
2 This is an amazing place pos
3 I feel very good about these beers pos
4 This is my best work pos
5 What an awesome view pos
6 I do not like this restaurant neg
7 I am tired of this stuff neg
8 I can't deal with this neg
9 He is my sworn enemy neg
10 My boss is horrible neg
11 This is an awesome place pos
12 I do not like the taste of this juice neg
13 I love to dance pos
14 I am sick and tired of this place neg
15 What a great holiday pos
16 That is a bad locality to stay neg
17 We will have good fun tomorrow pos
18 I went to my enemy's house today neg

Program:-

import pandas as pd

# Read the dataset

msg = pd.read_csv('naivetext.csv', names=['message', 'label'])
print('The dimensions of the dataset', msg.shape)

# Map label to a numerical value

msg['labelnum'] = msg.label.map({'pos': 1, 'neg': 0})
X = msg.message
y = msg.labelnum

print(X)
print(y)

# Splitting the dataset into train and test data

from sklearn.model_selection import train_test_split
xtrain, xtest, ytrain, ytest = train_test_split(X, y)

print('\n The total number of Training Data:', ytrain.shape)

print('\n The total number of Test Data:', ytest.shape)

# Output of count vectorizer is a sparse matrix

from sklearn.feature_extraction.text import CountVectorizer
count_vect = CountVectorizer()
xtrain_dtm = count_vect.fit_transform(xtrain)
xtest_dtm = count_vect.transform(xtest)

print('\n The words or Tokens in the text documents \n')

print(count_vect.get_feature_names_out())

# Create a DataFrame from the training data matrix

df = pd.DataFrame(xtrain_dtm.toarray(), columns=count_vect.get_feature_names_out())

# Training Naive Bayes (NB) classifier on training data

from sklearn.naive_bayes import MultinomialNB
clf = MultinomialNB().fit(xtrain_dtm, ytrain)
predicted = clf.predict(xtest_dtm)

# Printing accuracy, Confusion matrix, Precision and Recall

from sklearn import metrics

print('\n Accuracy of the classifier is', metrics.accuracy_score(ytest, predicted))

print('\n Confusion matrix')
print(metrics.confusion_matrix(ytest, predicted))
print('\n The value of Precision:', metrics.precision_score(ytest, predicted))
print('\n The value of Recall:', metrics.recall_score(ytest, predicted))

Output:
The dimensions of the dataset (18, 2)
0 I love this sandwich
1 This is an amazing place
2 I feel very good about these beers
3 This is my best work
4 What an awesome view
5 I do not like this restaurant
6 I am tired of this stuff
7 I can't deal with this
8 He is my sworn enemy
9 My boss is horrible
10 This is an awesome place
11 I do not like the taste of this juice
12 I love to dance
13 I am sick and tired of this place
14 What a great holiday
15 That is a bad locality to stay
16 We will have good fun tomorrow
17 I went to my enemy's house today
Name: message, dtype: object
0 1
1 1
2 1
3 1
4 1
5 0
6 0
7 0
8 0
9 0
10 1
11 0
12 1
13 0
14 1
15 0
16 1
17 0
Name: labelnum, dtype: int64

The total number of Training Data: (13,) The total number of Test Data: (5,)

The words or Tokens in the text documents

['about', 'am', 'amazing', 'an', 'and', 'awesome', 'beers', 'best', 'can', 'deal', 'do', 'enemy', 'feel',
'fun', 'good', 'great', 'have', 'he', 'holiday', 'house', 'is', 'like', 'love', 'my', 'not', 'of', 'place',
'restaurant', 'sandwich', 'sick', 'sworn', 'these', 'this', 'tired', 'to', 'today', 'tomorrow', 'very', 'view', 'we',
'went', 'what', 'will', 'with', 'work']

Accuracy of the classifier is 0.8 Confusion matrix

[[2 1]
[0 2]]
The value of Precision 0.6666666666666666
The value of Recall 1.0
 Experiment-7
Write a program to construct a Bayesian network considering medical data. Use this model to
demonstrate the diagnosis of heart patients using standard Heart Disease Data Set. You can use
Java/Python ML library classes/API.

Data Set:

Title: Heart Disease Databases

The Cleveland database contains 76 attributes, but all published experiments refer to using a
subset of 14 of them. In particular, the Cleveland database is the only one that has been used by
ML researchers to this date. The "Heartdisease" field refers to the presence of heart diseasein
the patient. It is integer valued from 0 (no presence) to 4.
Database: 0 1 2 3 4 Total
Cleveland: 164 55 36 35 13 303

Attribute Information:
1. age: age in years
2. sex: sex (1 = male; 0 = female)
3. cp: chest pain type
• Value 1: typical angina
• Value 2: atypical angina
• Value 3: non-anginal pain
• Value 4: asymptomatic
4. trestbps: resting blood pressure (in mm Hg on admission to the hospital)
5. chol: serum cholestoral in mg/dl
6. fbs: (fasting blood sugar > 120 mg/dl) (1 = true; 0 = false)
7. restecg: resting electrocardiographic results
• Value 0: normal
• Value 1: having ST-T wave abnormality (T wave inversions and/or ST
elevationor depression of > 0.05 mV)
• Value 2: showing probable or definite left ventricular hypertrophy by
Estes'criteria
8. thalach: maximum heart rate achieved
9. exang: exercise induced angina (1 = yes; 0 = no)
10. oldpeak = ST depression induced by exercise relative to rest
11. slope: the slope of the peak exercise ST segment
• Value 1: upsloping
• Value 2: flat
• Value 3: downsloping
12. thal: 3 = normal; 6 = fixed defect; 7 = reversable defect
13. Heartdisease: It is integer valued from 0 (no presence) to 4.
 Some instance from the dataset:

age sex cp trestbps chol fbs restecg thalach exang oldpeak slope ca thal Heartdisease
63 1 1 145 233 1 2 150 0 2.3 3 0 6 0
67 1 4 160 286 0 2 108 1 1.5 2 3 3 2
67 1 4 120 229 0 2 129 1 2.6 2 2 7 1
41 0 2 130 204 0 2 172 0 1.4 1 0 3 0
62 0 4 140 268 0 2 160 0 3.6 3 2 3 3
60 1 4 130 206 0 2 132 1 2.4 2 2 7 4

Program:-
import numpy as np
import pandas as pd
import csv
from pgmpy.estimators import MaximumLikelihoodEstimator
from pgmpy.models import BayesianModel
from pgmpy.inference import VariableElimination

# Read Cleveland Heart Disease data

heartDisease = pd.read_csv('heart.csv')
heartDisease = heartDisease.replace('?', np.nan)

# Display the data

print('Sample instances from the dataset are given below')
print(heartDisease.head())

# Display the Attributes names and datatypes

print('\n Attributes and datatypes')
print(heartDisease.dtypes)

# Create Model - Bayesian Network

model = BayesianModel([
('age', 'heartdisease'),
('sex', 'heartdisease'),
('exang', 'heartdisease'),
('cp', 'heartdisease'),
('heartdisease', 'restecg'),
('heartdisease', 'chol')
])

# Learning CPDs using Maximum Likelihood Estimators

print('\n Learning CPD using Maximum likelihood estimators')
model.fit(heartDisease, estimator=MaximumLikelihoodEstimator)

# Inferencing with Bayesian Network

print('\n Inferencing with Bayesian Network:')
HeartDiseasetest_infer = VariableElimination(model)

# Computing the Probability of HeartDisease given restecg

print('\n 1. Probability of HeartDisease given evidence= restecg:1')
q1 = HeartDiseasetest_infer.query(variables=['heartdisease'], evidence={'restecg': 1})
print(q1)

# Computing the Probability of HeartDisease given cp

print('\n 2. Probability of HeartDisease given evidence= cp:2')
q2 = HeartDiseasetest_infer.query(variables=['heartdisease'], evidence={'cp': 2})
print(q2)

Output:
 Experiment-8
Apply EM algorithm to cluster a set of data stored in a .CSV file. Use the same data set for
clustering using k-Means algorithm. Compare the results of these two algorithms and
comment on the quality of clustering. You can add Java/Python ML library classes/API in the
program.
Program:

import matplotlib.pyplot as plt

from sklearn import datasets
from sklearn.cluster import KMeans
import sklearn.metrics as sm
import pandas as pd
import numpy as np
from sklearn.mixture import GaussianMixture
from sklearn import preprocessing

# Load the Iris dataset

iris = datasets.load_iris()

# Create a DataFrame for the features

X = pd.DataFrame(iris.data)
X.columns = ['Sepal_Length', 'Sepal_Width', 'Petal_Length', 'Petal_Width']

# Create a DataFrame for the target

y = pd.DataFrame(iris.target)
y.columns = ['Targets']

# Apply KMeans clustering

model = KMeans(n_clusters=3)
model.fit(X)

# Plotting
plt.figure(figsize=(14, 7))

colormap = np.array(['red', 'lime', 'black'])

# Plot the Original Classifications

plt.subplot(1, 2, 1)
plt.scatter(X.Petal_Length, X.Petal_Width, c=colormap[y.Targets], s=40)
plt.title('Real Classification')
plt.xlabel('Petal Length')
plt.ylabel('Petal Width')

# Plot the Model's Classifications

plt.subplot(1, 2, 2)
plt.scatter(X.Petal_Length, X.Petal_Width, c=colormap[model.labels_], s=40)
plt.title('K Means Classification')
plt.xlabel('Petal Length')
plt.ylabel('Petal Width')

# Print the accuracy score and confusion matrix for KMeans

print('The accuracy score of K-Mean:', sm.accuracy_score(y, model.labels_))
print('The Confusion matrix of K-Mean:', sm.confusion_matrix(y, model.labels_))

# Standardize the features

scaler = preprocessing.StandardScaler()
scaler.fit(X)
xsa = scaler.transform(X)
xs = pd.DataFrame(xsa, columns=X.columns)

# Apply Gaussian Mixture Model (GMM)

gmm = GaussianMixture(n_components=3)
gmm.fit(xs)

y_gmm = gmm.predict(xs)

# Plot the GMM Classifications

plt.subplot(2, 2, 3)
plt.scatter(X.Petal_Length, X.Petal_Width, c=colormap[y_gmm], s=40)
plt.title('GMM Classification')
plt.xlabel('Petal Length')
plt.ylabel('Petal Width')

# Print the accuracy score and confusion matrix for GMM

print('The accuracy score of EM:', sm.accuracy_score(y, y_gmm))
print('The Confusion matrix of EM:', sm.confusion_matrix(y, y_gmm))

# Show the plots

plt.tight_layout()
plt.show()

Output:-
The accuracy score of K-Mean: 0.09333333333333334
The Confusion matrixof K-Mean: [[ 0 50 0]
[ 2 0 48]
[36 0 14]]
The accuracy score of EM: 0.9666666666666667
The Confusion matrix of EM: [[50 0 0]
[ 0 45 5]
[ 0 0 50]]
 Experiment-9
Write a program to implement k-Nearest Neighbour algorithm to classify the iris data set.
Print both correct and wrong predictions. Java/Python ML library classes can be used for this
problem.

Data Set:

Iris Plants Dataset: Dataset contains 150 instances (50 in each of three classes)
Number of Attributes: 4 numeric, predictive attributes and the Class

Program:

from sklearn.model_selection import train_test_split

from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import classification_report, confusion_matrix
from sklearn import datasets

# Load the Iris dataset

iris = datasets.load_iris()

# Features and target

x = iris.data
y = iris.target

print('sepal-length', 'sepal-width', 'petal-length', 'petal-width')

print(x)
print('class: 0-Iris-Setosa, 1- Iris-Versicolour, 2- Iris-Virginica')
print(y)

# Split the dataset into training and testing sets

x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.3)

# Training the model with K=5 nearest neighbors

 classifier = KNeighborsClassifier(n_neighbors=5)
classifier.fit(x_train, y_train)

# Make predictions on the test data

y_pred = classifier.predict(x_test)

# Display confusion matrix and accuracy metrics

print('Confusion Matrix')
print(confusion_matrix(y_test, y_pred))
print('Accuracy Metrics')
print(classification_report(y_test, y_pred))

Output:

sepal-length sepal-width petal-length petal-width

[[5.1 3.5 1.4 0.2]
[4.9 3. 1.4 0.2]
[4.7 3.2 1.3 0.2]
[4.6 3.1 1.5 0.2]
[5. 3.6 1.4 0.2]
. . . . .
. . . . .
[6.2 3.4 5.4 2.3]
[5.9 3.5.1 1.8]]
class: 0-Iris-Setosa, 1- Iris-Versicolour, 2- Iris-Virginica
[0 0 0 ………0 0 1 1 1 …………1 1 2 2 2 ………… 2 2]Confusion

Matrix
[[20 0 0]
[ 0 10 0]
[ 0 1 14]]

Accuracy Metrics

Precision recall f1-score support

0 1.00 1.00 1.00 20

1 0.91 1.00 0.95 10
2 1.00 0.93 0.97 15

avg / total 0.98 0.98 0.98 45

 Experiment-10
Implement the non-parametric Locally Weighted Regression algorithm in order to fit data
points. Select appropriate data set for your experiment and draw graphs.

Program:-
import numpy as np
from bokeh.plotting import figure, show, output_notebook
from bokeh.layouts import gridplot
from bokeh.io import push_notebook

# Local Regression Function

def local_regression(x0, X, Y, tau):
# Add bias term
x0 = np.r_[1, x0] # Add one to avoid the loss in information
X = np.c_[np.ones(len(X)), X]

# Fit model: normal equations with kernel

xw = X.T * radial_kernel(x0, X, tau) # XTranspose * W
beta = np.linalg.pinv(xw @ X) @ xw @ Y # @ Matrix Multiplication or Dot Product

# Predict value
return x0 @ beta # @ Matrix Multiplication or Dot Product for prediction

# Radial Kernel Function

def radial_kernel(x0, X, tau):
return np.exp(np.sum((X - x0) ** 2, axis=1) / (-2 * tau * tau))

n = 1000
# Generate dataset
X = np.linspace(-3, 3, num=n)
print("The Data Set (10 Samples) X :\n", X[:10])
Y = np.log(np.abs(X ** 2 - 1) + .5)
print("The Fitting Curve Data Set (10 Samples) Y :\n", Y[:10])

# Jitter X
X += np.random.normal(scale=.1, size=n)
print("Normalized (10 Samples) X :\n", X[:10])
domain = np.linspace(-3, 3, num=300)
print("X0 Domain Space (10 Samples) :\n", domain[:10])

# Plotting function
def plot_lwr(tau):
# Prediction through regression
prediction = [local_regression(x0, X, Y, tau) for x0 in domain]
plot = figure(plot_width=400, plot_height=400)
plot.title.text = 'tau=%g' % tau
plot.scatter(X, Y, alpha=.3)
plot.line(domain, prediction, line_width=2, color='red')
return plot

# Show the plots in a grid layout

show(gridplot([
[plot_lwr(10.), plot_lwr(1.)],
[plot_lwr(0.1), plot_lwr(0.01)]
]))

# -*- coding: utf-8 -*-

"""
Spyder Editor

This is a temporary script file. """

from numpy import *
from os import listdir
import matplotlib
import matplotlib.pyplot as plt import
pandas as pd
import numpy as np1 import
numpy.linalg as np
from scipy.stats.stats import pearsonr

def kernel(point,xmat, k):

m,n = np1.shape(xmat)
weights = np1.mat(np1.eye((m))) for
 j in range(m):
diff = point - X[j]
weights[j,j] = np1.exp(diff*diff.T/(-2.0*k**2))
return weights

def localWeight(point,xmat,ymat,k):
wei = kernel(point,xmat,k)
W = (X.T*(wei*X)).I*(X.T*(wei*ymat.T)) return W

def localWeightRegression(xmat,ymat,k):
m,n = np1.shape(xmat)

ypred = np1.zeros(m) for

i in range(m):
ypred[i] = xmat[i]*localWeight(xmat[i],xmat,ymat,k) return
ypred

# load data points

data = pd.read_csv('tips.csv') bill
= np1.array(data.total_bill) tip =
np1.array(data.tip)

#preparing and add 1 in bill mbill

= np1.mat(bill)
mtip = np1.mat(tip) # mat is used to convert to n dimesiona to 2 dimensional array form m=
np1.shape(mbill)[1]
# print(m) 244 data is stored in m one =
np1.mat(np1.ones(m))
X= np1.hstack((one.T,mbill.T)) # create a stack of bill from ONE #print(X)
#set k here
ypred = localWeightRegression(X,mtip,0.3) SortIndex =
X[:,1].argsort(0)
xsort = X[SortIndex][:,0]
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.scatter(bill,tip, color='green')
ax.plot(xsort[:,1],ypred[SortIndex], color = 'red', linewidth=5) plt.xlabel('Total bill')
plt.ylabel('Tip') plt.show();

Output: