AI Lab Assignment 08

Classification Workflow
Whenever you perform classification, the first step is to understand the problem and
identify potential features and label. Features are those characteristics or attributes
which affect the results of the label. For example, in the case of a loan distribution,
bank managers identify the customer’s occupation, income, age, location, previous
loan history, transaction history, and credit score. These characteristics are known
as features that help the model classify customers.

The classification has two phases, a learning phase and the evaluation phase. In the
learning phase, the classifier trains its model on a given dataset, and in the
evaluation phase, it tests the classifier's performance. Performance is evaluated on
the basis of various parameters such as accuracy, error, precision, and recall.

What is Naive Bayes Classifier?

Naive Bayes is a statistical classification technique based on Bayes Theorem. It is
one of the simplest supervised learning algorithms. Naive Bayes classifier is the fast,
accurate and reliable algorithm. Naive Bayes classifiers have high accuracy and
speed on large datasets.

Naive Bayes classifier assumes that the effect of a particular feature in a class is
independent of other features. For example, a loan applicant is desirable or not
depending on his/her income, previous loan and transaction history, age, and
location. Even if these features are interdependent, these features are still
considered independently. This assumption simplifies computation, and that's why it
is considered as naive. This assumption is called class conditional independence.
 P(h): the probability of hypothesis h being true (regardless of the data).
This is known as the prior probability of h.
 P(D): the probability of the data (regardless of the hypothesis). This is
known as the prior probability.
 P(h|D): the probability of hypothesis h given the data D. This is known as
posterior probability.
 P(D|h): the probability of data d given that the hypothesis h was true. This
is known as posterior probability.

How Naive Bayes Classifier Works?

Let’s understand the working of Naive Bayes through an example. Given an example
of weather conditions and playing sports. You need to calculate the probability of
playing sports. Now, you need to classify whether players will play or not, based on
the weather condition.

First Approach (In case of a single feature)

Naive Bayes classifier calculates the probability of an event in the following steps:

 Step 1: Calculate the prior probability for given class labels

 Step 2: Find Likelihood probability with each attribute for each class
 Step 3: Put these value in Bayes Formula and calculate posterior
probability.
 Step 4: See which class has a higher probability, given the input belongs to
the higher probability class.
For simplifying prior and posterior probability calculation, you can use the two tables
frequency and likelihood tables. Both of these tables will help you to calculate the prior
and posterior probability. The Frequency table contains the occurrence of labels for all
features. There are two likelihood tables. Likelihood Table 1 is showing prior
probabilities of labels and Likelihood Table 2 is showing the posterior probability.
 Now suppose you want to calculate the probability of playing when the weather is
overcast.

Probability of playing:

P(Yes | Overcast) = P(Overcast | Yes) P(Yes) / P (Overcast) .....................(1)

1. Calculate Prior Probabilities:

P(Overcast) = 4/14 = 0.29

P(Yes)= 9/14 = 0.64

1. Calculate Posterior Probabilities:

P(Overcast |Yes) = 4/9 = 0.44

1. Put Prior and Posterior probabilities in equation (1)

P (Yes | Overcast) = 0.44 * 0.64 / 0.29 = 0.98(Higher)

Similarly, you can calculate the probability of not playing:

Probability of not playing:

P(No | Overcast) = P(Overcast | No) P(No) / P (Overcast) .....................(2)

1. Calculate Prior Probabilities:

P(Overcast) = 4/14 = 0.29

P(No)= 5/14 = 0.36

1. Calculate Posterior Probabilities:

P(Overcast |No) = 0/9 = 0

1. Put Prior and Posterior probabilities in equation (2)

P (No | Overcast) = 0 * 0.36 / 0.29 = 0

The probability of a 'Yes' class is higher. So you can determine here if the weather is overcast
than players will play the sport.

Second Approach (In case of multiple features)

Now suppose you want to calculate the probability of playing when the weather is overcast,
and the temperature is mild.
 Probability of playing:

P(Play= Yes | Weather=Overcast, Temp=Mild) = P(Weather=Overcast, Temp=Mild | Play=

Yes)P(Play=Yes) ..........(1)

P(Weather=Overcast, Temp=Mild | Play= Yes)= P(Overcast |Yes) P(Mild |Yes) ………..(2)

1. Calculate Prior Probabilities: P(Yes)= 9/14 = 0.64

2. Calculate Posterior Probabilities: P(Overcast |Yes) = 4/9 = 0.44 P(Mild |Yes) = 4/9
= 0.44
3. Put Posterior probabilities in equation (2) P(Weather=Overcast, Temp=Mild |
Play= Yes) = 0.44 * 0.44 = 0.1936(Higher)
4. Put Prior and Posterior probabilities in equation (1) P(Play= Yes |
Weather=Overcast, Temp=Mild) = 0.1936*0.64 = 0.124

Similarly, you can calculate the probability of not playing:

Probability of not playing:

P(Play= No | Weather=Overcast, Temp=Mild) = P(Weather=Overcast, Temp=Mild | Play=

No)P(Play=No) ..........(3)

P(Weather=Overcast, Temp=Mild | Play= No)= P(Weather=Overcast

|Play=No) P(Temp=Mild | Play=No) ………..(4)

1. Calculate Prior Probabilities: P(No)= 5/14 = 0.36

2. Calculate Posterior Probabilities: P(Weather=Overcast |Play=No) = 0/9 = 0
P(Temp=Mild | Play=No)=2/5=0.4
3. Put posterior probabilities in equation (4) P(Weather=Overcast, Temp=Mild |
Play= No) = 0 * 0.4= 0
4. Put prior and posterior probabilities in equation (3) P(Play= No |
Weather=Overcast, Temp=Mild) = 0*0.36=0

The probability of a 'Yes' class is higher. So you can say here that if the weather is overcast
than players will play the sport.
 Classifier Building in Scikit-learn
Naive Bayes Classifier with Loan Dataset
Let’s train the Naive Bayes Classifier on the real dataset. We will be repeating most of the
tasks except for preprocessing and data exploration.

Data Loading

In this example, we will be loading Loan Data (Loan Data | DataCamp Workspace) from
DataCamp Workspace using pandas 'read_csv` function.

import pandas as pd

df = pd.read_csv('loan_data.csv')

df.head()

Data Exploration

To understand more about the dataset we will use `.info()`.

 The dataset consists of 14 columns and 9578 rows.

 Apart from “purpose”, columns are either floats or integers.
 Our target column is “not.fully.paid”.
df.info()

RangeIndex: 9578 entries, 0 to 9577

Data columns (total 14 columns):

# Column Non-Null Count Dtype

--- ------ -------------- -----

0 credit.policy 9578 non-null int64

1 purpose 9578 non-null object

2 int.rate 9578 non-null float64

3 installment 9578 non-null float64

4 log.annual.inc 9578 non-null float64

5 dti 9578 non-null float64

6 fico 9578 non-null int64

7 days.with.cr.line 9578 non-null float64

8 revol.bal 9578 non-null int64

9 revol.util 9578 non-null float64

10 inq.last.6mths 9578 non-null int64

11 delinq.2yrs 9578 non-null int64

12 pub.rec 9578 non-null int64

13 not.fully.paid 9578 non-null int64

dtypes: float64(6), int64(7), object(1)

memory usage: 1.0+ MB

POWER ED B Y DA TACAMP WORK SPAC E COPY CODE

In this example, we will be developing a model to predict the customers who have not fully
paid the loan. Let’s explore the purpose and target column by using seaborn’s countplot.

import seaborn as sns

import matplotlib.pyplot as plt

sns.countplot(data=df,x='purpose',hue='not.fully.paid')

plt.xticks(rotation=45, ha='right');
Our dataset is an imbalance that will affect the performance of the model. You can check
out Resample an Imbalanced Dataset tutorial to get hands-on experience in handling
imbalanced datasets.

Data Processing

We will now convert the ‘purpose’ column from categorical to integer using pandas
`get_dummies` function.

pre_df = pd.get_dummies(df,columns=['purpose'],drop_first=True)

pre_df.head()

After that, we will define feature (X) and target (y) variables, and split the dataset into
training and testing sets.

from sklearn.model_selection import train_test_split

X = pre_df.drop('not.fully.paid', axis=1)

y = pre_df['not.fully.paid']

X_train, X_test, y_train, y_test = train_test_split(

X, y, test_size=0.33, random_state=125

Model Building and Training

Model building and training is quite simple. We will be training a model on a training dataset
using default hyperparameters.

from sklearn.naive_bayes import GaussianNB

model = GaussianNB()
model.fit(X_train, y_train);

Model Evaluation

We will use accuracy and f1 score to determine model performance, and it looks like the
Gaussian Naive Bayes algorithm has performed quite well.

from sklearn.metrics import (

accuracy_score,

confusion_matrix,

ConfusionMatrixDisplay,

f1_score,

classification_report,

y_pred = model.predict(X_test)

accuray = accuracy_score(y_pred, y_test)

f1 = f1_score(y_pred, y_test, average="weighted")

print("Accuracy:", accuray)

print("F1 Score:", f1)

Accuracy: 0.8206263840556786

F1 Score: 0.8686606980013266
Due to the imbalanced nature of the data, we can see that the confusion matrix tells a
different story. On a minority target: `not fully paid`, we have more mislabeled.

labels = ["Fully Paid", "Not fully Paid"]

cm = confusion_matrix(y_test, y_pred)

disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=labels)

disp.plot();

Zero Probability Problem

Suppose there is no tuple for a risky loan in the dataset; in this scenario, the posterior
probability will be zero, and the model is unable to make a prediction. This problem is known
as Zero Probability because the occurrence of the particular class is zero.

The solution for such an issue is the Laplacian correction or Laplace Transformation.
Laplacian correction is one of the smoothing techniques. Here, you can assume that the
dataset is large enough that adding one row of each class will not make a difference in the
estimated probability. This will overcome the issue of probability values to zero.

For Example: Suppose that for the class loan risky, there are 1000 training tuples in the
database. In this database, the income column has 0 tuples for low income, 990 tuples for
medium income, and 10 tuples for high income. The probabilities of these events, without the
Laplacian correction, are 0, 0.990 (from 990/1000), and 0.010 (from 10/1000)

Now, apply Laplacian correction on the given dataset. Let's add 1 more tuple for each
income-value pair. The probabilities of these events: