DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING

CS3491 & ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING

UNIT III SUPERVISED LEARNING

Introduction to machine learning – Linear Regression Models:

Least squares, single & multiple variables, Bayesian linear
regression, gradient descent, Linear Classification Models:
Discriminant function – Probabilistic discriminative model -
Logistic regression, Probabilistic generative model – Naive Bayes,
Maximum margin classifier – Support vector machine, Decision
Tree, Random forests.

Introduction to machine learning:

Machine Learning (ML) is a field of artificial intelligence (AI) that
focuses on the development of algorithms and models that enable
computers to learn from and make predictions or decisions based on data.
The primary goal of machine learning is to build systems that can
automatically improve and adapt their performance over time without
being explicitly programmed for specific tasks.

Key Concepts in Machine Learning:

1. Data:
 Data is the foundation of machine learning. Algorithms learn patterns
and make predictions or decisions based on the information
contained in the data.
2. Features and Labels:
 In supervised learning, the data is often split into features (input
variables) and labels (output variable or target). The model learns to
map features to labels.
3. Algorithms:
 Machine learning algorithms are mathematical models that learn
patterns from data. They can be categorized into supervised learning,
unsupervised learning, and reinforcement learning, depending on the
type of data and learning task.
  Supervised Learning: The algorithm is trained on a labeled dataset,
where the correct output is provided. It learns to map inputs to
corresponding outputs.
 Unsupervised Learning: The algorithm is given unlabeled data and
must find patterns or relationships without explicit guidance on the
output.
 Reinforcement Learning: The algorithm learns by interacting with
an environment and receiving feedback in the form of rewards or
penalties.
4. Training and Testing:
 Machine learning models are trained on a subset of the data and
then evaluated on another subset to assess their generalization to
new, unseen data.
5. Model Evaluation:
 Metrics such as accuracy, precision, recall, and F1 score are used to
evaluate the performance of machine learning models.
6. Overfitting and Underfitting:
 Overfitting occurs when a model performs well on training data but
poorly on new data. Underfitting happens when a model is too
simple to capture the underlying patterns in the data.
7. Hyperparameters:
 Parameters of the machine learning algorithm that are not learned
from the data but must be set before training. Tuning
hyperparameters is essential for optimizing model performance.

Machine Learning Workflow:

1. Data Collection:
 Gather relevant data for the problem at hand. The quality and
quantity of data are crucial for the success of machine learning
models.
2. Data Preprocessing:
 Clean and preprocess the data, handling missing values, encoding
categorical variables, and scaling features.
3. Feature Engineering:
 Select or create relevant features that contribute to the predictive
power of the model.
4. Model Selection:
  Choose an appropriate machine learning algorithm based on the
nature of the problem (classification, regression, clustering) and
characteristics of the data.
5. Model Training:
 Train the selected model on a training dataset, adjusting its
parameters to minimize the difference between predicted and actual
outcomes.
6. Model Evaluation:
 Assess the model's performance on a separate testing dataset to
understand its ability to generalize to new, unseen data.
7. Hyperparameter Tuning:
 Fine-tune the model by adjusting hyperparameters to improve
performance.
8. Deployment:
 Once satisfied with the model's performance, deploy it to make
predictions on new, real-world data.

Applications of Machine Learning:

1. Image and Speech Recognition:
 ML is used for recognizing and interpreting visual and auditory
information, such as image and speech recognition systems.
2. Natural Language Processing (NLP):
 ML powers applications like language translation, sentiment analysis,
and chatbots by understanding and generating human language.
3. Healthcare:
 ML is applied for medical image analysis, disease diagnosis, and
predicting patient outcomes based on health data.
4. Finance:
 ML models are used for credit scoring, fraud detection, algorithmic
trading, and personalized financial recommendations.
5. Recommendation Systems:
 Platforms like Netflix and Amazon use ML to suggest personalized
content and products based on user preferences.
6. Autonomous Vehicles:
 ML plays a crucial role in enabling self-driving cars by helping them
perceive and respond to the surrounding environment.
 Challenges in Machine Learning:
1. Data Quality:
 The quality of machine learning models heavily depends on the
quality and representativeness of the training data.
2. Interpretability:
 Some complex machine learning models, such as deep neural
networks, lack interpretability, making it challenging to understand
how they make decisions.
3. Bias and Fairness:
 Models can inherit biases present in the training data, leading to
unfair or discriminatory outcomes.
4. Scalability:
 Scaling machine learning algorithms to handle large datasets and
high-dimensional data is a continuous challenge.
5. Deployment Challenges:
 Deploying and maintaining machine learning models in real-world
environments require addressing issues related to integration,
monitoring, and updates.

Machine learning is a rapidly evolving field with widespread applications

and ongoing research to address challenges. It continues to revolutionize
industries and is an integral part of technological advancements in various
domains.

Linear Regression Models:

Linear regression is a statistical method used to model the
relationship between a dependent variable and one or more independent
variables by fitting a linear equation to the observed data. The basic form of
a simple linear regression equation with one independent variable is:

y=mx+b

Where:

 y is the dependent variable,

 x is the independent variable,
 m is the slope of the line, and
 b is the y-intercept.

In the context of multiple linear regression, where there are multiple

independent variables, the equation is extended to:

y=b0+b1x1+b2x2+…+bnxn

Where:

 y is the dependent variable,

 x1,x2,…,xn are the independent variables,
 0b0 is the y-intercept,
 b1,b2,…,bn are the coefficients associated with each independent variable.

The goal of linear regression is to find the values of the coefficients that
minimize the difference between the predicted values and the actual values.
This is typically done using the method of least squares.

Here are the basic steps involved in building a linear regression model:

1. Data Collection: Gather the data on the dependent and independent

variables.
2. Data Preparation: Clean and preprocess the data. This may involve
handling missing values, scaling features, and other data cleaning tasks.
3. Model Selection: Choose between simple linear regression (one
independent variable) or multiple linear regression (multiple independent
variables) based on the nature of your data.
4. Model Training: Use the training data to find the values of the coefficients
that minimize the sum of squared differences between predicted and actual
values.
5. Model Evaluation: Assess the performance of the model using metrics
such as mean squared error, R-squared, or others depending on the
context.
6. Prediction: Once the model is trained and evaluated, it can be used to
make predictions on new, unseen data.

Linear regression is widely used in various fields, including economics,

finance, biology, and engineering, among others. Despite its simplicity,
 linear regression can be a powerful tool for understanding and predicting
relationships between variables when certain assumptions are met.

Least squares, single & multiple variables, Bayesian linear

regression, gradient descent, Linear Classification Models:
Discriminant function.
linear regression and linear classification models. Let me provide a
brief overview of each of these topics:

1. Least Squares:
 Least Squares is a method used in linear regression to find the best-
fitting line through a set of points. It minimizes the sum of the
squares of the vertical distances (residuals) between the observed
and predicted values.
2. Single and Multiple Variables:
 Single variable linear regression involves predicting a dependent
variable based on one independent variable. Multiple variable linear
regression extends this to predicting the dependent variable based
on multiple independent variables.
3. Bayesian Linear Regression:
 Bayesian linear regression incorporates Bayesian statistics into the
linear regression model. It provides a probabilistic framework for
estimating parameters and making predictions, considering
uncertainty in the model.
4. Gradient Descent:
 Gradient Descent is an optimization algorithm used to minimize the
cost function in machine learning models. In the context of linear
regression, it adjusts the model parameters iteratively to find the
minimum of the cost function.
5. Linear Classification Models: Discriminant Function:
 Linear classification models aim to classify data points into different
classes based on linear decision boundaries. The discriminant
function is a mathematical function that combines input features to
make predictions about class membership.

Each of these concepts plays a role in understanding and building linear

regression models and linear classification models. For example, least
squares helps in finding the best-fitting line, multiple variables extend the
 model to handle more complex relationships, Bayesian linear regression
introduces a probabilistic framework, and gradient descent optimizes the
model parameters. Linear classification models, with their discriminant
functions, are used in problems where the goal is to classify data points
into different categories based on linear decision boundaries.

Probabilistic discriminative mode:

This generally involves building models for classification tasks that explicitly model the
probability distribution of class labels given the input data. One common example of such
models is logistic regression.

Here's a brief explanation:

1. Logistic Regression as a Probabilistic Discriminative Model:

 In logistic regression, the goal is to model the probability that a given input belongs to a
particular class. The logistic function (sigmoid function) is often used to map the linear
combination of input features to a value between 0 and 1, representing the probability.
 The logistic regression model can be expressed as: P(Y=1∣X)=1+e−(β0+β1X1+…+βnXn)1 where
P(Y=1∣X) is the probability of the positive class given input X, and 0,β0,β1,…,βn are the
model parameters.
2. Discriminative vs. Generative Models:
 Logistic regression is a discriminative model because it directly models the decision
boundary between different classes. In contrast, generative models aim to model the joint
distribution of both the input features and the class labels.
3. Training and Inference:
 During training, the model's parameters (β values) are learned from the training data by
optimizing a suitable loss function, often using methods like maximum likelihood
estimation.
 During inference, the trained model can be used to predict the probability of a new
instance belonging to a particular class. The decision boundary is typically set at a
threshold (e.g., 0.5 for binary classification), and predictions are made based on whether
the estimated probability exceeds the threshold.
4. Extensions:
 Probabilistic discriminative models can be extended to handle multiple classes, and
regularization techniques can be applied to prevent overfitting.

These models are widely used in various applications, including binary and multiclass
classification problems, where understanding the uncertainty associated with predictions is
valuable.

Logistic regression:
Logistic regression is a statistical method used for binary and multi-
class classification problems. Despite its name, it is a classification algorithm
rather than a regression algorithm. The primary purpose of logistic
 regression is to model the probability of an instance belonging to a
particular class.

Here are the key points about logistic regression:

1. Model Formulation:
 Logistic regression models the probability that an instance belongs to
a particular class using the logistic function (sigmoid function). The
logistic function is defined as: P(Y=1∣X)=1+e−(β0+β1X1+…+βnXn)1 where:
 P(Y=1∣X) is the probability of the positive class given input X.
 X1,X2,…,Xn are the input features.
 β0,β1,…,βn are the model parameters or coefficients.
2. Training:
 The model parameters ( β values) are learned from the training data
by maximizing the likelihood function. This is typically done using
optimization algorithms such as gradient descent.
3. Decision Boundary:
 The decision boundary is set based on a threshold probability
(commonly 0.5 for binary classification). If the predicted probability is
above the threshold, the instance is classified as belonging to the
positive class; otherwise, it is classified as belonging to the negative
class.
4. Binary and Multi-class Classification:
 Logistic regression can be used for binary classification problems
(two classes) or extended to handle multi-class problems through
techniques like one-vs-all or one-vs-one.
5. Regularization:
 Regularization terms (e.g., L1 or L2 regularization) can be added to
the logistic regression model to prevent overfitting.
6. Interpretability:
 The coefficients ( β values) of logistic regression can be interpreted to
understand the impact of each feature on the probability of
belonging to a particular class.

Logistic regression is widely used in various fields, such as medicine,

finance, and social sciences, for tasks like predicting whether an email is
spam or not, assessing the risk of a disease, or determining the likelihood
 of customer churn. It is a fundamental and interpretable algorithm in the
field of machine learning and statistics.

Probabilistic generative model:

A probabilistic generative model is a type of statistical model that explicitly
represents the joint probability distribution of both the observed data and
the corresponding labels or classes. Generative models aim to capture how
the data is generated, allowing for the synthesis of new samples from the
learned distribution.

Here are some key characteristics and concepts related to probabilistic

generative models:

1. Generative Modeling:
 Generative models focus on modeling the entire distribution of the
data, including the relationship between input features and output
labels. This is in contrast to discriminative models, which model the
conditional probability of labels given the input data.
2. Joint Probability Distribution:
 In a probabilistic generative model, the joint probability distribution
P(X,Y) is explicitly modeled, where X is the input data and Y is the
corresponding labels or classes. This enables the model to generate
new samples by sampling from the joint distribution.
3. Examples of Generative Models:
 Gaussian Mixture Models (GMM): GMM is a generative model that
represents the data as a mixture of several Gaussian distributions. It is
often used for clustering and density estimation.
 Hidden Markov Models (HMM): HMM is a generative model
commonly used for modeling time-series data with hidden states.
 Naive Bayes Classifier: Although often used as a classifier, Naive
Bayes can be seen as a generative model. It models the joint
probability of features and labels using Bayes' theorem.
 Variational Autoencoders (VAEs): VAEs are a type of generative
model that combines ideas from neural networks and probabilistic
modeling. They learn a latent representation of the data and can
generate new samples.
4. Sampling from the Model:
  Once trained, a generative model allows for the generation of new
samples by sampling from the learned joint distribution. This is
valuable for tasks such as data synthesis, image generation, and
anomaly detection.
5. Challenges:
 Training generative models can be computationally intensive, and
accurately modeling complex distributions may require sophisticated
techniques.

Generative models are versatile and find applications in various domains,

including image generation, data synthesis, natural language processing,
and more. They provide a probabilistic framework for understanding the
underlying structure of data and are valuable for tasks that involve
generating new, realistic samples.

Naive Bayes:
Naive Bayes is a probabilistic classification algorithm based on Bayes' theorem. Despite
its simplicity and the "naive" assumption of feature independence, it has proven to be quite
effective in various practical applications. The algorithm is particularly popular for text
classification tasks, such as spam filtering and sentiment analysis.

Here are the key aspects of Naive Bayes:

1. Bayes' Theorem:
 Naive Bayes is based on Bayes' theorem, which relates the conditional and marginal
probabilities of random events. For a classification task, it can be expressed as:
P(Y∣X)=P(X)P(X∣Y)⋅P(Y) where:
 P(Y∣X) is the probability of class Y given the input features X,
 P(X∣Y) is the probability of observing the input features X given class Y,
 P(Y) is the prior probability of class Y, and
 P(X) is the probability of observing the input features X.
2. Naive Assumption:
 The "naive" assumption in Naive Bayes is that the features are conditionally independent
given the class. This simplifies the computation and makes the algorithm computationally
efficient, especially for high-dimensional data.
3. Types of Naive Bayes:
 There are different types of Naive Bayes classifiers, depending on the nature of the input
features and the distribution of the data. Common variants include:
 Gaussian Naive Bayes: Assumes that the features follow a Gaussian (normal)
distribution.
 Multinomial Naive Bayes: Suitable for discrete features, often used in text
classification.
 Bernoulli Naive Bayes: Suitable for binary features.
4. Training:
  The model is trained using a labeled dataset. The prior probabilities ( P(Y) ) and the
likelihoods (P(X∣Y)) are estimated from the training data.
5. Classification:
 To classify a new instance, the algorithm computes the posterior probabilities ( P(Y∣X) )
for each class and selects the class with the highest probability.
6. Applications:
 Naive Bayes is commonly used in text classification tasks, such as spam filtering and
sentiment analysis. It's also applied in areas like medical diagnosis and recommendation
systems.
7. Strengths and Weaknesses:
 Naive Bayes is simple, easy to implement, and computationally efficient. However, its
performance may suffer when the independence assumption is significantly violated or
when intricate dependencies between features exist.

Naive Bayes classifiers are particularly effective in scenarios where the features can be assumed
to be conditionally independent given the class. Despite the simplicity of the algorithm, it often
performs well and can be a good choice for certain types of classification tasks .

Maximum margin classifier:

The maximum margin classifier is a concept associated with support vector
machines (SVMs), a popular machine learning algorithm for classification
and regression tasks. The goal of the maximum margin classifier is to find
the hyperplane that maximizes the margin between different classes in the
feature space.

Here are the key points related to the maximum margin classifier:

1. Support Vector Machines (SVM):

 SVM is a supervised learning algorithm that can be used for both
classification and regression tasks. In classification, SVM aims to find
a hyperplane that best separates the data into different classes.
2. Hyperplane:
 In a two-dimensional space, a hyperplane is a line. In higher
dimensions, it is a flat subspace. For a binary classification problem,
the hyperplane is the decision boundary that separates the instances
of different classes.
3. Margin:
 The margin is the distance between the hyperplane and the nearest
data point from either class. The goal of the maximum margin
classifier is to find the hyperplane that maximizes this margin.
4. Support Vectors:
  Support vectors are the data points that lie closest to the hyperplane
and are crucial in defining the margin. These are the instances that
are most challenging to classify.
5. Optimization Objective:
 The objective of SVM is to maximize the margin while minimizing the
classification error. This is often formulated as a constrained
optimization problem.
6. Soft Margin SVM:
 In cases where the data is not perfectly separable by a hyperplane, a
"soft margin" SVM allows for some misclassifications. This is achieved
by introducing a regularization parameter that controls the trade-off
between maximizing the margin and minimizing the classification
error.
7. Kernel Trick:
 SVM can handle non-linear decision boundaries through the use of
kernel functions. These functions allow the algorithm to implicitly
map the input features into a higher-dimensional space where a
hyperplane can separate the classes.
8. C-Support Vector Classification (C-SVC):
 C-SVC is a variant of SVM where the parameter C determines the
trade-off between achieving a smooth decision boundary and
classifying training points correctly. A smaller C encourages a larger
margin but may allow more misclassifications.

The maximum margin classifier in SVM aims to create a decision boundary

that not only separates classes but does so with the maximum possible
margin, providing robustness to variations in the data. SVMs are particularly
effective in high-dimensional spaces and are widely used in various
applications, including image classification, text categorization, and
bioinformatics.

Support vector machine:

A Support Vector Machine (SVM) is a supervised machine learning
algorithm used for classification and regression tasks. The primary goal of
an SVM is to find the optimal hyperplane that separates data points of
different classes in the feature space while maximizing the margin between
the classes. SVMs are particularly effective in high-dimensional spaces and
are widely used in various applications.
 Here are the key concepts associated with Support Vector Machines:

1. Hyperplane:
 In a binary classification problem, a hyperplane is a decision
boundary that separates instances of one class from another. In two
dimensions, a hyperplane is a line, and in three dimensions, it is a
plane. In higher dimensions, it is a flat subspace.
2. Margin:
 The margin is the distance between the hyperplane and the nearest
data point from either class. SVM aims to find the hyperplane that
maximizes this margin. The margin is important because it provides a
measure of robustness to variations in the data.
3. Support Vectors:
 Support vectors are the data points that lie closest to the hyperplane.
These are the critical instances that influence the position and
orientation of the hyperplane. SVM is named after these support
vectors.
4. Linear and Non-linear SVM:
 SVM can be used with linear and non-linear kernels. In a linear SVM, a
straight-line hyperplane is used to separate classes. Non-linear SVMs
use kernel functions (e.g., polynomial, radial basis function) to map
the input features into a higher-dimensional space where a
hyperplane can be used for separation.
5. Soft Margin SVM:
 In cases where the data is not perfectly separable, a "soft margin"
SVM allows for some misclassifications. The regularization parameter
(C) controls the trade-off between maximizing the margin and
minimizing the classification error.
6. C-Support Vector Classification (C-SVC):
 C-SVC is a variant of SVM where the parameter C determines the
trade-off between achieving a smooth decision boundary and
classifying training points correctly. A smaller C encourages a larger
margin but may allow more misclassifications.
7. Regression with SVM (SVR):
 SVM can also be used for regression tasks. In this case, the goal is to
find a hyperplane that best fits the data while minimizing deviations
from the true values.
8. Applications:
  SVMs are used in a wide range of applications, including image
classification, text categorization, handwriting recognition,
bioinformatics, and more.

Support Vector Machines are known for their ability to handle high-
dimensional data and generalize well in various domains. They have
become a popular choice for classification tasks, particularly when the data
exhibits complex patterns and non-linear relationships.

Decision Tree:
A Decision Tree is a supervised machine learning algorithm used for both
classification and regression tasks. It is a tree-like model where each
internal node represents a decision based on the value of a particular
feature, each branch represents the outcome of that decision, and each leaf
node represents the final decision or the predicted outcome.

Here are the key characteristics and concepts related to Decision Trees:

1. Node Types:
 Root Node: The topmost node in the tree, representing the initial
decision.
 Internal Nodes: Nodes that make decisions based on the values of
specific features.
 Leaf Nodes: Terminal nodes that provide the final output or
prediction.
2. Decision Criteria:
 At each internal node, a decision is made based on the value of a
specific feature. The goal is to choose features that provide the best
separation of the data into classes (for classification) or minimize the
variance (for regression).
3. Splitting:
 The process of dividing a node into sub-nodes based on a decision
criterion. It involves selecting the best feature and a threshold value
to make the split.
4. Categorical and Continuous Features:
 Decision Trees can handle both categorical and continuous features.
The splitting process for categorical features involves creating
 branches for each category, while for continuous features, it involves
finding an optimal threshold.
5. Decision Making:
 As data moves through the tree, decisions are made at each internal
node, guiding the path to the leaf nodes, where the final predictions
are made.
6. Information Gain (Entropy):
 For classification tasks, Decision Trees often use information gain or
entropy to determine the best feature for splitting. The goal is to
choose features that result in the most homogenous subsets of data
in terms of class labels.
7. Gini Impurity:
 Another metric used for classification is Gini impurity, which measures
the likelihood of misclassifying a randomly chosen element. The
feature that minimizes the Gini impurity is selected for splitting.
8. Pruning:
 Pruning is a technique used to avoid overfitting by removing nodes
that do not significantly contribute to the predictive power of the
tree. Pruning helps improve the model's generalization to new,
unseen data.
9. Ensemble Methods:
 Decision Trees can be part of ensemble methods, such as Random
Forests and Gradient Boosted Trees, where multiple trees are
combined to improve overall performance and robustness.
10. Regression Trees:
 In regression tasks, Decision Trees predict a continuous target
variable. The prediction at each leaf node is typically the mean or
median of the target variable for the data points in that node.

Decision Trees are intuitive, easy to understand, and can capture complex
relationships in the data. However, they are prone to overfitting, especially
on noisy datasets. Techniques like pruning and ensemble methods help
address this limitation and enhance the robustness of Decision Trees in
various machine learning applications.
 Random forests:
Random Forest is an ensemble learning method based on the construction
of a multitude of decision trees at training time and outputting the class
that is the mode of the classes (classification) or mean prediction
(regression) of the individual trees. It is one of the most popular and
versatile machine learning algorithms.

Here are the key concepts associated with Random Forest:

1. Ensemble Learning:
 Random Forest belongs to the ensemble learning family, which
combines the predictions of multiple models to improve overall
performance and robustness.
2. Decision Trees as Base Learners:
 Random Forest builds multiple decision trees during training. Each
tree is trained on a different subset of the training data and may use
a random subset of features for each split.
3. Bootstrapped Sampling (Bagging):
 Each tree in the Random Forest is trained on a random subset of the
original dataset. This is done by sampling with replacement, a process
known as bootstrapped sampling. It helps create diversity among the
trees.
4. Feature Randomization:
 During the construction of each decision tree, only a random subset
of features is considered at each split. This further increases the
diversity among the trees and helps prevent overfitting.
5. Voting (Classification) or Averaging (Regression):
 For classification tasks, the class that receives the most votes from
individual trees is the final prediction. For regression tasks, the final
prediction is the average of the predictions made by individual trees.
6. Out-of-Bag (OOB) Error Estimation:
 Since each tree is trained on a subset of the data, the remaining data
(out-of-bag samples) can be used to estimate the performance of the
Random Forest without the need for a separate validation set.
7. Versatility and Robustness:
 Random Forests are versatile and can handle both classification and
regression problems. They are resistant to overfitting and tend to
perform well on a variety of datasets.
8. Feature Importance:
 Random Forests can provide a measure of feature importance based
on how much each feature contributes to the accuracy of the model.
This information can be useful for feature selection.
9. Parallelization:
 The construction of individual trees in a Random Forest can be
parallelized, making it computationally efficient, especially for large
datasets.
10. Applications:
 Random Forests are used in a wide range of applications, including
image classification, text classification, bioinformatics, and finance.

Random Forests have become a popular choice for many machine learning
tasks due to their robustness, ease of use, and ability to handle complex
datasets. They are particularly useful when dealing with high-dimensional
data and situations where overfitting is a concern.