Intelligent Computing

Unit – 3
Introduction to AI
Artificial Intelligence (AI) refers to the ability of machines or software systems to mimic human
intelligence and perform tasks that would typically require human cognition. These tasks can
include reasoning, decision-making, learning, problem-solving, speech recognition, and visual
perception.

Key Areas of AI:

1. Machine Learning (ML): Machine Learning is a subset of AI that focuses on building
algorithms that allow machines to learn from data and make predictions or decisions
without being explicitly programmed. ML can be categorized into:
o Supervised Learning: Where models are trained on labeled data.
o Unsupervised Learning: Where models find hidden patterns or groupings in data
without predefined labels.
o Reinforcement Learning: Where agents learn by interacting with an
environment and receiving feedback through rewards or punishments.
 2. Natural Language Processing (NLP): NLP involves the interaction between computers
and human language. It aims to enable machines to understand, interpret, and respond to
human language.
3. Computer Vision: Computer Vision focuses on enabling machines to interpret and
understand visual information from the world. It allows systems to recognize objects,
people, or even interpret images and videos.
4. Robotics: AI-powered robotics combines the principles of AI and robotics to create
autonomous systems capable of performing tasks in the real world.
5. Expert Systems: These are AI systems that emulate the decision-making abilities of a
human expert in specific domains.
6. AI Ethics and Safety: As AI becomes more integrated into society, ethical considerations
regarding its use, safety, and impact on jobs and human rights have become essential.
Questions of fairness, transparency, accountability, and bias in AI systems are actively
discussed.
Applications of AI:
1. Healthcare: AI is used in diagnostic tools, drug discovery, and personalized medicine,
helping healthcare professionals make faster and more accurate decisions.
2. Autonomous Vehicles: AI is a key technology behind self-driving cars, using sensors,
cameras, and machine learning to navigate and make decisions in real-time.
3. Finance: AI algorithms help in fraud detection, risk assessment, and algorithmic trading,
improving the accuracy and efficiency of financial operations.
4. Entertainment: AI is used in recommendation systems on platforms like Netflix,
Spotify, and YouTube to suggest content based on user preferences.
5. Manufacturing: AI-powered robots are used for automation, predictive maintenance,
and optimizing production lines.

AI Problems and Solution Approaches

AI problems span a broad range of topics, from basic algorithmic challenges to complex, real-
world applications.
Here are some key AI problems and their common solution approaches:
1. Classification Problems
 Problem: Categorizing data into predefined classes (e.g., email spam detection, medical
diagnosis).
  Solution Approaches:
o Supervised Learning: Algorithms like Decision Trees, Support Vector Machines
(SVM), k-Nearest Neighbors (k-NN), and Naive Bayes.
o Deep Learning: Convolutional Neural Networks (CNNs) for image classification
and Recurrent Neural Networks (RNNs) for sequence data like text.
o Ensemble Methods: Boosting, Bagging, and Random Forests improve
classification performance.
2. Regression Problems
 Problem: Predicting continuous values (e.g., house price prediction, stock price
prediction).
 Solution Approaches:
o Linear Regression: Simple approach when the relationship between variables is
linear.
o Polynomial Regression: Extends linear regression to model nonlinear
relationships.
o Ridge/Lasso Regression: Regularized regression techniques to avoid overfitting.
o Deep Learning: Neural networks can model complex non-linear relationships for
high-dimensional data.
3. Clustering Problems
 Problem: Grouping similar data points without labels (e.g., customer segmentation,
document clustering).
 Solution Approaches:
o K-Means Clustering: One of the most popular unsupervised learning algorithms.
o Hierarchical Clustering: Builds a tree-like structure for data, useful for
hierarchical groupings.
o DBSCAN: Density-Based Spatial Clustering of Applications with Noise, effective
in identifying arbitrarily shaped clusters and noise.
4. Optimization Problems
 Problem: Finding the best solution from a set of possible solutions (e.g., route
optimization, portfolio optimization).
 Solution Approaches:
 o Gradient Descent: An iterative method to minimize a cost function, widely used
in machine learning models.
o Genetic Algorithms: Used for searching large and complex spaces, especially in
combinatorial optimization.
o Simulated Annealing: A probabilistic technique to approximate the global
optimum of a given function.
5. Natural Language Processing (NLP)
 Problem: Understanding, interpreting, and generating human language (e.g., sentiment
analysis, machine translation).
 Solution Approaches:
o Tokenization & Vectorization: Techniques like Bag of Words, TF-IDF for
converting text into numerical form.
o Recurrent Neural Networks (RNNs): Particularly for tasks with sequential data
like text or speech.
6. Reinforcement Learning
 Problem: Teaching agents to make a series of decisions to maximize cumulative reward
(e.g., robotic control, game-playing AI).
 Solution Approaches:
o Q-Learning: A model-free algorithm for learning the value of actions in a given
state.
o Deep Q-Networks (DQN): Integrates deep learning with Q-learning for complex
environments.
7. Anomaly Detection
 Problem: Identifying unusual patterns or outliers in data (e.g., fraud detection, network
security).
 Solution Approaches:
o Statistical Methods: Z-scores or Grubbs' test for identifying anomalies.
o Isolation Forest: A tree-based method that isolates anomalies by randomly
partitioning the data.
8. Generative Models
 Problem: Generating new data similar to a given set (e.g., creating new images, music, or
text).
  Solution Approaches:
o Generative Adversarial Networks (GANs): A class of deep learning models
where two neural networks (generator and discriminator) compete to produce
realistic data.
o Variational Autoencoders (VAEs): A generative model for creating new data by
learning the probability distribution of the training data.

Fundamentals of Problem-Solving Using Search and

Heuristics
Fundamentals of Problem-Solving Using Search and Heuristics
Problem-solving is a key aspect of artificial intelligence (AI) and computer science. It involves
finding a solution to a problem by systematically searching through potential solutions and
applying appropriate heuristics. Let’s break down the core concepts:
1. Problem Representation
To solve a problem using search, the first step is to represent the problem clearly:
 State: A configuration of the problem at a specific point in time.
 Initial state: The starting point or configuration.
 Goal state: The desired configuration that signifies the solution.
 Operators: Actions that can be performed to transition from one state to another.
 Path cost: A numerical value representing the cost of a path taken from one state to
another.
2. Search Algorithms
Search algorithms systematically explore the space of possible states to find the solution. There
are two main categories of search:
Uninformed Search (Blind Search):
These algorithms do not have additional information about the problem other than the structure
of the state space. The most common uninformed search algorithms are:
 Breadth-First Search (BFS): Explores all the nodes at a given depth before moving to
the next depth level. It is optimal for unweighted graphs.
 Depth-First Search (DFS): Explores as far as possible along each branch before
backtracking. DFS can be more memory-efficient but might not find the optimal solution.
  Uniform Cost Search: Expands the node with the lowest path cost. It is optimal when
path costs are non-negative.
Informed Search (Heuristic Search):
These algorithms use additional knowledge (heuristics) to guide the search more efficiently. The
heuristic is a function that estimates the cost or distance to reach the goal from a given state.
 Greedy Best-First Search: Selects the node that appears to be closest to the goal, based
on the heuristic function.
 A Search*: Combines the advantages of both UCS and greedy search. It selects the node
with the lowest sum of the path cost and heuristic estimate, making it both complete and
optimal (when the heuristic is admissible and consistent).
3. Heuristics
A heuristic is a rule of thumb or approximation that helps search algorithms make decisions
more efficiently. It’s a function h(n) that estimates the cost from a given state n to the goal. There
are two key properties of a heuristic:
 Admissible: A heuristic is admissible if it never overestimates the cost to reach the goal.
A* search requires an admissible heuristic to guarantee optimality.
 Consistent (Monotonic): A heuristic is consistent if, for every node n and successor n',
the estimated cost from n to the goal is less than or equal to the cost of getting from n to
n' plus the heuristic estimate from n' to the goal. Consistency ensures that A* search is
optimal and doesn’t revisit states.
4. Types of Search Problems
 Single-Agent Search: Involves one agent exploring the state space to reach a goal.
Examples include puzzles like the 8-puzzle or the traveling salesman problem (TSP).
 Multi-Agent Search: Involves multiple agents interacting with each other, where each
agent may be trying to achieve its own goals, and the search space grows as a result.
Examples include multi-agent pathfinding and game playing.
5. Search Space Representation
The way the search space is represented is crucial for efficient problem-solving:
 State Space Graph: A graph where nodes represent states, and edges represent actions
that transition between states. The edges may have costs (path costs).
 Search Tree: A tree structure where nodes represent states, and edges represent actions.
Each branch in the tree represents a series of actions taken from the initial state to the
goal.
6. Applications of Search and Heuristics
Search and heuristics are used in many AI applications:
 Game AI: In games like chess or Go, search algorithms explore potential moves, and
heuristics help evaluate positions.
 Pathfinding: In robotics or navigation systems, search algorithms find the shortest or
most optimal path from one point to another.
 Optimization Problems: Algorithms like simulated annealing or genetic algorithms use
heuristics for finding solutions to complex optimization problems (e.g., TSP, scheduling).
 Natural Language Processing: Search is used in parsing and machine translation, where
the goal is to find the most likely translation or syntactic structure.

Foundations for AI:

Application areas
1. Healthcare: AI is used in diagnostic tools, drug discovery, and personalized medicine,
helping healthcare professionals make faster and more accurate decisions.
2. Autonomous Vehicles: AI is a key technology behind self-driving cars, using sensors,
cameras, and machine learning to navigate and make decisions in real-time.
3. Finance: AI algorithms help in fraud detection, risk assessment, and algorithmic trading,
improving the accuracy and efficiency of financial operations.
4. Entertainment: AI is used in recommendation systems on platforms like Netflix,
Spotify, and YouTube to suggest content based on user preferences.
5. Manufacturing: AI-powered robots are used for automation, predictive maintenance,
and optimizing production lines.
6. 4. Natural Language Processing (NLP): Speech recognition, translation, and chatbots
like GPT are powered by AI.
7. Robotics: AI enables robots to perform complex tasks, such as manufacturing and
household chores.
8. Gaming: AI is used to build non-playable characters (NPCs) and optimize game strategy.
9. Education: Personalized learning, grading systems, and virtual teaching assistants.
AI Basics (Divide and Conquer, Greedy, Branch and
Bound, Gradient Descent)
In AI and computational problem-solving, several fundamental algorithm design techniques are
employed to build efficient solutions. Let's explore each of these concepts in detail:

1. Divide and Conquer

 Concept: The divide and conquer paradigm involves breaking a problem down into
smaller sub-problems that are easier to solve. Each of these sub-problems is solved
independently, and their results are combined to solve the original problem.
 Steps:
1. Divide: Split the problem into smaller, more manageable sub-problems.
2. Conquer: Solve the sub-problems recursively.
3. Combine: Merge the solutions of the sub-problems to form the final solution.
 Time Complexity: Often characterized by recurrence relations, such as the recurrence
relation for merge sort, which is T(n)=2T(n/2) +O(n). Using the Master Theorem, this
gives a time complexity of O(nlogn).
 Examples:
o Merge Sort: Splits an array into halves, recursively sorts them, and merges the
results.
o Quick Sort: Divides the array around a pivot, recursively sorts the subarrays.
o Binary Search: Divides the search space in half at each step.
 Advantages:
o Efficient for large data sets.
o Can often be parallelized (as sub-problems are independent).

2. Greedy Algorithms
 Concept: A greedy algorithm makes the best local choice at each step with the hope that
these local choices will lead to a globally optimal solution.
 Key Idea: The greedy approach does not reconsider its choices, assuming that local
optimums lead to the global optimum.
  Steps:
1. Make the optimal choice at the current step.
2. Proceed to the next step with the decision made.
 Time Complexity: Typically, linear or logarithmic, depending on the problem and how
the choices are made.
 Examples:
o Activity Selection Problem: Select the maximum number of activities that don’t
overlap, choosing the one with the earliest finish time.
o Huffman Coding: Used in data compression, greedy algorithm selects the two
smallest frequencies to combine them into a new node.
o Dijkstra’s Algorithm: Finds the shortest path in a weighted graph by repeatedly
choosing the vertex with the smallest tentative distance.
 Advantages:
o Fast and simple.
o Can solve many problems efficiently when the greedy choice property holds.

3. Branch and Bound

 Concept: Branch and bound is a general algorithm for finding optimal solutions to
optimization problems, particularly useful for problems where the solution space is large,
and brute force is inefficient.
 Key Idea: It systematically explores the solution space by branching into possible sub-
solutions (states) and using bounds to prune branches that cannot yield a better solution
than the current best.
 Steps:
1. Branch: Generate sub-problems by making a decision that splits the problem
space.
2. Bound: For each sub-problem, calculate an upper or lower bound to estimate its
optimal solution.
3. Prune: Discard sub-problems that cannot possibly lead to a better solution than
the best solution found so far.
 Time Complexity: Can vary widely. It depends on the problem being solved and how
effectively pruning is used.
  Examples:
o Traveling Salesman Problem (TSP): Finding the shortest route that visits every
city exactly once. Branch and bound helps by pruning paths that exceed the best-
known path length.
o Knapsack Problem: In the 0/1 knapsack problem, this approach can prune
combinations of items that are not feasible or optimal.
 Advantages:
o Guarantees finding the optimal solution if implemented correctly.
o Often leads to significant reductions in computation time for large search spaces
compared to brute force methods.

4. Gradient Descent
 Concept: Gradient descent is an optimization algorithm used to minimize a function by
iteratively moving towards the minimum of the function in the direction of the negative
gradient (the steepest descent).
 Key Idea: By taking steps proportional to the negative of the gradient, you can gradually
converge to a minimum (or a local minimum) of the function.
 Steps:
1. Start with an initial guess for the parameters.
2. Compute the gradient (partial derivatives) of the loss function with respect to each
parameter.
3. Update the parameters by moving in the direction opposite to the gradient.
4. Repeat until convergence (i.e., the change in the loss function is very small).
 Time Complexity: The number of iterations is key. Each iteration involves calculating
the gradient and updating the parameters, typically leading to a time complexity of O(n)
per iteration for large datasets.
 Examples:
o Neural Networks: Used in backpropagation to minimize the error between
predicted and actual outputs.
o Linear Regression: Minimize the sum of squared errors between the predicted
and actual outputs.
o Logistic Regression: Minimizing the cross-entropy loss between predicted
probabilities and true labels.
  Advantages:
o Widely used in machine learning, especially for training models.
o Can handle large datasets efficiently.
o Works well for many optimization problems, especially when the exact solution is
hard to compute directly.

Perceptron and MLP, FFN, Backpropagation

Let's dive deeper into Perceptrons, MLPs (Multilayer Perceptrons), FFNs (Feedforward
Networks), and Backpropagation, all of which are fundamental concepts in neural networks
and deep learning.

Perceptron
The Perceptron is the simplest form of a neural network model and can be considered the
building block for more complex neural networks.
Key Features:
 Single Layer: It consists of a single layer of neurons (also called units or nodes), which
perform a binary classification task.
 Inputs and Weights: The perceptron receives inputs, each of which is multiplied by a
corresponding weight.
 Activation Function: The weighted sum of the inputs is passed through an activation
function, typically a step function, which determines the output (0 or 1).
Formula:
Let’s define:
 x1, x2…, xn as the input features.
 w1, w2…, wn as the weights.
 b as the bias term.
 y as the output.
The perceptron computes:
Output = f (∑ (i=1 to n) wixi + b)
where f is the activation function. For a basic perceptron, f is usually a step function:
f(x)= {1 if x > 0
0 if x ≤ 0
Learning Rule (Training):
 The weights w1, w2,…,wn and the bias b are updated during training using a simple rule:
wi←wi+Δwi
 where: Δwi = η(ytrue−ypred) *xi
 η is the learning rate
 ytrue is the true label and ypred is the predicted output.
Limitations:
 Linearly Separable Problems: Perceptrons can only solve problems where the data is
linearly separable (i.e., it can be divided by a straight line or hyperplane).
 No Hidden Layers: A perceptron is very limited due to its single-layer structure, making
it unable to model complex, nonlinear relationships.

Multilayer Perceptron (MLP)

An MLP (Multilayer Perceptron) is an extension of the perceptron that consists of multiple
layers of neurons, making it capable of solving nonlinear problems.
Key Features:
 Multiple Layers: An MLP consists of an input layer, one or more hidden layers, and an
output layer.
 Hidden Layers: The hidden layers provide the network with the ability to model more
complex, nonlinear relationships between inputs and outputs.
 Nonlinear Activation Functions: Unlike the perceptron’s step function, MLPs use
nonlinear activation functions like Sigmoid, ReLU (Rectified Linear Unit), or Tanh.
Architecture:
 Input Layer: Takes input features (same as in a perceptron).
 Hidden Layers: One or more layers that perform nonlinear transformations of the inputs.
 Output Layer: Produces the final prediction.
Activation Functions:
 Sigmoid: σ(x) = 1 / 1+e^−x
 Tanh: tanh(x) = e^x - e^−x / e^x + e^−x
  ReLU (most common in deep learning): ReLU(x) = max (0, x).
Learning Rule:
The MLP is typically trained using gradient-based optimization methods, particularly
Gradient Descent and Backpropagation.
Feedforward Neural Networks (FFN)
A Feedforward Neural Network (FFN) is one of the simplest types of neural networks.
It consists of layers of neurons, where each layer only receives input from the previous layer, and
sends output to the next layer.
In a FFN, the data flows in one direction: from the input layer to the output layer, without any
feedback loops. This means there are no recurrent connections, unlike in Recurrent Neural
Networks (RNNs).
Feedforward networks are commonly used for tasks like classification, regression, and many
others in machine learning, especially when the task involves structured data.
Key Components of FFN
1. Input Layer:
o The first layer of the network that receives the input features.
o Each neuron in this layer represents a feature in the data.
o The input layer passes the data forward to the next layer.
2. Hidden Layers:
o These layers come between the input and output layers.
o They process the inputs using weighted sums and apply an activation function.
o The number of hidden layers and the number of neurons in each hidden layer are
hyperparameters that define the capacity of the network.
o Activation Functions: Common choices include ReLU (Rectified Linear Unit),
Sigmoid, Tanh, and others that introduce non-linearity into the model, allowing it
to solve more complex problems.
3. Output Layer:
o The final layer of the network, which produces the output based on the processed
input.
o In a classification task, this could output probabilities (using activation functions
like SoftMax for multi-class problems or Sigmoid for binary classification).
 o In a regression task, the output layer would typically have a single neuron with
no activation function or a linear activation.
4. Weights and Biases:
o Weights are parameters that connect the neurons of one layer to the next. The
weight defines the strength of the connection.
o Biases are constants added to the weighted sum of inputs, helping the model make
better predictions even when all input values are zero.
5. Activation Functions: Activation functions introduce non-linearity, allowing the network
to learn complex patterns.
 Sigmoid: σ(x) = 1 / 1+e^−x
 Tanh: tanh(x) = e^x - e^−x / e^x + e^−x
 ReLU (most common in deep learning): ReLU(x) = max (0, x).
Feedforward Propagation
The core of FFNs is the forward pass (or feedforward propagation), where the input data flows
from the input layer to the output layer.
Steps:
1. Input Layer: The input features are presented to the network.
2. Weighted Sum: Each neuron in the hidden layer calculates a weighted sum of its inputs.
3. Activation: The weighted sum is passed through the activation function, which
transforms the output to a desired range or non-linearity.
4. Output Layer: After passing through all the hidden layers, the result reaches the output
layer, which generates the final prediction.

Training FFN with Backpropagation

Feedforward neural networks are typically trained using backpropagation (the method of
computing gradients and updating weights), with gradient descent being the optimization
method that minimizes the loss.
Key Steps in Backpropagation:
1. Forward Pass: Compute the output of the network for a given input.
2. Loss Calculation: Compare the predicted output with the actual output using a loss
function (e.g., mean squared error, cross-entropy).
3. Backward Pass (Backpropagation):
 o Calculate the gradient of the loss function with respect to each weight using the
chain rule of calculus.
o Update the weights by subtracting a fraction (learning rate) of the gradient to
minimize the loss.
4. Repeat: This process is repeated for all training examples and across multiple epochs.
The backpropagation process is crucial for adjusting the weights of the network, which helps in
improving the accuracy of predictions.

Feedforward Networks in Practice

Feedforward neural networks are used in many applications:
 Classification: Predicting a category (e.g., spam vs. not spam).
 Regression: Predicting a continuous value (e.g., predicting house prices).

Advantages of FFN
 Simple Architecture: Easy to implement and understand.
 Flexibility: Can be used for a variety of tasks, including classification and regression.
Backpropagation
Backpropagation is the key algorithm used for training neural networks, including MLPs and
FFNs. It allows the network to adjust the weights based on the error between the predicted
output and the actual target.
Key Idea:
 Goal: Minimize the loss function by adjusting the weights in such a way that the error is
minimized.
 How It Works: Backpropagation computes the gradient of the loss function with
respect to each weight in the network by applying the chain rule of calculus. This allows
us to perform gradient descent and update the weights efficiently.
Steps in Backpropagation:
1. Forward Pass: Compute the output of the network for a given input by passing it
through all the layers.
2. Compute the Loss: Calculate the loss (or error) by comparing the predicted output with
the true label using a loss function (e.g., mean squared error for regression or cross-
entropy for classification).
 3. Backward Pass:
o Compute the gradient of the loss function with respect to the output layer.
o Using the chain rule, propagate the gradient back through the network, layer by
layer, adjusting the weights.
o Update the weights using the gradient descent update rule: w = w – η ⋅ ∂L/∂w
where L is the loss function and η is the learning rate.
4. Repeat: Continue this process iteratively across the entire dataset, updating the weights
after each mini-batch or epoch until convergence.
Key Components:
 Loss Function: Measures the difference between the predicted output and the true output.
Common choices are Mean Squared Error (MSE) for regression tasks and Cross-
Entropy for classification tasks.
 Gradient Descent: Used to update the weights based on the gradients computed during
backpropagation.

Use cases in Business and Scope:

Credit card fraud analysis
Use Case: Credit card fraud analysis helps businesses detect and prevent fraudulent transactions,
ensuring the security of both customers and the company’s financial systems. By leveraging
machine learning algorithms and data analytics, companies can monitor real-time transactions
and flag suspicious activity for further investigation.
Scope:
 Data Collection: Gathering data on customer transactions, including details like
purchase patterns, location, device used, and more.
 Fraud Detection: Machine learning models (like decision trees, neural networks, or
ensemble methods) are used to identify unusual patterns, such as sudden large
transactions or out-of-pattern behavior.
 Risk Management: Identifying high-risk customers and transactions to reduce the
overall risk of financial losses.
 Customer Protection: Ensuring customer trust by alerting them to fraudulent activity
and taking corrective actions, such as freezing accounts or sending notifications.
 Business Impact: This minimizes fraud-related losses, enhances customer trust, and can
lead to better customer retention and satisfaction.
Recommendation systems and collaborative filtering
Use Case: Recommendation systems are used by businesses, especially in e-commerce,
streaming services, and social media, to suggest products, services, or content based on users'
past behavior or preferences. Collaborative filtering is a common technique used in
recommendation systems where recommendations are made based on the behavior or
preferences of similar users.
Scope:
 Data Collection: Gathering user activity, ratings, clicks, and purchase history to build
user profiles and interactions.
 User Behavior Analysis: Identifying patterns in user interactions, such as which
products are often viewed or purchased together.
 Collaborative Filtering: Using either user-based or item-based collaborative filtering to
make predictions. For example, recommending a product that similar users have bought.
 Personalized Experience: Businesses can provide a tailored experience to each user,
which can increase engagement, conversion rates, and overall sales.
 Business Impact: Increased sales, higher customer satisfaction, and retention by offering
relevant product recommendations. It also provides deeper insights into customer
preferences.

Sales funnel analysis

Use Case: Sales funnel analysis helps businesses understand and optimize the stages customers
go through before making a purchase. By analyzing how leads move through different funnel
stages (awareness, consideration, decision), businesses can identify where prospects drop off and
how to improve conversion rates.
Scope:
 Data Collection: Gathering data from various stages of the sales process, such as
customer inquiries, website visits, engagement with marketing campaigns, and sales team
interactions.
 Funnel Stages Analysis: Breaking down the sales process into stages and identifying
where prospects are getting stuck or dropping off.
 Conversion Rate Optimization: Analyzing data to improve conversion rates at each
funnel stage, adjusting marketing strategies, or sales tactics accordingly.
 Customer Journey Mapping: Understanding the typical customer journey, from first
contact to the final sale, and optimizing touchpoints to improve customer engagement.
 Business Impact: Higher conversion rates, improved sales strategies, and a more
efficient use of marketing resources. By understanding where customers drop off,
businesses can tailor their approach to reduce these losses.