MODULE 2

A Neural Network (NN) is a computational model inspired by the human brain. It consists of interconnected nodes
(neurons) arranged in layers that process input data, learn patterns, and make predictions or decisions. Neural
networks are the backbone of deep learning and a subset of machine learning.

🔹 Structure / Components:

1. Input Layer: Takes in raw data (e.g., images, text, numbers).

2. Hidden Layers: Intermediate layers where computations happen using weights and activation functions.

3. Output Layer: Produces the final result (e.g., class label, value).

4. Neurons: Units that receive input, apply transformation, and pass output to the next layer.

5. Weights: Parameters that determine the importance of inputs.

6. Bias: A constant added to help the model fit data better.

7. Activation Function: Decides whether a neuron should be activated or not (e.g., ReLU, Sigmoid).

🔹 How Neural Networks Work:

1. Forward Propagation: Input passes through layers, weighted sums are calculated, activation functions
applied, and an output is generated.

2. Loss Calculation: The difference between predicted output and actual output is measured using a loss
function.

3. Backpropagation: The error is propagated back to adjust weights and biases using optimization (usually
Gradient Descent).

4. Training: This process repeats over many iterations (epochs) to minimize error and improve accuracy.

🔹 Functions of Neural Networks:

 Classification (e.g., spam detection, image recognition)

 Regression (e.g., price prediction)

 Pattern recognition

 Feature extraction

 Data generation (GANs)

 Clustering (in unsupervised models)

🔹 Types of Neural Networks:

1. Feedforward Neural Network (FNN) – Basic architecture where data flows in one direction.

2. Convolutional Neural Network (CNN) – Used for image and video processing.

3. Recurrent Neural Network (RNN) – Handles sequence data like time-series, speech, or text.

4. Radial Basis Function Network (RBFN)

 5. Modular Neural Network (MNN)

6. Generative Adversarial Network (GAN)

🔹 Applications:

 Healthcare: Disease prediction, medical imaging analysis.

 Finance: Stock market prediction, fraud detection.

 Retail: Recommendation systems, customer behavior analysis.

 Autonomous Vehicles: Object detection, navigation.

 Natural Language Processing (NLP): Chatbots, language translation, sentiment analysis.

 Cybersecurity: Intrusion detection, malware classification.

 Robotics: Visual recognition, motor control.

🔹 Importance of Neural Networks:

 Mimic human intelligence for complex tasks.

 Capable of automatic feature extraction.

 Adaptable and scalable to large datasets.

 Enable deep learning architectures for higher accuracy.

 Drive modern AI applications (self-driving cars, voice assistants, etc.).

 Outperform traditional algorithms in tasks involving unstructured data (images, text, audio).

🔹 Advantages:

 Handles non-linear and complex data efficiently.

 Learns from data without explicit programming.

 Generalizes well to unseen data when trained properly.

 Highly versatile across domains.

🔹 Disadvantages:

 Requires large datasets and computation power.

 Training can be time-consuming.

 Acts as a black box – hard to interpret internal decision-making.

 Prone to overfitting if not regularized.

1. Advent of Modern Neuroscience

Definition

Modern neuroscience is the scientific field that studies how the brain and nervous system function. It explores how
neurons communicate, process information, and control cognitive functions like memory, learning, decision-making,
and emotions.

Functions

 Explains how electrical and chemical signals transmit information across neurons.

 Studies brain functions like perception, motor control, and language processing.

 Models how information is encoded, stored, and retrieved in the brain.

Applications

 Brain-computer interfaces for communication and control.

 Diagnosis and treatment of neurological and psychological disorders.

 Cognitive computing and neuro-inspired machine learning models.

 Development of prosthetic devices and rehabilitation technologies.

Importance

 Serves as a foundation for building intelligent systems that mimic the brain.

 Enhances understanding of human cognition, learning, and behavior.

 Bridges biology and computational sciences for AI and robotics.

2. Classical AI (Symbolic AI)

Definition

Classical AI, also known as symbolic AI, is a rule-based approach where intelligence is achieved through logic,
reasoning, and manipulation of symbols. It uses predefined rules and knowledge bases to simulate intelligent
behavior.

Functions

 Performs reasoning, decision-making, and planning using logical inference.

 Uses formal languages and structured data for problem-solving.

 Employs search techniques and heuristics for navigation and optimization tasks.

Applications

 Expert systems for medical, legal, and financial decision-making.

 Logic-based games and puzzle solvers.

 Rule-driven chatbots and dialogue systems.

 Scheduling, route planning, and symbolic data interpretation.

Importance

 Provides a transparent and interpretable approach to AI.

 Ideal for domains where knowledge can be explicitly defined.

  Useful in applications requiring step-by-step logical processes.

3. Neural Networks (Connectionist AI)

Definition

Neural networks are computational models inspired by the structure and functioning of the brain. They consist of
layers of interconnected artificial neurons that learn patterns and relationships from data through training.

Functions

 Automatically extract features and learn patterns from raw data.

 Perform tasks like classification, regression, and clustering.

 Support non-linear and high-dimensional data modeling.

 Adjust internal weights based on errors to improve accuracy over time.

Applications

 Image and speech recognition systems.

 Natural language processing tasks such as sentiment analysis and translation.

 Predictive modeling in healthcare, finance, and manufacturing.

 Autonomous systems including self-driving cars and intelligent robots.

 Content generation using generative models.

Importance

 Eliminates the need for manual feature engineering.

 Excels at handling large-scale, unstructured, and complex datasets.

 Forms the core of modern artificial intelligence and deep learning systems.

 Enables advanced capabilities like real-time object detection and language understanding.

Summary Table

Category Modern Neuroscience Classical AI Neural Networks

Biological brain and neural Symbolic logic and rule-based Artificial neurons and learning
Based On
function reasoning models

Learning Capability Biological, natural learning No learning (predefined rules) Learns from data

Biological and physiological Structured and unstructured

Data Type Structured, symbolic data
data data

Flexibility High (adaptive to stimuli) Low (rigid and deterministic) High (adaptive and scalable)

Interpretability Medium High Low (black-box nature)

Neuroscience, medicine, AI Expert systems, logic-based Deep learning, AI, automation,

Application Scope
inspiration automation robotics
Model of Artificial Neuron

Definition

An Artificial Neuron is a mathematical model inspired by the structure and function of a biological neuron. It is the
basic building block of an artificial neural network (ANN). It takes input values, processes them using weights and a
bias, applies an activation function, and produces an output.

Main Components

1. Inputs (x1,x2,….xn)
Numeric data received by the neuron from the outside or from previous layers.

2. Weights (w1,w2,..., wn)

Each input is multiplied by a weight, representing its importance in the decision-making process.

3. Bias (b)
A constant added to the weighted sum. It helps shift the output curve and enables the neuron to produce a
result even when all inputs are zero.

4. Weighted Sum (z)

The neuron computes the total input using:

z = x1*w1 + x2*w2 + x3*w3 + ... + xn*wn + b

5. Activation Function (f(z))

A function that processes the weighted sum to introduce non-linearity and produce the final output:

y = f(z)

Common Activation Functions

Output
Function Formula Use Case
Range

f(z) = 1 if z > 0
Step 0 or 1 Binary classification
f(z) = 0 if z <= 0

Sigmoid f(z) = 1 / (1 + e^(-z)) (0, 1) Smooth, for probabilities

Tanh f(z) = (e^z - e^(-z)) / (e^z + e^(-z)) (-1, 1) Centered output

Deep networks, fast

ReLU f(z) = max(0, z) [0, ∞)
convergence

Working Process of an Artificial Neuron

1. Takes multiple input values.

2. Multiplies each input with its corresponding weight.

3. Adds the bias to the weighted sum.

4. Passes the result through an activation function.

5. Produces the final output.

Applications

 Image and speech recognition

 Sentiment analysis and recommendation systems

 Medical diagnosis and financial forecasting

 Self-driving cars and robotics

 Natural language processing

Importance

 Acts as the fundamental unit in building complex neural networks.

 Learns from data and makes predictions.

 Allows modeling of non-linear decision boundaries.

 Supports the development of deep learning and AI systems.

Learning Methods in Neural Networks

Definition:

Learning methods in neural networks refer to algorithms or rules that guide how the network adjusts its internal
parameters (weights and biases) based on input data, to perform tasks such as classification, prediction, clustering,
etc.

These methods help the neural network learn patterns or knowledge from data and can be broadly categorized into:

 Supervised Learning

 Unsupervised Learning

 Reinforcement Learning

Below are three key unsupervised learning methods: Hebbian, Competitive, and Boltzmann Learning.

1. Hebbian Learning

Definition:

Hebbian Learning is based on the idea proposed by Donald Hebb:

"If two neurons are activated simultaneously, the connection between them strengthens."

It models the biological process of synaptic plasticity in the brain.

Formula:

[ADD FORMULA]

Function:

 Strengthens the connection between two co-activated neurons.

 No target output is used (unsupervised learning).

 Learning is local and dependent on the activity of connected neurons.

Applications:

 Associative memory (e.g., Hopfield Networks)

 Feature detectors in unsupervised learning

 Learning in biologically inspired systems

Importance:

 Mimics actual learning in the brain.

 Forms the foundation of self-organizing networks.

 Encourages pattern discovery from data.

2. Competitive Learning

Definition:

In Competitive Learning, neurons compete to become active, and only one neuron (the winner) is updated during
training. This is also known as Winner-Takes-All (WTA) learning.

Formula (Weight Update for Winner Neuron):

[ADD FORMULA]

Function:

 Only one neuron (with the closest match) is allowed to learn from the input.

 Encourages neurons to specialize in representing different input patterns.

 Reduces redundancy in representation.

Applications:

 Clustering (e.g., k-means, Kohonen SOM)

 Pattern classification

 Data compression

 Adaptive vector quantization

Importance:

 Useful for dimensionality reduction and visualization.

 Enhances diversity in learned features.

 Self-organizes neurons based on input distribution.

3. Boltzmann Learning

Definition:

Boltzmann Learning is a probabilistic, stochastic learning algorithm used in Boltzmann Machines, based on energy
minimization and thermodynamics. It aims to reach a global optimum through simulated annealing.

Energy Function:
 [ADD FORMULA]

Weight Update Rule:

[ADD FORMULA]

Function:

 Trains neural networks to model a probability distribution.

 Allows hidden layers to learn complex features.

 Gradually updates weights to reduce the difference between actual and model data.

Applications:

 Deep Belief Networks (DBNs)

 Restricted Boltzmann Machines (RBMs)

 Feature extraction and dimensionality reduction

 Collaborative filtering and recommendation systems

Importance:

 Effective in learning generative models.

 Enables unsupervised pre-training of deep neural networks.

 Can discover hidden structure in data.

Comparison of Learning Methods

Feature Hebbian Learning Competitive Learning Boltzmann Learning

Learning Type Unsupervised Unsupervised Unsupervised

Probabilistic, based on
Weight Update Based on co-activation Winner neuron only
energy

Target Output
No No No
Required

Neuron Interaction Cooperative Competitive Stochastic

Feature learning, generative
Key Use Associative memory Clustering, Self-organization
models

Neural Network Models: Perceptron, Adaline, and Madaline

Perceptron, Adaline, and Madaline are early artificial neural network models, with Adaline and Madaline being
extensions of the perceptron concept. Adaline is a single-layer network using a linear activation function, while
Madaline is a multi-layer network of Adaline units.

 Perceptron:
  A simple, single-layer artificial neural network that learns by adjusting weights based on the
difference between the predicted and actual output.

 It takes weighted inputs, sums them, and applies a threshold function to produce a binary output (0
or 1).

 Can only learn linearly separable patterns, meaning it can only separate data points that can be
divided by a straight line (or hyperplane in higher dimensions).

 A single-layer perceptron is a linear model, while a multilayer perceptron (MLP) is a more complex
network with multiple layers and non-linear activation functions.

wi(new) = wi(old) + η * (target - output) * xi

Where:

 η = learning rate

 target = actual class

 output = predicted class

Advantages:

 Very easy to implement using basic mathematical concepts

 Performs well when the data is linearly separable

 Requires minimal computational resources

 Fast training on small and simple datasets

 Helps build foundational understanding of neural networks

Limitations:

 Cannot solve problems where data is not linearly separable (e.g., XOR problem)

 Only supports binary classification (outputs are 0 or 1)

 Performance depends heavily on the choice of learning rate

 Not capable of learning complex or non-linear relationships

Applications:

 Implementing basic logic gates like AND and OR

 Solving linearly separable binary classification tasks (e.g., spam vs. non-spam)

 Used as a starting point in designing more complex neural networks

 Helpful in teaching fundamental concepts of machine learning, like weight updates and error correction

 Adaline (Adaptive Linear Neuron):

 An extension of the perceptron, but instead of a step activation function, it uses a linear activation
function.

 It's a single-layer artificial neural network.

  Uses the Widrow-Hoff rule (also known as the delta rule) for training, which minimizes the mean
squared error between the activation and target values.

Activation Function:

Linear activation initially, then thresholded:

f(z) = z

Final output = 1 if z > 0, else 0

Advantages:
• Minimizes continuous error using Mean Squared Error (MSE)
• Converges more reliably than the Perceptron algorithm
• More accurate for real-valued outputs
• Uses differentiable activation functions, allowing smooth gradient-based optimization
• Can be extended to multilayer architectures for more complex tasks
• Better generalization performance for linearly separable data
• Supports efficient learning using gradient descent

Limitations:
• Still limited to linearly separable data when used in a single layer
• Cannot handle non-linear patterns or complex decision boundaries on its own
• Sensitive to input feature scaling
• Performance degrades with noisy or overlapping data
• Requires differentiable activation and loss functions
• May get stuck in local minima depending on the optimization method

Applications:
• Signal processing tasks like enhancement and transformation
• Noise filtering in audio, images, or sensor data
• Real-time prediction and regression tasks (e.g., weather forecasting, stock prediction)
• Function approximation in control systems and robotics
• Time-series forecasting with simple trend detection
• Calibration of sensor data and normalization functions
• Used as a foundational layer in multilayer neural networks and deep learning models

 Madaline (Many Adaline):

 A multi-layer network of Adaline units, allowing it to perform more complex tasks than a single
Adaline.

 It consists of multiple Adaline units arranged in layers, including a hidden layer.

 Can be trained using the Madaline Rule II algorithm.

 Madaline networks can be used for tasks like echo cancellation, signal prediction, and adaptive
beamforming antennas.

Advantages:
  Can handle non-linearly separable problems using hidden layers

 Supports multi-layer architecture (basis of deep learning)

 Learns complex patterns and decision boundaries

 Improves accuracy over Adaline and Perceptron models

 Works well with non-linear activation functions like sigmoid, tanh, or ReLU

 Capable of learning both classification and regression tasks

 Adaptable to a wide range of real-world problems

 Forms the foundation of modern neural networks like CNNs and RNNs

Limitations:

 Training is more complex due to backpropagation and multiple layers

 May get stuck in local minima, affecting model performance

 Requires more computational resources and time

 Sensitive to initial weight values and hyperparameters

 Needs careful tuning of learning rate, number of layers, and neurons

 Harder to interpret the learned model (acts like a black box)

Applications:

 Speech recognition systems (e.g., voice assistants)

 Handwritten digit recognition (e.g., postal code or MNIST dataset)

 Pattern classification in image, text, and signal data

 Medical diagnosis systems using image or symptom data

 Financial forecasting and stock price prediction

 Face detection and recognition in images and videos

 Language translation using deep learning models

 Autonomous vehicles for object detection and decision-making

Feature Perceptron Adaline Madaline

Architecture Single-layer Single-layer Multi-layer (with Adaline units)

Activation
Step (Binary) Linear (followed by threshold) Linear + Majority Logic
Function

Binary after multiple unit

Output Binary (0 or 1) Continuous before thresholding
decisions

Learning Rule Perceptron Learning Rule Least Mean Square (LMS) Madaline Rule (MR-I, MR-II)
Feature Perceptron Adaline Madaline

Linearly separable with continuous

Data Compatibility Linearly separable data only Handles more complex data
targets

Convergence May not converge always Converges better Better accuracy and learning

Applications Logic gate simulation Noise filtering, prediction Speech, pattern recognition

Single Layer Neural Network:

A single-layer neural network, also known as a perceptron, is the simplest form of an artificial neural network,
consisting of only an input layer and an output layer, without any hidden layers.

A single layer neural network is the simplest type of feedforward network with only one layer of weights connecting
input features directly to the output node(s). It does not contain hidden layers and is mainly used for solving linearly
separable classification problems.

• Structure:

– It has an input layer that receives data, and an output layer that produces the final prediction or classification.

– There are no hidden layers between the input and output, meaning the input data is directly processed to
generate the output.

– Each input is connected directly to the output neuron using adjustable weights.

• Importance:

– Forms the basic structure of neural networks and introduces the concept of weight-based learning.

– Useful for understanding core ML principles like decision boundaries and linear classification.

– Offers fast training and low computational cost.

• Function:

– It's used for simple tasks, such as linear classification, where the decision boundary is a straight line.

– Each input is multiplied by a weight, and the weighted inputs are summed, and then passed through an activation
function to produce an output.

• Limitations:

– Single-layer neural networks are limited to linearly separable data, meaning they cannot learn complex, non-linear
patterns.

– They are not suitable for tasks that require feature extraction or complex decision-making.

• Applications:

– Used to simulate basic logic gates like AND, OR.

– Binary classification tasks such as spam vs. not spam, pass/fail detection.

– Educational purposes to demonstrate the working of neural networks.

• Example:
 – A single-layer perceptron can be used for simple binary classification tasks, such as distinguishing between emails
that are spam and not spam.

Backpropagation:

Definition:
Backpropagation is a learning algorithm that helps a neural network learn by reducing the error between actual and
predicted output. It updates weights by sending error backwards.

Backpropagation is a machine learning algorithm that adjusts the weights of a neural network to reduce errors. It's a
key part of artificial neural networks (ANNs) and deep learning.

• Structure:

– Backpropagation is a training algorithm used for multilayer neural networks.

– It involves two phases: forward pass and backward pass.

– During the forward pass, input data is propagated through the network to produce an output.

– During the backward pass, the error between predicted and actual output is propagated backward to adjust
weights.

• Importance:

– Enables learning in multilayer neural networks by minimizing prediction error.

– Makes training deep networks possible, forming the backbone of modern deep learning.

– Improves model accuracy by optimizing weights based on calculated error.

• Function:

– Computes the error at the output layer.

– Uses the chain rule of calculus to distribute the error back through the layers.

– Adjusts weights to reduce error using a learning rate, refining the model's accuracy over time.

– Repeats this process across multiple epochs during training.

• Limitations:

– Can be computationally expensive for very deep networks.

– Requires differentiable activation functions.

– Can get stuck in local minima during training and may suffer from vanishing gradients.

• Applications:

– Training deep learning models in image and speech recognition.

– Natural language processing tasks like sentiment analysis and translation.

– Financial forecasting, medical diagnosis, and autonomous systems.

• Example:

– Used in training a neural network on the MNIST dataset to recognize handwritten digits by minimizing prediction
error.

Multilayer Neural Network:

Definition:
A network with more than one layer, including one or more hidden layers between input and output layers. Also
called Multilayer Perceptron (MLP).

A Multilayer Neural Network, also known as a Multilayer Perceptron (MLP), is a type of artificial neural network
characterized by having multiple layers, including an input layer, one or more hidden layers, and an output layer.

• Structure:

– Consists of an input layer, one or more hidden layers, and an output layer.

– Neurons in each layer are fully connected to the neurons in the next layer.

– Non-linear activation functions like ReLU, sigmoid, or tanh are used in hidden layers.

• Importance:

– Can model complex, non-linear relationships that single-layer networks cannot.

– Enables deep learning and advanced AI applications.

– Automatically extracts features from input data, reducing manual effort.

• Function:

– Takes input data and transforms it through multiple layers to generate a prediction.

– Each layer captures increasingly abstract features.

– Uses forward propagation to calculate output and backpropagation to update weights.

– Learns from data iteratively over time to improve accuracy.

• Limitations:

– Requires more computational resources and longer training time.

– Needs large datasets and careful tuning of hyperparameters.

– May overfit on small datasets if not properly regularized.

• Applications:

– Image and object recognition (e.g., detecting cats, dogs, faces).

– Text classification and machine translation in NLP.

– Fraud detection, medical image analysis, and autonomous driving.

• Example:

– A multilayer neural network can classify images by identifying features like edges, shapes, and objects,
distinguishing between cars, people, and animals.