DWDM

UNIT 3 DATA MINING

To Do:
1. Overview, Motivation, Definition & Functionalities
2. Data Processing, Form of Data Pre-processing
3. Data Cleaning: Missing Values, Noisy Data (Binning, Clustering, Regression, Computer
and Human inspection)
4. Inconsistent Data
5. Data Integration and Transformation
6. Data Reduction:
a. Data Cube Aggregation
b. Dimensionality reduction
c. Data Compression
d. Numerosity Reduction
e. Discretization and Concept Hierarchy Generation
7. Decision Tree
# Overview, Motivation, Definition, and Functionalities of Data Mining:

1. Overview of Data Mining

Data mining is a process of extracting meaningful patterns, knowledge, and insights from
large datasets. It is a core part of the larger process known as Knowledge Discovery in
Databases (KDD). Data mining uses mathematical models, algorithms, and statistical
methods to uncover hidden patterns, correlations, or anomalies in data.
• Why it matters: Organizations generate and store vast amounts of data every day.
Data mining helps turn raw data into useful information that can guide decisions.
• Examples:
o Identifying purchasing patterns in e-commerce.
o Analysing customer churn rates in telecommunications.
o Predicting fraud in banking and insurance.

2. Motivation for Data Mining

The motivation behind data mining lies in the explosion of data in recent years and the
need for businesses, scientists, and institutions to make sense of it. Key motivators include:
• Data Overload: Massive amounts of data are collected daily in fields like business,
healthcare, and social media, making manual analysis impossible.
• Decision-Making: Businesses need insights to make data-driven decisions rather than
relying on intuition.
• Competitive Advantage: Extracting patterns from data allows companies to optimize
processes, personalize services, and predict trends.
• Advancements in Technology: Growth in computational power, storage capacity, and
algorithms has made data mining more accessible.
• Real-Life Applications: From predicting stock trends to diagnosing diseases, data
mining offers tangible solutions to problems.

3. Definition of Data Mining

Data Mining is defined as:
"The process of discovering patterns, trends, and knowledge from large datasets using
methods from statistics, machine learning, and database systems."
It is also known as Knowledge Discovery in Data (KDD).
• Key Terms:
o Patterns: Hidden relationships or trends in data.
o Dataset: A structured collection of data points, often stored in databases or
warehouses.
o Machine Learning Algorithms: Techniques like classification, clustering, and
association rules to analyse data.
o Insights: Actionable knowledge discovered from raw data.
4. Functionalities of Data Mining ( diff types of tasks Data mining can do )
Data mining has various functionalities that help achieve different objectives. These
include:
1. Classification
o Assigns data to predefined categories based on rules or algorithms.
o Example: Categorizing emails as spam or not spam.
2. Clustering
o Groups similar data points together without predefined labels.
o Example: Segmenting customers based on purchasing behaviour.
3. Association Rule Mining
o Discovers relationships between variables or items.
o Example: Market Basket Analysis shows that customers buying "bread" often
buy "butter."
4. Prediction
o Forecasts future trends based on historical data.
o Example: Predicting stock prices or customer churn rates.
5. Outlier Detection
o Identifies unusual or rare patterns in the data that differ significantly from the
norm.
o Example: Detecting fraudulent credit card transactions.
6. Regression Analysis
o Establishes a relationship between variables to predict numerical outcomes.
o Example: Predicting house prices based on area, location, and amenities.
7. Summarization
o Provides a compact representation of data for visualization or reporting.
o Example: Generating a summary of customer demographics for a marketing
report.

Conclusion
The overview and motivation for data mining highlight its importance in handling large
datasets for actionable insights. The definition shows that it is a systematic process
involving machine learning and statistical methods, while the functionalities provide
specific tasks that can help businesses, researchers, and industries.
This powerful tool allows organizations to unlock the value hidden within data and improve
decision-making in diverse fields like healthcare, retail, finance, and more.
# Data Processing and Forms of Data Pre-processing:

1. Data Processing
Data processing refers to the steps taken to transform raw data into meaningful and usable
information. Raw data collected from various sources often contains errors, inconsistencies,
and noise. To make it suitable for data mining and analysis, it needs to undergo several
processes, collectively called data preprocessing.
• Why is Data Processing Important?
o Improves Data Quality: Clean and accurate data leads to better analysis and
predictions.
o Reduces Redundancy: Eliminates duplicate or irrelevant data.
o Optimizes Performance: Well-processed data improves the efficiency of data
mining algorithms.
o Ensures Consistency: Standardizes data formats and removes inconsistencies.

2. Forms of Data Pre-processing

Data pre-processing involves multiple tasks that clean, transform, and prepare data for
analysis. The main steps include:

A. Data Cleaning
Definition: Data cleaning involves identifying and correcting errors, inconsistencies, or
missing values in the dataset to ensure high-quality data.
• Steps in Data Cleaning:
o Handling Missing Values:
▪ Replace missing values with a constant (e.g., zero) or a mean/median.
▪ Remove rows or columns with too many missing values.
▪ Use predictive models to estimate missing values.
▪ Example: If income data has missing values, you can replace them with
the average income.
o Removing Noisy Data (Irrelevant or outlier data):
▪ Binning: Sort data into bins (ranges) and smooth values by mean,
median, or boundaries.
▪ Regression: Use a regression model to predict and smooth noisy data
points.
▪ Clustering: Group data into clusters and remove outliers.
▪ Human Inspection: Manual review for small datasets.
▪ Example: In a sales dataset, entries like "99999" as age could be outliers.
o Resolving Inconsistent Data:
▪ Standardize data formats, units, or naming conventions.
▪ Correct inconsistencies (e.g., "NY" vs. "New York").
B. Data Integration
Definition: Combining data from multiple sources into a single, unified view.
• Purpose: Ensures that all the data required for analysis is merged and accessible.
• Challenges:
o Data may have different formats (e.g., databases, spreadsheets, files).
o Redundancies: Duplicate records or fields must be removed.
o Schema Integration: Combining fields like "Customer_ID" vs. "C_ID".
• Example: Combining sales data from a CRM (Customer Relationship Management)
and product data from an ERP (Enterprise Resource Planning) system into one
dataset.

C. Data Transformation
Definition: Converting data into a suitable format for data mining.
• Types of Data Transformation:
1. Normalization: Scaling data values into a specific range to ensure fairness and
balance.
▪ Methods:

▪ Example: If income ranges from $10,000 to $100,000, normalize it to [0,

1] for machine learning.
2. Smoothing: Reducing noise in data to improve accuracy. Methods like binning
and regression can smooth data.
3. Aggregation: Summarizing or combining data.
▪ Example: Aggregating daily sales data into monthly sales totals.
4. Attribute Construction: Creating new features or attributes from existing data
to make it more informative.
▪ Example: In a dataset with "height" and "weight," you can create a new
feature "BMI" (Body Mass Index).

D. Data Reduction
Definition: Reducing the size of the dataset while preserving its integrity, so it remains
useful for analysis.
• Purpose: To improve processing time and algorithm efficiency.
• Techniques:
 1. Data Cube Aggregation: Summarizing data into a data cube for
multidimensional analysis.
▪ Example: Aggregating total sales across regions, months, and product
categories.
2. Dimensionality Reduction: Reducing the number of attributes (features) in a
dataset while maintaining essential information.
▪ Methods:
▪ Principal Component Analysis (PCA)
▪ Linear Discriminant Analysis (LDA)
▪ Example: If you have 20 features in a dataset, PCA might reduce them to
5 key components.
3. Data Compression: Storing data in a compact form using encoding techniques.
▪ Example: Encoding categorical values as integers.
4. Numerosity Reduction: Representing data with fewer records through
sampling or clustering.

E. Data Discretization
Definition: Converting continuous numerical data into categorical (discrete) intervals.
• Purpose: Simplifies data for analysis and algorithms that work better with categorical
inputs.
• Techniques:
1. Binning: Dividing data into intervals or bins.
▪ Equal-width Binning: Bins have equal size (e.g., [0-10], [10-20]).
▪ Equal-frequency Binning: Bins have equal data points.
2. Histogram Analysis: Grouping data into frequency intervals.
• Example: Age data can be discretized into categories: 0–18 (child), 19–35 (youth), 36–
60 (adult), 60+ (senior).

F. Concept Hierarchy Generation

Definition: Organizing data attributes into a hierarchy, from low-level to high-level concepts.
• Purpose: Helps in summarizing and analysing data at different levels of abstraction.
• Example:
o Location Hierarchy: Street → City → State → Country
o Product Hierarchy: Item → Category → Department

Conclusion
data preprocessing forms/methods:
→ data cleaning (missing value, noisy data, inconsistent data)
→ data integration
→ data transformation (normalization, smoothing, agg)
→ data reduction (dim. Red, data compression)
→ data discretization (binning, hist. analysis)
→ hierarchy gen
# Data Cleaning—focusing on missing values, noisy data, and the methods used
to address them:

Data Cleaning Overview

Data cleaning is the process of identifying and correcting errors, inconsistencies, and
inaccuracies in data to ensure it is reliable and usable. Raw data collected from different
sources often has issues like missing values, noise (errors or outliers), and inconsistencies.
These issues, if not resolved, can negatively impact data mining, machine learning models,
and decision-making.

Handling Missing Values

Definition: Missing values occur when no data is available for one or more attributes in a
dataset. They can arise due to:
• Human errors during data entry.
• Faulty sensor readings or software issues.
• Intentional omission when data is unavailable.
Techniques for Handling Missing Values
1. Ignore the Tuple
o Remove rows with missing values.
o When to use:
▪ If the dataset is large and only a small fraction of data has missing values.
o Risk:
▪ If missing values are widespread, deleting data may cause loss of
valuable information.
o Example:
In a 10,000-row dataset, if 50 rows have missing values, they can be safely
removed.
2. Fill in Missing Values Manually
o Replace missing values manually with estimated values.
o When to use:
▪ For small datasets.
▪ Requires domain expertise.
o Example:
Filling missing ages in a dataset manually based on similar entries.
3. Replace Missing Values with a Constant or Default
o Use a fixed value (e.g., “Unknown” for categorical attributes or “0” for
numerical attributes).
o Example:
In a dataset of salaries, missing values can be replaced with 0 or “Not
Disclosed”.
4. Replace Missing Values with Statistical Measures
 o Numerical attributes:
▪ Replace missing values with the mean, median, or mode of the attribute.
o Categorical attributes:
▪ Use the most frequent category (mode).
o Example:
If income data is missing for some records, replace it with the average income
(e.g., ₹50,000).
5. Predict Missing Values Using Models
o Use machine learning models like regression or classification to predict missing
values based on other available data.
o Example:
Predict a student’s missing grade based on attendance and previous
performance.
6. Use Probabilistic Methods
o Use probability distributions (e.g., Bayesian methods) to estimate the missing
values.

Handling Noisy Data

Definition: Noisy data refers to data that contains errors, outliers, or irrelevant variations.
It can arise due to:
• Sensor malfunctions.
• Data entry errors.
• Unintended deviations during data collection.
Example of Noise:
• A person’s age recorded as 200 or salary as ₹1,000,000,000 in a general dataset.

Techniques to Handle Noisy Data

1. Binning
o Definition: Binning involves sorting data values into equal-sized intervals (bins)
and smoothing the values within each bin.
o Steps:
▪ Sort the data into ascending order.
▪ Divide the data into equal-sized bins.
▪ Replace the values in each bin using one of the following methods:
▪ Bin Mean: Replace all values in a bin with the mean (average).
▪ Bin Median: Replace all values with the median of the bin.
▪ Bin Boundaries: Replace each value with the closest boundary
(minimum or maximum).
o Example:
Original Data: [4, 8, 15, 21, 22, 23, 25]
▪ Divide into 3 bins: [4, 8], [15, 21], [22, 23, 25]
▪ Replace with bin mean: [6, 18, 23.33]
2. Clustering
o Definition: Group data points into clusters. Outliers (noise) are treated as data
points that do not belong to any cluster or fall far from the cluster centroid.
o Method:
▪ Apply clustering algorithms like K-Means or DBSCAN.
▪ Remove or smooth points far from clusters.
o Example:
In a dataset of house prices, clusters may group houses priced between
₹50,000 to ₹1,00,000, but a price like ₹10,000,000 may be an outlier.

3. Regression
o Definition: Use regression models to smooth noisy data by fitting a line or
curve.
o Types:
▪ Linear Regression: Fits a straight line.
▪ Polynomial Regression: Fits a curve.
o Example:
If sales data fluctuates with noise, regression can estimate a trend line to
smooth the values.
4. Computer and Human Inspection
o Computer Inspection: Use automated scripts or tools to detect and flag noisy
data.
o Human Inspection: Experts manually review small datasets to identify and fix
errors.
o When to use:
▪ For small datasets or critical data.
o Example:
In medical data, experts manually check abnormal entries like inconsistent
patient records.
5. Smoothing Techniques
o Moving Average: Replace each value with the average of neighboring values
over a specified window.
o Weighted Moving Average: Assign higher weights to recent values.
o Example:
Daily temperature data can be smoothed by averaging values over 3 days.
Summary of Key Concepts
Issue Technique Description Example
Delete rows/columns with missing Remove rows with null
Remove Data
data. values.
Replace missing salaries with
Replace with Default Use constant values like 0 or “N/A.”
Missing ₹0.
Values Replace with Replace with mean or mode Fill missing age with average
Mean/Median/Mode values. age = 30.
Use regression or ML models to Predict income based on
Predict Values
predict. education level.
Smooth data using mean, median, Replace [15, 20, 22] with [20,
Binning
or boundaries. 20, 20].
Remove outliers using clustering Exclude a house price of
Clustering
methods. ₹10,000,000.
Noisy Data
Fit a regression line to smooth Sales trends smoothed using
Regression
data. regression.
Apply moving average to reduce Smooth daily temperatures
Smoothing
variations. over a week.

Conclusion
Handling missing values and noisy data is an essential step in data cleaning to improve the
quality and accuracy of datasets. By using techniques like binning, clustering, regression,
and statistical measures, raw data can be prepared effectively for data mining and analysis.
This process ensures that algorithms work efficiently and produce reliable results.
# Inconsistent Data, Data Integration, and Data Transformation as they relate to the data
cleaning process in data mining:

1. Inconsistent Data
Definition
Inconsistent data refers to data that conflicts within the dataset or with other datasets due
to errors, discrepancies, or lack of standardization. This inconsistency can impact data
analysis and decision-making.

Sources of Inconsistent Data

1. Format Issues: Data stored in different formats across systems.
o Example: Date formats like DD/MM/YYYY vs MM/DD/YYYY.
2. Duplicate Data: Repeated entries or records with slight variations.
o Example: John Doe and John D., both referring to the same person.
3. Unit Inconsistencies: Mismatches in units of measurement.
o Example: Weight in kilograms (kg) vs pounds (lb).
4. Naming Conflicts: Variations in attribute names across datasets.
o Example: CustomerID in one table and Cust_ID in another.
5. Contradictory Values: Two or more entries with conflicting data.
o Example: A product listed as “In Stock” in one table and “Out of Stock” in
another.

Techniques to Resolve Inconsistent Data

1. Data Standardization
o Definition: Bringing data into a consistent format.
o Steps:
▪ Define standardized rules (e.g., date format YYYY-MM-DD).
▪ Apply the same units of measurement.
o Example:
Convert all currency values to a single standard like USD.
2. Removing Duplicates
o Identify and eliminate duplicate records.
o Use tools like SQL’s DISTINCT clause or clustering techniques to group similar
records.
o Example:
▪ Data: John Doe, John D. → Deduplicate to John Doe.
3. Data Reconciliation
o Correct contradictory data by verifying the source of truth.
o Cross-check data against reliable references.
o Example:
If one table says a product has 10 units and another says 0 units, verify
inventory records.
 4. Naming Harmonization
o Resolve differences in attribute naming conventions.
o Use mapping techniques to unify names across datasets.
o Example:
Map Customer_ID and Cust_ID to a common name CustomerID.

2. Data Integration
Definition
Data integration is the process of combining data from multiple sources into a unified,
consistent, and accurate dataset. This step ensures data from various systems, formats, or
databases is ready for analysis.

Steps in Data Integration

1. Schema Integration
o Merging schemas (structures) of different datasets.
o Resolve conflicts like:
▪ Attribute Conflicts: Same concept with different names (e.g., Price vs
Cost).
▪ Data Type Conflicts: Different types for the same attribute (e.g., integer
vs string).
o Example:
Combine two datasets where one has CustomerID as an integer and the other
as a string.
2. Entity Identification
o Identify and map records referring to the same entity across datasets.
o Tools like primary keys or clustering algorithms help detect duplicates.
o Example:
Combine John Doe in Table A and John D. in Table B as one entity.
3. Data Redundancy
o Remove redundant data to avoid duplication.
o Use normalization techniques to restructure data.
o Example:
Remove duplicate customer entries across sales and support tables.
4. Data Source Alignment
o Align data from different sources based on a common key (e.g., customer ID or
timestamp).
o Example:
Combine sales data (Table A) and customer support tickets (Table B) using a
common CustomerID.
Benefits of Data Integration
• Creates a single version of truth for analysis.
• Reduces data redundancy and inconsistency.
• Enables a unified view across systems (e.g., ERP, CRM, databases).

3. Data Transformation
Definition
Data transformation involves converting raw data into a format suitable for data mining and
analysis. It includes processes like normalization, aggregation, and feature engineering to
make data consistent and useful.

Techniques of Data Transformation

1. Normalization
o Scaling numeric values to a common range (e.g., [0, 1]).
o Techniques:
▪ Min-Max Normalization
▪ Z-Score Normalization
o Example:
▪ Raw values: [20, 40, 60, 80]
▪ Scaled values (0–1): [0, 0.33, 0.67, 1]
2. Aggregation
o Combining data at a higher level of abstraction.
o Example: Converting daily sales data into monthly sales.
3. Discretization
o Converting continuous numeric data into categorical bins.
o Example:
Convert age [0-100] into bins:
▪ 0–18 → Teen
▪ 19–60 → Adult
▪ 61+ → Senior
4. Generalization
o Replacing low-level data with higher-level concepts.
o Example: Replacing exact addresses with city names.
5. Smoothing
o Reducing noise in data using techniques like binning, regression, or moving
averages.
o Example: Smoothing stock prices to remove daily fluctuations.
6. Feature Construction
o Deriving new features (attributes) from existing data.
o Example:
▪ Existing attributes: Height and Weight.
▪ New feature: BMI = Weight / Height².
Examples of Transformation
Original Data Transformation Technique Result
[20, 40, 60] Min-Max Normalization [0, 0.5, 1]
[Jan Sales: 500, Feb Sales: 600] Aggregation Total Sales: 1100
Age: 23 Discretization Age Group: Adult
₹50,000 Generalization “High Income”

Summary
Inconsistent Data
• Deals with errors like format mismatches, naming conflicts, and contradictory
values.
• Resolved using standardization, reconciliation, and duplicate removal.
Data Integration
• Combines data from multiple sources into a unified view.
• Handles schema conflicts, entity identification, and redundancy.
Data Transformation
• Converts data into a suitable format for analysis.
• Includes normalization, aggregation, discretization, and feature engineering.
By resolving inconsistencies, integrating data, and transforming it into a usable format, data
mining models can operate efficiently, resulting in accurate insights and improved decision-
making.
# Data Reduction

What is Data Reduction?

Definition
Data reduction refers to the process of reducing the volume of data while maintaining its
integrity and analytical quality. It helps in efficiently storing, processing, and analysing large
datasets without losing valuable patterns and information.
Why is Data Reduction Important?
• Reduces computational costs.
• Speeds up data mining algorithms.
• Removes redundancy and irrelevant data.
• Makes large-scale data manageable for visualization and interpretation.

1. Data Cube Aggregation

Definition
Data cube aggregation is the process of summarizing data in a multi-dimensional space by
grouping and aggregating it. This is especially useful in OLAP (Online Analytical Processing)
systems, where data is analysed across dimensions.

Key Concepts
1. Data Cube: A data structure that organizes data across multiple dimensions, allowing
analysis at different levels of abstraction.
o Example Dimensions: Time, Location, Product Category, etc.
2. Aggregation: Performing summary operations (like sum, count, average, etc.) on data
grouped by specific dimensions.

Example
Imagine a sales dataset with three dimensions:
• Time: Year, Month
• Location: City, Country
• Product: Category, Brand
Time Location Product Sales
Jan New York Electronics 1000
Feb New York Electronics 1200
Jan New York Furniture 800
Step 1: Aggregation by Time
Summarize data by month:
Month Total Sales
Jan 1800
Feb 1200
Step 2: Aggregation by Location
Summarize data by city:
 City Total Sales
New York 3000

Benefits of Data Cube Aggregation

• Reduces the size of data by summarizing at higher levels.
• Supports multi-dimensional analysis.
• Improves query response time in large datasets.

2. Dimensionality Reduction
Definition
Dimensionality reduction refers to reducing the number of attributes (features) in the
dataset while preserving important information.

Why Dimensionality Reduction?

• High-dimensional data can lead to overfitting in machine learning models.
• Reduces computational time and memory usage.
• Helps in visualizing data effectively (e.g., reducing 3D data to 2D).

Techniques
1. Principal Component Analysis (PCA)
o Projects data onto a new set of axes (principal components) that maximize
variance.
o Reduces redundancy by combining correlated features.
2. Linear Discriminant Analysis (LDA)
o Focuses on maximizing class separability while reducing dimensions.
3. Feature Selection
o Selects the most relevant features using methods like:
▪ Filter Methods (e.g., Correlation).
▪ Wrapper Methods (e.g., Recursive Feature Elimination).

Example
Imagine a dataset with 10 features, but only 3 contribute significantly to predicting the
target variable. PCA can reduce these 10 features to the top 3 principal components.
Features After PCA
10 Attributes 3 Components

3. Data Compression
Definition
Data compression reduces the storage space required for data by encoding or reorganizing
it in a way that retains its meaning.
Techniques
1. Lossless Compression
o Compresses data without losing any information.
o Techniques: Huffman coding, Run-length encoding.
2. Lossy Compression
o Reduces size by removing insignificant or redundant data.
o Used in multimedia data (images, videos).

Example
• Original Dataset: AAAAABBBCC
• Compressed Dataset (Lossless): 5A3B2C

Benefits
• Saves storage space.
• Reduces data transmission time.
• Improves performance of data processing tasks.

4. Numerosity Reduction
Definition
Numerosity reduction involves representing data using a smaller set of numbers. It
approximates the original data without losing important patterns.

Techniques
1. Parametric Methods
o Assume that the data fits a model and represents it using model parameters.
o Example: Using a regression equation to approximate data points.
2. Non-Parametric Methods
o Store a reduced representation of the data, such as:
▪ Histograms
▪ Clustering (e.g., k-means)

Example
• Original Data: 1, 2, 3, 4, 5, 6
• Parametric: Approximate as y = x + 1
• Non-Parametric: Represent using clusters:
o Cluster 1: {1, 2} → 1.5
o Cluster 2: {3, 4} → 3.5

5. Discretization and Concept Hierarchy Generation

Definition
Discretization involves converting continuous data into categorical data (intervals or bins).
Concept hierarchy organizes data into levels of abstraction.
Techniques
1. Binning
o Divide data into fixed intervals (bins).
o Example: Age [0-100] → Bins: 0-18 (Teen), 19-60 (Adult), 61+ (Senior).
2. Cluster-Based Discretization
o Use clustering algorithms to form groups (bins).
3. Concept Hierarchy
o Organizes attributes into hierarchical levels.

Example: Concept Hierarchy

Level Example
Country India
State Karnataka
City Bangalore

Summary
Data Reduction
Data reduction helps in simplifying data for efficient storage, analysis, and computation.
1. Data Cube Aggregation: Summarizes data across multiple dimensions.
2. Dimensionality Reduction: Reduces the number of attributes using PCA, LDA, etc.
3. Data Compression: Reduces the physical size of data.
4. Numerosity Reduction: Represents data with fewer numbers using parametric or
non-parametric methods.
5. Discretization & Concept Hierarchy: Converts continuous data into categories and
organizes it hierarchically.
These techniques ensure that large datasets become manageable while retaining key
insights.
Decision Tree - Detailed Explanation

What is a Decision Tree?

A Decision Tree is a tree-like structure used to model decisions and their possible
consequences. It is widely used in classification and regression tasks within data mining and
machine learning.
• It divides data into smaller subsets based on specific criteria or decisions.
• The structure resembles a flowchart where each node represents a decision (test on
an attribute), each branch represents an outcome of that test, and each leaf node
represents a class label or predicted value.

Key Components of a Decision Tree

1. Root Node:
o The topmost node representing the entire dataset.
o A decision is made here to split the dataset based on the most significant
feature.
2. Decision Nodes:
o Intermediate nodes where a test is performed on one of the
attributes/features.
3. Branches:
o Arrows connecting the nodes that represent outcomes of a test/decision.
4. Leaf Nodes:
o The terminal nodes that represent the final output/class label for a subset of
data.

How Does a Decision Tree Work?

The process of constructing a Decision Tree involves splitting the data into smaller and
smaller subsets until an appropriate stopping condition is reached. At each step, the goal is
to identify the best attribute to split the data based on some criterion.

Key Steps in Building a Decision Tree

1. Select the Best Attribute to Split:
o Choose an attribute that maximizes information gain (or similar metrics).
2. Split the Data:
o Divide the data into subsets based on the selected attribute’s values.
3. Repeat for Child Nodes:
o Recursively repeat the process for each subset until all data belongs to a single
class (pure) or another stopping condition is reached.

Splitting Criteria
1. Gini Index:
o Measures the impurity of a dataset.
 o Lower Gini index → Higher purity → Better split.

2. Entropy and Information Gain (used in ID3 Algorithm):

o Entropy measures the level of disorder (impurity) in a dataset.
o Information Gain measures the reduction in entropy after a split:
3. Reduction in Variance (for Regression Trees):
o Minimizes variance in the target variable.

Types of Decision Trees

1. Classification Tree:
o Target variable is categorical.
o Each leaf node represents a class label.
2. Regression Tree:
o Target variable is continuous.
o Each leaf node represents a numeric value (average of data points in the
subset).

Example of a Decision Tree

Problem: Predict if a person buys a laptop based on income and age.
Age Income Buys Laptop (Y/N)
25 High Yes
20 Low No
30 Medium Yes
35 High Yes
18 Low No
Step 1: Start at the Root
We analyse which feature (Age or Income) best splits the data. Assume Income provides
higher Information Gain.
Step 2: Split the Data
• Income = Low → Buys Laptop = No
• Income = Medium → Buys Laptop = Yes
• Income = High → Further split on Age.
Step 3: Build Tree
Final Decision Tree:
• If Income = Low, → No.
• If Income = Medium, → Yes.
• If Income = High and Age < 30, → Yes.
• If Income = High and Age > 30, → Yes.

Advantages of Decision Trees

1. Easy to Understand: Visual representation makes interpretation simple.
2. Handles Both Numerical and Categorical Data: Versatile for all types of features.
3. Non-Linear Relationships: Captures complex patterns without requiring assumptions.
4. Feature Importance: Identifies which features are most significant.

Disadvantages of Decision Trees

1. Overfitting:
o Trees can become too complex, leading to poor generalization.
2. Instability:
o Small changes in data can cause drastic changes in the tree.
3. Bias Towards Features with More Levels:
o Features with many categories may dominate the splits.

Pruning in Decision Trees

Pruning is a technique used to prevent overfitting by removing parts of the tree that
provide little value.
• Pre-Pruning: Stop growing the tree early based on a threshold.
• Post-Pruning: Allow the full tree to grow and then prune nodes with low significance.

Decision Tree Algorithms

1. ID3 (Iterative Dichotomiser 3):
o Uses Information Gain for splitting.
2. C4.5:
o An extension of ID3, it handles missing values and uses Gain Ratio to avoid
bias.
3. CART (Classification and Regression Trees):
o Uses Gini Index for classification and Reduction in Variance for regression.

Applications of Decision Trees

1. Customer Segmentation: Identify potential customers based on behavior.
2. Medical Diagnosis: Predict disease based on symptoms.
3. Credit Risk Analysis: Assess loan approval based on applicant attributes.
4. Fraud Detection: Classify transactions as genuine or fraudulent.
Conclusion
A Decision Tree is an intuitive and powerful tool for classification and regression problems.
It systematically breaks down data using decision rules, allowing for easy interpretation and
analysis. However, proper pruning and tuning of parameters are necessary to avoid
overfitting and ensure accuracy in predictions.