Data Mining: Data Pre-

processing

Unit-1

Introduction to Data Mining

1
Outline

 Attributes and Objects

 Types of Data

 Data Quality

 Similarity and Distance

 Data Preprocessing

2
What is Data?

 Collection of data objects Attributes

and their attributes is known
as data set.
 An attribute is a property or
characteristic of an object
– Examples: eye color of a
person, temperature, etc.
– Attribute is also known as
variable, field, characteristic,

Objects
dimension, or feature
 A collection of attributes
describe an object
– Object is also known as
record, point, case, sample,
entity, or instance
Attribute Values

 Attribute values are numbers or symbols assigned to an attribute for

a particular object. For e.g eye color possible value{ brown, black,
blue, green, etc.} while temperature is numerical.

 Distinction between attributes and attribute values

– Same attribute can be mapped to different attribute values
 Example: height can be measured in feet or meters

– Different attributes can be mapped to the same set of values

 Example: Attribute values for ID and age are integers

– But properties of attribute can be different than the

properties of the values used to represent the attribute . For e.g.
we can find average age of an employee but not average of
employee id. For the age attribute, the properties of the integers
used to represent an age are very much the properties of the
attribute. 4
 Types of Attributes

 There are different types of attributes

– Nominal
 Examples: ID numbers, eye color, zip codes
– Ordinal
 Examples: rankings (e.g., taste of potato chips on a
scale from 1-10), grades, height {tall, medium, short}
– Interval
 Examples: calendar dates, temperatures in Celsius or
Fahrenheit.
– Ratio
 Examples: temperature in Kelvin, length, counts,
elapsed time (e.g., time to run a race)
5
Types of Attributes

6
Properties of Attribute Values

 The type of an attribute depends on which of the

following properties/operations it possesses:
– Distinctness: =
– Order: 
– Addition or <
+
difference are meaniful ->:
– Multiplication and division are *
meaningful /
– Nominal attribute: distinctness
– Ordinal attribute: distinctness & order
– Interval attribute: distinctness, order & meaningful
differences and addition
– Ratio attribute: all 4 properties/operations

7
 Attribut Description Examples Operations
e Type
Nominal Nominal zip codes, employee mode, entropy,
attribute values ID numbers, eye contingency
only color, sex: {male, correlation, 
distinguish. (=, ) female} 2 test
Categorical
Qualitative

Ordinal Ordinal attribute hardness of minerals, median,

values also {good, better, best}, percentiles,
order objects. grades, street rank
(<, >) numbers correlation, run
tests, sign tests
Interval For interval calendar dates, mean, standard
attributes, temperature in deviation,
differences between Celsius or Pearson's
Quantitative
Numeric

values are Fahrenheit correlation, t

meaningful. (+, - ) and F tests
Ratio For ratio variables, temperature in Kelvin, geometric mean,
both differences and monetary harmonic mean,
ratios are quantities, counts, percent
meaningful. (*, /) age, mass, length, variation
current
This categorization of attributes is due to S. S. Stevens
 Attribute Transformation Comments
Type
Nominal Any permutation of values If all employee ID numbers
were reassigned, would it
make any difference?
Categorical
Qualitative

Ordinal An order preserving change of An attribute encompassing

values, i.e., the notion of good, better best
new_value = f(old_value) can be represented equally
where f is a monotonic function well by the values {1, 2, 3} or
by { 0.5, 1, 10}.

Interval new_value = a * old_value + b Thus, the Fahrenheit and

where a and b are constants Celsius temperature scales
differ in terms of where their
Quantitative
Numeric

zero value is and the size of a

unit (degree).
Ratio new_value = a * old_value Length can be measured in
meters or feet.

This categorization of attributes is due to S. S. Stevens

Discrete and Continuous Attributes

 Discrete Attribute
– Has only a finite or countably infinite set of values
– Examples: zip codes, counts, or the set of words in a
collection of documents
– Often represented as integer variables.
– Note: binary attributes are a special case of discrete
attributes
 Continuous Attribute
– Has real numbers as attribute values
– Examples: temperature, height, or weight.
– Practically, real values can only be measured and
represented using a finite number of digits.
– Continuous attributes are typically represented as floating-
point variables.
Asymmetric Attributes

 Only presence (a non-zero attribute value) is regarded as

important
 Words present in documents
 Items present in customer transactions

 For e.g. Consider a data set where each object is a student and each attribute
records whether or not a student took a particular course at a university. For a specific
student, an attribute has a value of 1 if the student took the course associated with that
attribute and a value of 0 otherwise. Because students take only a small fraction of all
available courses, most of the values in such a data set would be 0. Therefore, it is
more meaningful and more efficient to focus on the non-zero values.
Critiques of the attribute categorization

 Incomplete
– Asymmetric binary: Binary attribute where only non-zero value is
important.
– Cyclical : A cyclic attribute has values that repeat in a period of
time. Ex. hour, week, year.
– Multivariate : multivalued attribute

 Real data is approximate and noisy

– This can complicate recognition of the proper attribute type
– Treating one attribute type as another may be approximately
correct
Important Characteristics of Data

– Dimensionality (number of attributes)

 High dimensional data brings a number of challenges

– Distribution (frequency of occurrence)

– Sparsity
 Only presence counts

– Resolution
 Patterns depend on the scale

– Size
 Type of analysis may depend on size of data
Types of data sets
 Record
– Data Matrix
– Document Data
– Transaction Data
 Graph
– World Wide Web
– Molecular Structures
 Ordered
– Spatial Data
– Temporal Data
– Sequential Data
– Genetic Sequence Data
Record Data

 Data that consists of a collection of records, each

of which consists of a fixed set of attributes
Tid Refund Marital Taxable
Status Income Cheat

1 Yes Single 125K No

2 No Married 100K No
3 No Single 70K No
4 Yes Married 120K No
5 No Divorced 95K Yes
6 No Married 60K No
7 Yes Divorced 220K No
8 No Single 85K Yes
9 No Married 75K No
10 No Single 90K Yes
10
Data Matrix or Pattern Matrix

 If data objects have the same fixed set of numeric

attributes, then the data objects can be thought of as
points in a multi-dimensional space, where each
dimension represents a distinct attribute

 Such a data set can be represented by an m by n matrix,

where there are m rows, one for each object, and n
columns, one for each attribute
Projectio Projection Distance Load Thickness
n of x of y load
Load

10.23 5.27 15.22 2.7 1.2

12.65 6.25 16.22 2.2 1.1
Document Data

 Each document becomes a ‘term’ vector

– Each term is a component (attribute) of the vector
– The value of each component is the number of times
the corresponding term occurs in the document.

timeout

season
coach

game
score
team

play

lost
ball

win
Document 1 3 0 5 0 2 6 0 2 0 2

Document 2 0 7 0 2 1 0 0 3 0 0

Document 3 0 1 0 0 1 2 2 0 3 0
Transaction Data

 A special type of data, where

– Each transaction involves a set of items.
– For example, consider a grocery store. The set of products
purchased by a customer during one shopping trip constitute a
transaction, while the individual products that were purchased
are the items.
– Can represent transaction data as record data

TID Items
1 Bread, Coke, Milk
2 Beer, Bread
3 Beer, Coke, Diaper, Milk
4 Beer, Bread, Diaper, Milk
5 Coke, Diaper, Milk
 Graph Data

 Examples: Generic graph, a molecule, and webpages

2
5 1
2
5

Benzene Molecule: C6H6

Ordered Data

 Sequences of transactions
Ordered Data

 Genomic sequence data

GGTTCCGCCTTCAGCCCCGCGCC
CGCAGGGCCCGCCCCGCGCCGTC
GAGAAGGGCCCGCCTGGCGGGCG
GGGGGAGGCGGGGCCGCCCGAGC
CCAACCGAGTCCGACCAGGTGCC
CCCTCTGCTCGGCCTAGACCTGA
GCTCATTAGGCGGCAGCGGACAG
GCCAAGTAGAACACGCGAAGCGC
TGGGCTGCCTGCTGCGACCAGGG
Ordered Data

 Spatio-Temporal Data

Average Monthly
Temperature of
land and ocean
Major Tasks in Data Preprocessing

Data cleaning
Fill in missing values, smooth noisy data, identify or remove outliers, and
resolve inconsistencies
Data integration
Integration of multiple databases, data cubes, or files
Data reduction
Dimensionality reduction
Numerosity reduction
Data compression
Data transformation and data discretization
Normalization
Concept hierarchy generation

23
Data Quality

 Poor data quality negatively affects many data processing

Efforts (Garbage in garbage out)

 Data mining example: a classification model for detecting

people who are loan risks is built using poor data
– Some credit-worthy candidates are denied loans
– More loans are given to individuals that default
Data Quality …

 What kinds of data quality problems?

 Why these data quality problem occur?
 How can we detect problems with the data?
 What can we do about these problems?
Data Cleaning: the detection and correction of
data quality.
 Examples of data quality problems:
– Noise and outliers
– Wrong data
– Fake data
– Missing values
– Duplicate data
 Measurement and Data Collection Issue

• It is unrealistic to expect that data will be perfect. There may

be problems due to human error, limitations of measuring
devices, flaws in the data collection process or transmission
error.
• The term measurement error refers to any problem resulting
from the measurement process. A common problem is that the
value recorded differs from the true value to some extent.

• The term data collection error refers to errors such as

omitting data objects or attribute values, or inappropriately
including a data object.
Missing Values
 Reasons for missing values
– Information is not collected (e.g., people decline to give their age and weight)

– Attributes may not be applicable to all cases (e.g., annual income is not applicable
to children)
 Handling missing values
– Eliminate data objects or tuple
– Fill the missing value manually S.No Actual Value Mean Median Mode
– Estimate missing values 1 67 67 67 67

 By attribute mean/ median

2 51 58 67
3 67 67 67 67
 Assign a global constant such as - 4 56 56 56 56

– Ignore the missing value during analysis 5 58 58 58 58

6 48 48 48 48
7 89 89 89 89
8 51 58 67
9 74 74 74 74

Mean = (67+67+56+58+48+89+74)/9=51
Median = 48, 56,58,67,74,89 = 58
Mode (most frequent occur value
Duplicate Data

 Data set may include data objects that are

duplicates, or almost duplicates of one another
– Major issue when merging data from heterogeneous
sources
 Examples:
– Same person with multiple email addresses
Inconsistent Data
Noise

 For objects, noise is an extraneous object or outliers

 For attributes, noise refers to modification of original values
– Examples: distortion of a person’s voice when talking on a poor phone
and “snow” on television screen
– The figures below show two sine waves of the same magnitude and
different frequencies, the waves combined, and the two sine waves with
random noise
 The magnitude and shape of the original signal is distorted
Outliers

 Outliers are data objects with characteristics that

are considerably different than most of the other
data objects in the data set
– Outliers are noise
that interferes with
data analysis
How to Handle Noisy Data?

Binning
first sort data and partition into
(equal-frequency) bins
then one can smooth by bin means,
smooth by bin median, smooth by bin
boundaries, etc.
Regression
smooth by fitting the data into regression functions
Clustering
detect and remove outliers
Combined computer and human inspection
detect suspicious values and check by human (e.g., deal with
possible outliers)

32
Similarity and Dissimilarity Measures

 Similarity measure
– Numerical measure of how alike two data objects are.
– Is higher when objects are more alike.
– Often falls in the range [0,1]
 Dissimilarity measure
– Numerical measure of how different two data objects
are
– Lower when objects are more alike
– Minimum dissimilarity is often 0
– Upper limit varies
 Proximity refers to a similarity or dissimilarity
 Similarity/Dissimilarity for Simple Attributes

The following table shows the similarity and dissimilarity

between two objects, x and y, with respect to a single, simple
attribute.
Euclidean Distance

 Euclidean Distance

where n is the number of dimensions (attributes) and

xk and yk are, respectively, the kth attributes
(components) or data objects x and y.
 Standardization is necessary, if scales differ.
Consider the following points and attribute . Calculate the Euclidean distance

point x y
p1 0 2
p2 2 0
p3 3 1
p4 5 1
Euclidean Distance

2 p1
p3 p4
1
p2
0
01 2 3 5
4 6
p1 p2 p3 p4
p1 0 2.828 3.162 5.099
p2 2.828 0 1.414 3.162
p3 3.162 1.414 0 2
p4 5.099 3.162 2 0
Distance Matrix
Minkowski Distance

 Minkowski Distance is a generalization of Euclidean

Distance

Where r is a parameter, n is the number of dimensions

(attributes) and xk and yk are, respectively, the kth
attributes (components) or data objects x and y.
Minkowski Distance: Examples

 r = 1. City block (Manhattan, taxicab, L1 norm) distance.

– A common example of this for binary vectors is the
Hamming distance, which is just the number of bits that are
different between two binary vectors

 r = 2. Euclidean distance

 r  . “supremum” (Lmax norm, L norm) distance.

– This is the maximum difference between any component of
the vectors

 Do not confuse r with n, i.e., all these distances are

defined for all numbers of dimensions.
Minkowski Distance

L1 p1 p2 p3 p4
p1 0 4 4 6
p2 4 0 2 4
p3 4 2 0 2
p4 6 4 2 0
point x y
p1 0 2 L2 p1 p2 p3 p4
p2 2 0 p1 0 2.828 3.162 5.099
p3 3 1 p2 2.828 0 1.414 3.162
p4 5 1 p3 3.162 1.414 0 2
p4 5.099 3.162 2 0
L p1 p2 p3 p4
p1 0 2 3 5
p2 2 0 1 3
p3 3 1 0 2
p4 5 3 2 0
Distance Matrix
Common Properties of a Distance

 Distances, such as the Euclidean distance,

have some well known properties.

1. d(x, y)  0 for all x and y and d(x, y) = 0 if and only

if x = y.
2. d(x, y) = d(y, x) for all x and y. (Symmetry)
3. d(x, z)  d(x, y) + d(y, z) for all points x, y, and z.
(Triangle Inequality)

where d(x, y) is the distance (dissimilarity) between

points (data objects), x and y.

 A distance that satisfies these properties is a

metric
Common Properties of a Similarity

 Similarities, also have some well known

properties.

1. s(x, y) = 1 (or maximum similarity) only if x = y.

(does not always hold, e.g., cosine)
1. s(x, y) = s(y, x) for all x and y. (Symmetry)

where s(x, y) is the similarity between points (data

objects), x and y.
 Mahalanobis Distance

For red points, the Euclidean distance is 14.7, Mahalanobis distance is

6.
 Mahalanobis Distance

Step 1: Input Raw Data

Step 2: Calculate Mean
Step 3 : Find Difference (x-m) and transponse (x-m)’
Step 4: Calculate Covariance matrix
Step 5 : Find Inverse of Covariance matrix
Step 6: Calculate Mahalabonis distance

A 1 2 4 2 5 X

4
B 100 300 200 600 100
500

40
C 10 15 20 10 30
 Mahalanobis Distance

mean
Calculate Mean 2.8
260
17

x-mean

Find Difference (x-m) and transpose (x-m)’ (x-mean)’ = 1.2 240 23

1.2

240

23

2.7 -110 13
Calculate Covariance matrix -110 43000 -900
13 -900 70

5.5 -0.01 -1.15

Find Inverse of Covariance matrix -0.01 0.0005 0.0025
-1.15 0.0025 5.2

= 106.7
5.5 -0.01 -1.15 1.2

Calculate Mahalabonis 1.2 240 23 X -0.01 0.0005 0.0025 X 240

distance -1.15 0.0025 5.2 23

MD = (106.7)1/2= 10.33
Similarity Between Binary Vectors
 Common situation is that objects, x and y, have only
binary attributes

 Compute similarities using the following quantities

 f01 = the number of attributes where x was 0 and y was 1

 f10 = the number of attributes where x was 1 and y was 0

 f00 = the number of attributes where x was 0 and y was 0

 f11 = the number of attributes where x was 1 and y was 1

 Simple Matching and Jaccard Coefficients

SMC = number of matches / number of attributes
= (f11 + f00) / (f01 + f10 + f11 + f00)
J = number of 11 matches / number of non-zero attributes
= (f11) / (f01 + f10 + f11)
Calcúlate SMC and Jaccard Coefficients of the following binary data
x= 1000000000
y= 0000001001
SMC versus Jaccard: Example

x= 1000000
000
y= 0000001
001
f01 = 2 (the number of attributes where x was 0 and y was 1)
f10 = 1 (the number of attributes where x was 1 and y was 0)
f00 = 7 (the number of attributes where x was 0 and y was 0)
f11 = 0 (the number of attributes where x was 1 and y was 1)

SMC = (f11 + f00) / (f01 + f10 + f11 + f00)

= (0+7) / (2+1+0+7) = 0.7

J = (f11) / (f01 + f10 + f11) = 0 / (2 + 1 + 0) = 0

Cosine Similarity
The cosine similarity, defined next, is one of the most common measure of document
similarity. If x and y are two document vectors, then

 Example:

 x= 3205000200

 y= 1000000102
Cosine Similarity

 x= 3205000200

 y= 1000000102

x. y = 3*1 + 2*0 + 0*0 + 5*0 + 0*0 + 0*0 + 0*0 + 2*1 + 0*0 + 0*2 = 5
||x || = (3*3+2*2+0*0+5*5+0*0+0*0+0*0+2*2+0*0+0*0)0.5 = (42) 0.5 = 6.481
|| y || = (1*1+0*0+0*0+0*0+0*0+0*0+0*0+1*1+0*0+2*2) 0.5 = (6) 0.5 = 2.449 cos(x
, y ) = 0.3150
Extended Jaccard Coefficient (Tanimoto coefficient)

X= (1,0,1,0,1)
Y=(1,1,1,0,1)
X.Y=1*1+0*1+1*1+0*0+1*1=3
||x||2 = ((1*1+0*0+1*1+0*0+1*1)1/2 )2 =3
||y||2 = ((1*1+1*1+1*1+0*0+1*1)1/2 )2 =4

EJ(x,y)=3/ (3+4-3)=3/4=0.75
Correlation measures the linear relationship
between objects

Find Correlation Coefficient X= (-3,6,0,3,-6) Y= (1,-2,0,-1,2)

Correlation measures the linear relationship between
objects

X= (-3,6,0,3,-6) Y= (1,-2,0,-1,2)
Mean of x= 0 Mean of y= 0 n=5

Cov(x,y) =(-3-12+0-3-12)/4 = -7.25

Sx= [ (9+36+9+36)/4]½ = 4.716

Sy = [ (1+4+1+4)/4]½ =1.5

Corr(x,y)= -7.25/(4.716+1.5) = -1

Correlation coefficients lies between -1 to 1

1 means perfect positive correlated
0 no correlation
-1 means perfect negative correlated
Correlation vs Cosine vs Euclidean Distance

 Compare the three proximity measures according to their behavior under

variable transformation
– scaling: multiplication by a value
– translation: adding a constant
Property Cosine Correlation Euclidean Distance
Invariant to scaling Yes Yes No
(multiplication)
Invariant to translation No Yes No
(addition)

 Consider the example

– x = (1, 2, 4, 3, 0, 0, 0), y = (1, 2, 3, 4, 0, 0, 0)
– ys = y * 2 (scaled version of y), yt = y + 5 (translated
version)
Measure (x , y) (x , ys) (x , yt)
Cosine 0.9667 0.9667 0.7940

Correlation 0.9429 0.9429 0.9429

Euclidean Distance 1.4142 5.8310 14.2127

Entropy

 For
– a variable (event), X,
– with n possible values (outcomes), x1, x2 …, xn
– each outcome having probability, p1, p2 …, pn
– the entropy of X , H(X), is given by

=− log2

=1
 log2(x)=ln(x)/ln(2)
Entropy Examples
Mutual Information

 Information one variable provides about another

Formally, , = +
− ( , ), where

H(X) is the entropy of X

H(Y) is the entropy of Y
H(X,Y) is the joint entropy of X and Y,
 Mutual Information Example

Student Count p -plog2p Student Grade Count p -plog2p

Status Status
Undergrad 45 0.45 0.5184
Undergrad A 5 0.05 0.2161
Grad 55 0.55 0.4744
Undergrad B 30 0.30 0.5211
Total 100 1.00 0.9928
Undergrad C 10 0.10 0.3322

Grade Count p -plog2p Grad A 30 0.30 0.5211

A 35 0.35 0.5301 Grad B 20 0.20 0.4644

B 50 0.50 0.5000 Grad C 5 0.05 0.2161
C 15 0.15 0.4105 Total 100 1.00 2.2710
Total 100 1.00 1.4406

Mutual information of Student Status and Grade = 0.9928 + 1.4406 - 2.2710 = 0.1624
Using Weights to Combine Similarities

 Can also define a weighted form of distance

Data Preprocessing

 Aggregation
 Sampling
 Discretization and Binarization
 Attribute Transformation
 Dimensionality Reduction
 Feature subset selection
 Feature creation
Aggregation

 Combining two or more attributes (or objects) into a single

attribute (or object)
 Purpose
– Data reduction - reduce the number of attributes or objects
– Change of scale
 Cities aggregated into regions, states, countries, etc.
 Days aggregated into weeks, months, or years
– More “stable” data - aggregated data tends to have less variability
– Data Compression
Sampling
 Sampling is the main technique employed for data
reduction.
– It is often used for both the preliminary investigation of
the data and the final data analysis.

 Statisticians often sample because obtaining the

entire set of data of interest is too expensive or
time consuming.

 Sampling is typically used in data mining because

processing the entire set of data of interest is too
expensive or time consuming.
Sampling …

 The key principle for effective sampling is the

following:

– Using a sample will work almost as well as using the

entire data set, if the sample is representative

– A sample is representative if it has approximately the

same properties (of interest) as the original set of data
Types of Sampling

Simple random sampling

There is an equal probability of selecting any particular item
Sampling without replacement
Once an object is selected, it is removed from the population
Sampling with replacement
A selected object is not removed from the population
Stratified sampling:
Partition the data set, and draw samples from each partition
(proportionally, i.e., approximately the same percentage of the
data)
Used in conjunction with skewed data

62
Sampling: With or without Replacement

W O R
SRS le random
i m p h o u t
( s e w i t
l
samp ment)
p l a c e
re

SRSW
R

Raw Data
63
Sampling: Cluster or Stratified Sampling

Raw Data Cluster/Stratified Sample

64
Sample Size

8000 points 2000 Points 500 Points

 Data Reduction Strategies

Data reduction: Obtain a reduced representation of the data set that is

much smaller in volume but yet produces the same (or almost the same)
analytical results
Why data reduction? — A database/data warehouse may store terabytes of
data. Complex data analysis may take a very long time to run on the
complete data set.
Data reduction strategies
Dimensionality reduction, e.g., remove unimportant attributes
Wavelet transforms
Principal Components Analysis (PCA)
Feature subset selection, feature creation
Numerosity reduction (technique to replace the original data volume by
alternative smaller form of data representation)
Types : Parameteric e.g Regression and Log-Linear Models
Non- Parameteric e.g Histograms, clustering, sampling, Data cube
aggregation
Data compression : transformation are applied so as to obtain reduced or
compressed data.
Type : lossy, lossless
66
 Curse of Dimensionality

 When dimensionality
increases, data becomes
increasingly sparse in the
space that it occupies

 Definitions of density and

distance between points,
which are critical for
clustering and outlier
detection, become less
meaningful
Dimensionality Reduction

 Purpose:
– Avoid curse of dimensionality
– Reduce amount of time and memory required by data
mining algorithms
– Allow data to be more easily visualized
– May help to eliminate irrelevant features or reduce
noise
 What Is Wavelet Transform?
The wavelet transforms the data can be truncated
and this is helpful in data reduction. If we store a
small fraction of the strongest wavelet
coefficients, then the compressed approximation
of the original data can be obtained. For example,
the wavelet coefficients larger than some
determined threshold can be retained.

Decomposes a signal into different frequency sub-

bands. Applicable to n-dimensional signals.
Data are transformed to preserve relative distance
between objects at different levels of resolution
Allow natural clusters to become more
distinguishable. Used for image compression

69
Wavelet Transformation
Haar2 Daubechie4
Discrete wavelet transform (DWT) for linear signal processing,
multi-resolution analysis
Compressed approximation: store only a small fraction of the
strongest of the wavelet coefficients
Similar to discrete Fourier transform (DFT), but better lossy
compression, localized in space
Method:
Length, L, must be an integer power of 2 (padding with 0’s, when
necessary)
Each transform has 2 functions: smoothing, difference
Applies to pairs of data, resulting in two set of data of length L/2
Applies two functions recursively, until reaches the desired length

70
 Wavelet Decomposition

Wavelets: A math tool for space-efficient hierarchical

decomposition of functions
S = [2, 2, 0, 2, 3, 5, 4, 4] can be transformed to S^ = [23/4, -11/4
, 1/2, 0, 0, -1, -1, 0]
Compression: many small detail coefficients can be replaced by
0’s, and only the significant coefficients are retained

71
Wavelet Decomposition & regeneration of Signal
Dimensionality Reduction: PCA

Principal Component Analysis is

an unsupervised learning
algorithm that is used for the
dimensionality reduction in
machine learning. It is a statistical
process that converts the
observations of correlated
features into a set of linearly
uncorrelated features with the
help of orthogonal transformation.
These new transformed features
are called the Principal
Components.
Dimensionality Reduction using PCA
Dimensionality Reduction using PCA
 Dimensionality Reduction using PCA

Step 2: Calculate Covariance matrix

Dimensionality Reduction using PCA
Dimensionality Reduction using PCA
Dimensionality Reduction using PCA
Dimensionality Reduction using PCA
Dimensionality Reduction using PCA
 Dimensionality Reduction using PCA

Geometrical meaning of first principal components

https://www.youtube.com/watch?v=ZtS6sQUAh0c
Feature Subset Selection

 Another way to reduce dimensionality of data

 Redundant features
– Duplicate much or all of the information contained in
one or more other attributes
– Example: purchase price of a product and the amount
of sales tax paid
 Irrelevant features
– Contain no information that is useful for the data
mining task at hand
– Example: students' ID is often irrelevant to the task of
predicting students' GPA
 Many techniques developed, especially for
classification
 Heuristic Search in Attribute Selection

There are 2d possible attribute combinations of d attributes

Typical heuristic attribute selection methods:
Best single attribute under the attribute independence
assumption: choose by significance tests
Best step-wise feature selection:
The best single-attribute is picked first
Then next best attribute condition to the first, ...
Step-wise attribute elimination:
Repeatedly eliminate the worst attribute
Best combined attribute selection and elimination
 Approaches for feature subset selection

Embedded approaches Feature selection occurs naturally as

part of the data mining algorithm. Specifically, during the
operation of the data mining algorithm, the algorithm itself
decides which attributes to use and which to ignore.

Filter approaches Features are selected before the data

mining algorithm is run, using some approach that is
independent of the data mining task.

Wrapper approaches These methods use the target data

mining algorithm as a black box to find the best subset of
attributes, in a way similar to that of the ideal algorithm
described above, but typically without enumerating all possible
subsets
An Architecture for Feature Subset Selection
Feature Creation

 Create new attributes that can capture the

important information in a data set much more
efficiently than the original attributes
 Three general methodologies:
– Feature extraction
 Creating a new set of features from original data set
 Example: extracting edges from images to detect human face
 Domain- specific
– Feature construction
 Creating a new set of features by combining original data set
 Example: dividing mass by volume to get density
– Mapping data to new space : to understanding
interesting and important features.
 Example: Fourier and wavelet analysis
Discretization

 Discretization is the process of converting a

continuous attribute into categorial attribute
– A potentially infinite number of values are mapped
into a small number of categories
– Discretization is used in both unsupervised
(class information unknown) and supervised
(class information known) settings
Unsupervised Discretization
Unsupervised Discretization

Equal width (binning)

Equal interval width
(binning)

Equal frequency (binning) K-means clustering leads to better results

90
Discretization in Supervised Settings

– Many classification algorithms work best if both the independent

and dependent variables have only a few values
– We give an illustration of the usefulness of discretization using
the following example.
Binarization

 Binarization maps a continuous or categorical

attribute into one or more binary variables

Binarization is used to prepare images for pattern recognition tasks, such

as fingerprint identification, where the focus is on the structure of the
object rather than its color or grayscale intensity.

The process of binarization involves the selection of a threshold value,

and then converting all pixel values below the threshold to 0 and all pixel
values above the threshold to 1.
Attribute Transformation/Variable Transformation

 An attribute transform is a function that maps the

entire set of values of a given attribute to a new
set of replacement values such that each old
value can be identified with one of the new values
– Simple functions: xk, log(x), ex, |x|
– Normalization
Normalization: Scaled to fall within a smaller, specified range
min-max normalization
z-score normalization
normalization by decimal scaling
Normalization

Min-max normalization: to [new_minA, new_maxA]

v  minA
v'  (new _ maxA  new _ minA)  new _ minA
maxA  minA
Ex. Let income range $12,000 to $98,000 normalized to [0.0, 1.0]. Then
$73,000 is mapped to 73,600  12,000
(1.0  0)  0  0.716
98,000  12,000

Z-score normalization (μ: mean, σ: standard deviation):

v  A
v'
A
Ex. Let μ = 54,000, σ = 16,000. Then 73,600  54,000
 1.225
16,000
Normalization by decimal scaling

v
v'  j Where j is the smallest integer such that Max(|ν’|) < 1
10
94
Normalization