CLASSIFICATION & PREDICTION

- Shailesh Yadav
Central University Of Rajasthan
CONTENTS
 Classification & Prediction
 Methods Of Classification
 Other Classification Methods
 Prediction
 Conclusion
Classification vs. Prediction

 Classification
 predicts categorical class labels (discrete or nominal)
 classifies data (constructs a model) based on the training set and the
values (class labels) in a classifying attribute and uses it in classifying
new data
 Prediction
 models continuous-valued functions, i.e., predicts unknown or missing
values
 Typical applications:-
1-Credit/loan approval: Medical diagnosis: if a tumor is cancerous or benign
2-Fraud detection: if a transaction is fraudulent
3-Web page categorization: which category it is
Classification—A Two-Step Process

 Model construction: describing a set of predetermined classes

 Each tuple /sample is assumed to belong to a predefined class, as determined
by the class label attribute
 The set of tuples used for model construction is training set
 The model is represented as classification rules, decision trees, or
mathematical formulae
 Model usage: for classifying future or unknown objects
 Estimate accuracy of the model
 The known label of test sample is compared with the classified result
from the model
 Accuracy rate is the percentage of test set samples that are correctly
classified by the model
 Test set is independent of training set, otherwise over-fitting will occur
 If the accuracy is acceptable, use the model to classify data tuples whose class
labels are not known
 Process (1): Model Construction

Classification
Algorithms
Training
Data

NAME RANK YEARS TENURED Classifier

Mike Assistant Prof 3 no (Model)
Mary Assistant Prof 7 yes
Bill Professor 2 yes
Jim Associate Prof 7 yes
Dave Assistant Prof 6 no IF rank = ‘professor’
Anne Associate Prof 3 no OR years > 6
THEN tenured = ‘yes’
 Process (2): Using the Model in Prediction

Classifier

Testing
Data Unseen Data

(Jeff, Professor, 4)
NAME RANK YEARS TENURED
Tom
Merlisa
Assistant Prof
Associate Prof
2
7
no
no
Tenured?
George Professor 5 yes
Joseph Assistant Prof 7 yes
 Issues: Data Preparation

 Data cleaning
 Preprocess data in order to reduce noise and handle
missing values
 Relevance analysis (feature selection)
 Remove the irrelevant or redundant attributes
 Data transformation
 Generalize and/or normalize data
Issues: Evaluating Classification Methods

 Accuracy
 classifier accuracy: predicting class label
 predictor accuracy: guessing value of predicted attributes
 Speed
 time to construct the model (training time)
 time to use the model (classification/prediction time)
 Robustness: handling noise and missing values
 Scalability: efficiency in disk-resident databases
 Interpretability
 understanding and insight provided by the model
 Other measures, e.g., goodness of rules, such as decision tree size or
compactness of classification rules
 Methods Of Classification

 By Decision Tree Induction

 Bayesian Classification
 Rule Based Classification
 Decision Tree Induction
 A decision tree induction is the learning of
decision trees from class- labeled training
tuples. A decision tree is a flowchart- like tree
structure, where each internal node (non leaf
node) denotes a test on an attribute, each
branch represents an outcome of the test, and
each leaf node (or terminal node) hold a class
label. The topmost node in a tree is the root
node.
 Decision Tree Induction: Training Dataset

age income student credit_rating buys_computer

This <=30
<=30
high
high
no
no
fair
excellent
no
no
follows 31…40 high no fair yes
>40 medium no fair yes
an >40 low yes fair yes
example >40
31…40
low
low
yes
yes
excellent
excellent
no
yes
of <=30 medium no fair no

Quinlan’s <=30
>40
low
medium
yes
yes
fair
fair
yes
yes
ID3 <=30 medium yes excellent yes
31…40 medium no excellent yes
(Playing 31…40 high yes fair yes
Tennis) >40 medium no excellent no
Output: A Decision Tree for “buys_computer”

age?

<=30 overcast
31..40 >40

student? yes credit rating?

no yes excellent fair

no yes no yes
Algorithm for Decision Tree Induction

 Basic algorithm (a greedy algorithm)

 Tree is constructed in a top-down recursive divide-and-conquer manner
 At start, all the training examples are at the root
 Attributes are categorical (if continuous-valued, they are discretized in advance)
 Examples are partitioned recursively based on selected attributes
 Test attributes are selected on the basis of a heuristic or statistical measure (e.g.,
information gain)
 Conditions for stopping partitioning
 All samples for a given node belong to the same class
 There are no remaining attributes for further partitioning – majority voting is
employed for classifying the leaf
 There are no samples left
 Bayesian Classification: Why?

 A statistical classifier: performs probabilistic prediction, i.e.,

predicts class membership probabilities
 Foundation: Based on Baye’s Theorem.
 Performance: A simple Bayesian classifier, naïve Bayesian
classifier, has comparable performance with decision tree and
selected neural network classifiers
 Incremental: Each training example can incrementally
increase/decrease the probability that a hypothesis is correct —
prior knowledge can be combined with observed data
 Standard: Even when Bayesian methods are computationally
intractable, they can provide a standard of optimal decision
making against which other methods can be measured
 Bayesian Theorem: Basics
 Let X be a data sample (“evidence”): class label is unknown
 Let H be a hypothesis that X belongs to class C
 Classification is to determine P(H|X), the probability that the hypothesis
holds given the observed data sample X
 P(H) (prior probability), the initial probability
 E.g., X will buy computer, regardless of age, income, …
 P(X): probability that sample data is observed
 P(X|H) (posteriori probability), the probability of observing the sample X,
given that the hypothesis holds
 E.g., Given that X will buy computer, the prob. that X is 31..40, medium
income
 Bayesian Theorem

 Given training data X, posteriori probability of a hypothesis H, P(H|X),

follows the Bayes theorem
P( H | X)  P(X | H ) P(H )
P(X)
 Informally, this can be written as
posteriori = likelihood x prior/evidence
 Predicts X belongs to C2 iff the probability P(Ci|X) is the highest among
all the P(Ck|X) for all the k classes
 Practical difficulty: require initial knowledge of many probabilities,
significant computational cost
 Using IF-THEN Rules for Classification

 Represent the knowledge in the form of IF-THEN rules

R: IF age = youth AND student = yes THEN buys_computer = yes
 Rule antecedent/precondition vs. rule consequent
 Assessment of a rule: coverage and accuracy
 ncovers = # of tuples covered by R
 ncorrect = # of tuples correctly classified by R
coverage(R) = ncovers /|D| /* D: training data set */
accuracy(R) = ncorrect / ncovers
 If more than one rule is triggered, need conflict resolution
 Size ordering: assign the highest priority to the triggering rules that has the “toughest”
requirement (i.e., with the most attribute test)
 Class-based ordering: decreasing order of prevalence or misclassification cost per class
 Rule-based ordering (decision list): rules are organized into one long priority list, according
to some measure of rule quality or by experts
 Other Classification Methods

 Genetic Algorithms
 Rough Set Approach
 Fuzzy Set Approach
 What Is Prediction?
 (Numerical) prediction is similar to classification
 construct a model
 use model to predict continuous or ordered value for a given input
 Prediction is different from classification
 Classification refers to predict categorical class label
 Prediction models continuous-valued functions
 Major method for prediction: regression
 model the relationship between one or more independent or predictor variables
and a dependent or response variable
 Regression analysis
 Linear and multiple regression
 Non-linear regression
 Other regression methods: generalized linear model, Poisson regression, log-linear
models, regression trees
 Linear Regression
 Linear regression: involves a response variable y and a single predictor variable x
y = w0 + w1 x
where w0 (y-intercept) and w1 (slope) are regression coefficients
 Method of least squares: estimates the best-fitting straight line
| D|

 (x  x )( yi  y )
w  w  yw x
i
i 1

1 | D|
0 1
 (x i 1
i  x)2

 Multiple linear regression: involves more than one predictor variable

 Training data is of the form (X1, y1), (X2, y2),…, (X|D|, y|D|)
 Ex. For 2-D data, we may have: y = w0 + w1 x1+ w2 x2
 Solvable by extension of least square method or using SAS, S-Plus
 Many nonlinear functions can be transformed into the above
 Nonlinear Regression

 Some nonlinear models can be modeled by a polynomial function

 A polynomial regression model can be transformed into linear regression
model. For example,
y = w0 + w 1 x + w2 x 2 + w3 x3
convertible to linear with new variables: x2 = x2, x3= x3
y = w0 + w 1 x + w2 x 2 + w3 x3
 Other functions, such as power function, can also be transformed to linear
model
 Some models are intractable nonlinear (e.g., sum of exponential terms)
 possible to obtain least square estimates through extensive calculation on
more complex formulae
 Other Regression-Based Models

 Generalized linear model:

 Foundation on which linear regression can be applied to modeling
categorical response variables
 Variance of y is a function of the mean value of y, not a constant
 Logistic regression: models the prob. of some event occurring as a linear
function of a set of predictor variables
 Poisson regression: models the data that exhibit a Poisson distribution
 Log-linear models: (for categorical data)
 Approximate discrete multidimensional prob. distributions
 Also useful for data compression and smoothing
 Regression trees and model trees
 Trees to predict continuous values rather than class labels
 Predictor Error Measures
 Measure predictor accuracy: measure how far off the predicted value is from the
actual known value
 Loss function: measures the error betw. y i and the predicted value yi’
 Absolute error: | yi – yi’|
 Squared error: (yi – yi’)2
 Test error (generalization error): the average loss over the test set
d d
 Mean absolute error:  | yi  yiMean
'| squared error: (y i  yi ' ) 2
i 1 i 1
d d
d
 Relative absolute error: | y Relative squared error:
 yi '| d

(y  yi ' ) 2
i
i 1 i
d
i 1
| y i y| d
i 1
(y i  y)2
The mean squared-error exaggerates the presence of outliers i 1

Popularly use (square) root mean-square error, similarly, root relative squared
error
 Conclusion
 Classification and prediction are two forms of data analysis that can be used to
extract models describing important data classes or to predict future data trends.
 Effective and scalable methods have been developed for decision trees
induction, Naive Bayesian classification, Bayesian belief network, rule-based
classifier, Back propagation, Support Vector Machine (SVM), associative
classification, nearest neighbor classifiers, and case-based reasoning, and other
classification methods such as genetic algorithms, rough set and fuzzy set
approaches.
 Linear, nonlinear, and generalized linear models of regression can be used for
prediction. Many nonlinear problems can be converted to linear problems by
performing transformations on the predictor variables. Regression trees and
model trees are also used for prediction.
 Conclusion (cont.)

 Stratified k-fold cross-validation is a recommended method for accuracy

estimation. Bagging and boosting can be used to increase overall accuracy
by learning and combining a series of individual models.
 There have been numerous comparisons of the different classification and
prediction methods, and the matter remains a research topic
 No single method has been found to be superior over all others for all data
sets
 Issues such as accuracy, training time, robustness, interpretability, and
scalability must be considered and can involve trade-offs, further
complicating the quest for an overall superior method
 References

 L. Breiman, J. Friedman, R. Olshen, and C.

Stone. Classification and Regression
Trees. Wadsworth International Group,
1984.
 Jiawei Han and Micheline Kamber :Data
Mining- Concepts and Techniques
.
ANY QUERY ?
THANK YOU!