COURSE CODE: SCSX1030

COURSE NAME: DATA MINING AND WAREHOUSING

CHAPTER NAME: ASSOCIATION RULE MINING AND CLASSIFICATION
SUBJECT COORDINATORS: Dr. S. JUSTIN SAMUEL, MARY RENI B

Unit –II

Mining Frequent Patterns, Associations and Correlations – Mining Methods – Mining

Various Kinds of Association Rules – Correlation Analysis – Constraint Based
Association Mining – Classification and Prediction - Basic Concepts - Decision Tree
Induction - Bayesian Classification – Rule Based Classification – Classification by
Back propagation – Support Vector Machines – Associative Classification – Lazy
Learners – Other Classification Methods – Prediction

Association Mining

• Association rule mining:

– Finding frequent patterns, associations, correlations, or causal structures
among sets of items or objects in transaction databases, relational databases,
and other information repositories.
• Applications:
– Basket data analysis, cross-marketing, catalog design, loss-leader analysis,
clustering, classification, etc.
• Examples.
– Rule form: ―Body ® Head [support, confidence]‖.
– buys(x, ―diapers‖) ® buys(x, ―beers‖) [0.5%, 60%]
– major(x, ―CS‖) ^ takes(x, ―DB‖) ® grade(x, ―A‖) [1%, 75%]

Association Rule: Basic Concepts

• Given: (1) database of transactions, (2) each transaction is a list of items (purchased
by a customer in a visit)
• Find: all rules that correlate the presence of one set of items with that of another set of
items
– E.g., 98% of people who purchase tires and auto accessories also get
automotive services done
• Applications
– *  Maintenance Agreement (What the store should do to boost
Maintenance Agreement sales)
– Home Electronics  * (What other products should the store stocks up?)
– Attached mailing in direct marketing
– Detecting ―ping-pong‖ing of patients, faulty ―collisions‖

Rule Measures: Support and Confidence

• Find all the rules X & Y  Z with minimum confidence and support
– support, s, probability that a transaction contains {X  Y  Z}
– confidence, c, conditional probability that a transaction having {X  Y} also
contains Z
Let minimum support 50%, and minimum confidence 50%, we have

– A  C (50%, 66.6%)
– C  A (50%, 100%)
Transaction ID Items Bought
2000 A,B,C
1000 A,C
4000 A,D
5000 B,E,F

Association Rule Mining: A Road Map

• Boolean vs. quantitative associations (Based on the types of values handled)

– buys(x, ―SQLServer‖) ^ buys(x, ―DMBook‖) ® buys(x, ―DBMiner‖) [0.2%,
60%]
– age(x, ―30..39‖) ^ income(x, ―42..48K‖) ® buys(x, ―PC‖) [1%, 75%]
• Single dimension vs. multiple dimensional associations (see ex. Above)
• Single level vs. multiple-level analysis
– What brands of beers are associated with what brands of diapers?
• Various extensions
– Correlation, causality analysis
• Association does not necessarily imply correlation or causality
– Maxpatterns and closed itemsets
– Constraints enforced
• E.g., small sales (sum < 100) trigger big buys (sum > 1,000)?

Market – Basket analysis

A market basket is a collection of items purchased by a customer in a single

transaction, which is a well-defined business activity. For example, a customer's visits to a
grocery store or an online purchase from a virtual store on the Web are typical customer
transactions. Retailers accumulate huge collections of transactions by recording business
activities over time. One common analysis run against a transactions database is to find sets
of items, or itemsets, that appear together in many transactions. A business can use
knowledge of these patterns to improve the Placement of these items in the store or the layout
of mail- order catalog page and Web pages. An itemset containing i items is called an i-
itemset. The percentage of transactions that contain an itemset is called the itemset's support.
For an itemset to be interesting, its support must be higher than a user-specified minimum.
Such itemsets are said to be frequent.

Figure : Market basket analysis.

 Rule support and confidence are two measures of rule interestingness. They
respectively reflect the usefulness and certainty of discovered rules. A support of 2% for
association Rule means that 2% of all the transactions under analysis show that computer and
financial management software are purchased together. A confidence of 60% means that 60%
of the customers who purchased a computer also bought the software. Typically, association
rules are considered interesting if they satisfy both a minimum support threshold and a
minimum confidence threshold.

Mining Frequent Patterns

The method that mines the complete set of frequent itemsets with candidate generation.
Apriori property & The Apriori Algorithm.

Apriori property

• All nonempty subsets of a frequent item set most also be frequent.

– An item set I does not satisfy the minimum support threshold, min-sup, then I
is not frequent, i.e., support(I) < min-sup
– If an item A is added to the item set I then the resulting item set (I U A) can
not occur more frequently than I.
• Monotonic functions are functions that move in only one direction.
• This property is called anti-monotonic.
• If a set can not pass a test, all its supersets will fail the same test as well.
• This property is monotonic in failing the test.

The Apriori Algorithm

• Join Step: Ck is generated by joining Lk-1with itself

• Prune Step: Any (k-1)-itemset that is not frequent cannot be a subset of a frequent k-
itemset
Example
The method that mines the complete set of frequent itemsets without generation.

• Compress a large database into a compact, Frequent-Pattern tree (FP-tree) structure

– highly condensed, but complete for frequent pattern mining
– avoid costly database scans
• Develop an efficient, FP-tree-based frequent pattern mining method
– A divide-and-conquer methodology: decompose mining tasks into smaller
ones
– Avoid candidate generation: sub-database test only!

Construct FP-tree from a Transaction DB

TID Items bought (ordered) frequent items

100 {f, a, c, d, g, i, m, p} {f, c, a, m, p}
200 {a, b, c, f, l, m, o} {f, c, a, b, m} min_support = 0.5
300 {b, f, h, j, o} {f, b}
400 {b, c, k, s, p} {c, b, p}
500 {a, f, c, e, l, p, m, n} {f, c, a, m, p}
Steps:

1. Scan DB once, find frequent 1-itemset (single item pattern)

2. Order frequent items in frequency descending order
3. Scan DB again, construct FP-tree

Header Table

Item frequency head

f 4
c 4
a 3
b 3
m 3
p 3

Benefits of the FP-tree Structure

• Completeness:
– never breaks a long pattern of any transaction
– preserves complete information for frequent pattern mining
• Compactness
– reduce irrelevant information—infrequent items are gone
– frequency descending ordering: more frequent items are more likely to be
shared
– never be larger than the original database (if not count node-links and counts)
– Example: For Connect-4 DB, compression ratio could be over 100
Mining Frequent Patterns Using FP-tree

• General idea (divide-and-conquer)

– Recursively grow frequent pattern path using the FP-tree
• Method
– For each item, construct its conditional pattern-base, and then its conditional
FP-tree
– Repeat the process on each newly created conditional FP-tree
– Until the resulting FP-tree is empty, or it contains only one path (single path
will generate all the combinations of its sub-paths, each of which is a frequent
pattern)

Major Steps to Mine FP-tree

1) Construct conditional pattern base for each node in the FP-tree

2) Construct conditional FP-tree from each conditional pattern-base
3) Recursively mine conditional FP-trees and grow frequent patterns obtained so far
 If the conditional FP-tree contains a single path, simply enumerate all the
patterns

Principles of Frequent Pattern Growth

• Pattern growth property

– Let  be a frequent itemset in DB, B be 's conditional pattern base, and  be
an itemset in B. Then    is a frequent itemset in DB iff  is frequent in B.
• ―abcdef ‖ is a frequent pattern, if and only if
– ―abcde ‖ is a frequent pattern, and
– ―f ‖ is frequent in the set of transactions containing ―abcde ‖

Why Is Frequent Pattern Growth Fast?

• Our performance study shows

– FP-growth is an order of magnitude faster than Apriori, and is also faster than
tree-projection
• Reasoning
– No candidate generation, no candidate test
– Use compact data structure
– Eliminate repeated database scan
Basic operation is counting and FP-tree building

Mining multilevel association rules from transactional databases.

• Items often form hierarchy.

• Items at the lower level are expected to have lower support.
• Rules regarding itemsets at
appropriate levels could be quite useful.

• Transaction database can be encoded based on dimensions and levels

 • We can explore shared multi-level mining

Food

Milk Bread

Skim 2% Wheat White

Fraser Sunset

TID Items
T1 {111, 121, 211, 221}
T2 {111, 211, 222, 323}
T3 {112, 122, 221, 411}
T4 {111, 121}
T5 {111, 122, 211, 221, 413}

Mining Multi-Level Associations

• A top_down, progressive deepening approach:

– First find high-level strong rules:
milk ® bread [20%, 60%].

– Then find their lower-level ―weaker‖ rules:

2% milk ® wheat bread [6%, 50%].

• Variations at mining multiple-level association rules.

– Level-crossed association rules:
2% milk ® Wonder wheat bread

– Association rules with multiple, alternative hierarchies:

2% milk ® Wonder bread

Multi-level Association: Uniform Support vs. Reduced Support

• Uniform Support: the same minimum support for all levels

– + One minimum support threshold. No need to examine itemsets containing
any item whose ancestors do not have minimum support.
– – Lower level items do not occur as frequently. If support threshold
• too high  miss low level associations
• too low  generate too many high level associations
• Reduced Support: reduced minimum support at lower levels
– There are 4 search strategies:
 • Level-by-level independent
• Level-cross filtering by k-itemset
• Level-cross filtering by single item
• Controlled level-cross filtering by single item

Multi-level Association: Redundancy Filtering

• Some rules may be redundant due to ―ancestor‖ relationships between items.

• Example
– milk  wheat bread [support = 8%, confidence = 70%]
– 2% milk  wheat bread [support = 2%, confidence = 72%]
• We say the first rule is an ancestor of the second rule.
• A rule is redundant if its support is close to the ―expected‖ value, based on the rule’s
ancestor

Multi-Level Mining: Progressive Deepening

• A top-down, progressive deepening approach:

– First mine high-level frequent items:
milk (15%), bread (10%)

– Then mine their lower-level ―weaker‖ frequent itemsets:

2% milk (5%), wheat bread (4%)

• Different min_support threshold across multi-levels lead to different algorithms:

– If adopting the same min_support across multi-levels then toss t if any of t’s
ancestors is infrequent.
– If adopting reduced min_support at lower levels then examine only those
descendents whose ancestor’s support is frequent/non-negligible.

Correlation in detail.

• Interest (correlation, lift)

– taking both P(A) and P(B) in consideration
– P(A^B)=P(B)*P(A), if A and B are independent events
– A and B negatively correlated, if the value is less than 1; otherwise A and B
positively correlated

X11110000 Itemset Support Interest

Y11000000 X,Y 25% 2
2ZCorrelation
01111111 X,Z 37.50% 0.9
• 2 measures correlation Y,Z 12.50% attributes
between categorical 0.57
 (observed _ exp ected ) 2
 2   exp ected
game not game sum(row)
video 4000(4500) 3500(3000) 7500
not video 2000(1500) 500 (1000) 2500
sum(col.) 6000 4000 10000

• expected(i,j) = count(row i) * count(column j) / N

• 2 = (4000 - 4500)2 / 4500 - (3500 - 3000)2 / 3000 - (2000 - 1500)2 / 1500 - (500 -
1000)2 / 1000 = 555.6
• 2 > 1 and observed value of (game, video) < expected value, there is a negative
correlation

Numeric correlation

• Correlation concept in statistics

– Used to study the relationship existing between 2 or more numeric variables
– A correlation is a measure of the linear relationship between variables
Ex: number of hours spent studying in a class with grade received
– Outcomes:
•  positively related
•  Not related
•  negatively related
– Statistical relationships
• Covariance
• Correlation coefficient

Constraint-Based Mining in detail.

• Interactive, exploratory mining giga-bytes of data?

– Could it be real? — Making good use of constraints!
• What kinds of constraints can be used in mining?
– Knowledge type constraint: classification, association, etc.
– Data constraint: SQL-like queries
• Find product pairs sold together in Vancouver in Dec.’98.
– Dimension/level constraints:
• in relevance to region, price, brand, customer category.
– Rule constraints
• small sales (price < $10) triggers big sales (sum > $200).
– Interestingness constraints:
• strong rules (min_support  3%, min_confidence  60%).

Rule Constraints in Association Mining

• Two kind of rule constraints:

 – Rule form constraints: meta-rule guided mining.
• P(x, y) ^ Q(x, w) ® takes(x, ―database systems‖).
– Rule (content) constraint: constraint-based query optimization (Ng, et al.,
SIGMOD’98).
• sum(LHS) < 100 ^ min(LHS) > 20 ^ count(LHS) > 3 ^ sum(RHS) >
1000
• 1-variable vs. 2-variable constraints (Lakshmanan, et al. SIGMOD’99):
– 1-var: A constraint confining only one side (L/R) of the rule, e.g., as shown
above.
– 2-var: A constraint confining both sides (L and R).
• sum(LHS) < min(RHS) ^ max(RHS) < 5* sum(LHS)

Constrain-Based Association Query

• Database: (1) trans (TID, Itemset ), (2) itemInfo (Item, Type, Price)
• A constrained asso. query (CAQ) is in the form of {(S1, S2 )|C },
– where C is a set of constraints on S1, S2 including frequency constraint
• A classification of (single-variable) constraints:
– Class constraint: S  A. e.g. S  Item
– Domain constraint:
• S v,   { , , , , ,  }. e.g. S.Price < 100
• v S,  is  or . e.g. snacks  S.Type
• V S, or S V,   { , , , ,  }
– e.g. {snacks, sodas }  S.Type
– Aggregation constraint: agg(S)  v, where agg is in {min, max, sum, count,
avg}, and   { , , , , ,  }.
• e.g. count(S1.Type)  1 , avg(S2.Price)  100

Constrained Association Query Optimization Problem

• Given a CAQ = { (S1, S2) | C }, the algorithm should be :

– sound: It only finds frequent sets that satisfy the given constraints C
– complete: All frequent sets satisfy the given constraints C are found
• A naïve solution:
– Apply Apriori for finding all frequent sets, and then to test them for constraint
satisfaction one by one.
• Our approach:
– Comprehensive analysis of the properties of constraints and try to push them
as deeply as possible inside the frequent set computation.
Categories of Constraints.

1. Anti-monotone and Monotone Constraints

• constraint Ca is anti-monotone iff. for any pattern S not satisfying Ca, none of the
super-patterns of S can satisfy Ca
• A constraint Cm is monotone iff. for any pattern S satisfying Cm, every super-pattern
of S also satisfies it
 2. Succinct Constraint
• A subset of item Is is a succinct set, if it can be expressed as p(I) for some selection
predicate p, where  is a selection operator
• SP2I is a succinct power set, if there is a fixed number of succinct set I1, …, Ik I,
s.t. SP can be expressed in terms of the strict power sets of I1, …, Ik using union and
minus
• A constraint Cs is succinct provided SATCs(I) is a succinct power set

3. Convertible Constraint
• Suppose all items in patterns are listed in a total order R
• A constraint C is convertible anti-monotone iff a pattern S satisfying the constraint
implies that each suffix of S w.r.t. R also satisfies C
• A constraint C is convertible monotone iff a pattern S satisfying the constraint implies
that each pattern of which S is a suffix w.r.t. R also satisfies C

Property of Constraints: Anti-Monotone

• Anti-monotonicity: If a set S violates the constraint, any superset of S violates the

constraint.
• Examples:
– sum(S.Price)  v is anti-monotone
– sum(S.Price)  v is not anti-monotone
– sum(S.Price) = v is partly anti-monotone
• Application:
– Push ―sum(S.price)  1000‖ deeply into iterative frequent set computation.

Example of Convertible Constraints: Avg(S)  V

• Let R be the value descending order over the set of items

– E.g. I={9, 8, 6, 4, 3, 1}
• Avg(S)  v is convertible monotone w.r.t. R
– If S is a suffix of S1, avg(S1)  avg(S)
• {8, 4, 3} is a suffix of {9, 8, 4, 3}
• avg({9, 8, 4, 3})=6  avg({8, 4, 3})=5
– If S satisfies avg(S) v, so does S1
• {8, 4, 3} satisfies constraint avg(S)  4, so does {9, 8, 4, 3}

Property of Constraints: Succinctness

• Succinctness:
– For any set S1 and S2 satisfying C, S1  S2 satisfies C
– Given A1 is the sets of size 1 satisfying C, then any set S satisfying C are
based on A1 , i.e., it contains a subset belongs to A1 ,
• Example :
– sum(S.Price )  v is not succinct
– min(S.Price )  v is succinct
• Optimization:
 – If C is succinct, then C is pre-counting prunable. The satisfaction of the
constraint alone is not affected by the iterative support counting.

Classification and Prediction

 Classification:

– predicts categorical class labels

– classifies data (constructs a model) based on the training set and the values
(class labels) in a classifying attribute and uses it in classifying new data
• Prediction

– models continuous-valued functions, i.e., predicts unknown or missing values

• Typical applications

– Credit approval
– Target marketing
– Medical diagnosis
– Fraud detection
–
Classification—A Two-Step Process

• Model construction: describing a set of predetermined classes

– Each tuple/sample is assumed to belong to a predefined class, as determined
by the class label attribute
– The set of tuples used for model construction: training set
– The model is represented as classification rules, decision trees, or
mathematical formulae
• Model usage: for classifying future or unknown objects
– Estimate accuracy of the model
• The known label of test sample is compared with the classified result
from the model
• Accuracy rate is the percentage of test set samples that are correctly
classified by the model
• Test set is independent of training set, otherwise over-fitting will occur
Process (1): Model Construction
Process (2): Using the Model in Prediction

Supervised vs. Unsupervised Learning

 Supervised learning (classification)
 Supervision: The training data (observations, measurements, etc.) are
accompanied by labels indicating the class of the observations
 New data is classified based on the training set
 Unsupervised learning (clustering)
 The class labels of training data is unknown
 Given a set of measurements, observations, etc. with the aim of establishing
the existence of classes or clusters in the data

Classification by Decision Tree Induction

Decision tree

– A flow-chart-like tree structure

– Internal node denotes a test on an attribute
 – Branch represents an outcome of the test
– Leaf nodes represent class labels or class distribution
• Decision tree generation consists of two phases
– Tree construction
• At start, all the training examples are at the root
• Partition examples recursively based on selected attributes
– Tree pruning
• Identify and remove branches that reflect noise or outliers
• Use of decision tree: Classifying an unknown sample
– Test the attribute values of the sample against the decision tree

Training Dataset

This follows an example from Quinlan’s ID3

age income student credit_rating
<=30 high no fair
<=30 high no excellent
31…40 high no fair
>40 medium no fair
>40 low yes fair
>40 low yes excellent
31…40 low yes excellent
<=30 medium no fair
<=30 low yes fair
>40 medium yes fair
<=30 medium yes excellent
31…40 medium no excellent
31…40 high yes fair
>40 medium no excellent

Algorithm for Decision Tree Induction

• Basic algorithm (a greedy algorithm)

– Tree is constructed in a top-down recursive divide-and-conquer manner
– At start, all the training examples are at the root
– Attributes are categorical (if continuous-valued, they are discretized in
advance)
– Examples are partitioned recursively based on selected attributes
– Test attributes are selected on the basis of a heuristic or statistical measure
(e.g., information gain)
• Conditions for stopping partitioning
– All samples for a given node belong to the same class
– There are no remaining attributes for further partitioning – majority voting is
employed for classifying the leaf
– There are no samples left

Extracting Classification Rules from Trees

• Represent the knowledge in the form of IF-THEN rules

• One rule is created for each path from the root to a leaf
• Each attribute-value pair along a path forms a conjunction
• The leaf node holds the class prediction
• Rules are easier for humans to understand
•
 • Example

IF age = ―<=30‖ AND student = ―no‖ THEN buys_computer = ―no‖

IF age = ―<=30‖ AND student = ―yes‖ THEN buys_computer = ―yes‖

IF age = ―31…40‖ THEN buys_computer = ―yes‖

IF age = ―>40‖ AND credit_rating = ―excellent‖ THEN buys_computer = ―yes‖

IF age = ―>40‖ AND credit_rating = ―fair‖ THEN buys_computer = ―no‖

Avoid Overfitting in Classification

• The generated tree may overfit the training data

– Too many branches, some may reflect anomalies due to noise or outliers
– Result is in poor accuracy for unseen samples
• Two approaches to avoid overfitting
– Prepruning: Halt tree construction early—do not split a node if this would
result in the goodness measure falling below a threshold
• Difficult to choose an appropriate threshold
– Postpruning: Remove branches from a ―fully grown‖ tree—get a sequence of
progressively pruned trees
• Use a set of data different from the training data to decide which is the
―best pruned tree‖

Tree Mining in Weka and Tree Mining in Clementine.

Tree Mining in Weka

• Example:
– Weather problem: build a decision tree to guide the decision about whether or
not to play tennis.
– Dataset
(weather.nominal.arff)

• Validation:
– Using training set as a test set will provide optimal classification accuracy.
– Expected accuracy on a different test set will always be less.
– 10-fold cross validation is more robust than using the training set as a test set.
• Divide data into 10 sets with about same proportion of class label
values as in original set.
• Run classification 10 times independently with the remaining 9/10 of
the set as the training set.
• Average accuracy.
– Ratio validation: 67% training set / 33% test set.
– Best: having a separate training set and test set.
 • Results:
– Classification accuracy (correctly classified instances).
– Errors (absolute mean, root squared mean, …)
– Kappa statistic (measures agreement between predicted and observed
classification; -100%-100% is the proportion of agreements after chance
agreement has been excluded; 0% means complete agreement by chance)
• Results:
– TP (True Positive) rate per class label
– FP (False Positive) rate
– Precision = TP rate = TP / (TP + FN)) * 100%
– Recall = TP / (TP + FP)) * 100%
– F-measure = 2* recall * precision / recall + precision
• ID3 characteristics:
– Requires nominal values
– Improved into C4.5
• Dealing with numeric attributes
• Dealing with missing values
• Dealing with noisy data
• Generating rules from trees

Tree Mining in Clementine

• Methods:
– C5.0: target field must be categorical, predictor fields may be numeric or
categorical, provides multiple splits on the field that provides the maximum
information gain at each level
– QUEST: target field must be categorical, predictor fields may be numeric
ranges or categorical, statistical binary split
– C&RT: target and predictor fields may be numeric ranges or categorical,
statistical binary split based on regression
– CHAID: target and predictor fields may be numeric ranges or categorical,
statistical binary split based on chi-square

Bayesian Classification:
• Probabilistic learning: Calculate explicit probabilities for hypothesis, among the most
practical approaches to certain types of learning problems
• Incremental: Each training example can incrementally increase/decrease the
probability that a hypothesis is correct. Prior knowledge can be combined with
observed data.
• Probabilistic prediction: Predict multiple hypotheses, weighted by their probabilities
• Standard: Even when Bayesian methods are computationally intractable, they can
provide a standard of optimal decision making against which other methods can be
measured

Bayesian Theorem
 • Given training data D, posteriori probability of a hypothesis h, P(h|D) follows the
Bayes theorem

P(h | D)  P(D | h)P(h)

P(D)
• MAP (maximum posteriori) hypothesis

h  arg max P(h | D)  arg max P(D | h)P(h).

MAP hH hH

• Practical difficulty: require initial knowledge of many probabilities, significant

computational cost

Naïve Bayes Classifier (I)

• A simplified assumption: attributes are conditionally independent:

n
P (C j | V )  P (C j ) P (vi | C j )
i 1

• Greatly reduces the computation cost, only count the class distribution.

Naive Bayesian Classifier (II)

Given a training set, we can compute the probabilities

Outlook P N Humidity P N
sunny 2/9 3/5 high 3/9 4/5
overcast 4/9 0 normal 6/9 1/5
rain 3/9 2/5
Bayesian Temperature W indy
classification
hot 2/9 2/5 true 3/9 3/5
• The mild 4/9 2/5 false 6/9 2/5
classification
problem may cool 3/9 1/5 be formalized
using a- posteriori
probabilities:
• P(C|X) = prob. that the sample tuple
• X=<x1,…,xk> is of class C.
• E.g. P(class=N | outlook=sunny, windy=true,…)
• Idea: assign to sample X the class label C such that P(C|X) is maximal

Estimating a-posteriori probabilities

• Bayes theorem:
P(C|X) = P(X|C)·P(C) / P(X)
 • P(X) is constant for all classes
• P(C) = relative freq of class C samples
• C such that P(C|X) is maximum =
C such that P(X|C)·P(C) is maximum
• Problem: computing P(X|C) is unfeasible!

Naïve Bayesian Classification

• Naïve assumption: attribute independence

P(x1,…,xk|C) = P(x1|C)·…·P(xk|C)

• If i-th attribute is categorical:

P(xi|C) is estimated as the relative freq of samples having value xi as i-th attribute in
class C
• If i-th attribute is continuous:
P(xi|C) is estimated thru a Gaussian density function
• Computationally easy in both cases

Bayesian Belief Networks

 Bayesian belief network allows a subset of the variables conditionally independent

 A graphical model of causal relationships
 Represents dependency among the variables
 Gives a specification of joint probability distribution

Bayesian Belief Network: An Example

The conditional probability table (CPT) for variable LungCancer:

CPT shows the conditional probability for each possible combination of its parents

Derivation of the probability of a particular combination of values of X, from CPT:

n
P( x1 ,...,xn )   P( xi | Parents(Y i ))
i 1

Association-Based Classification

• Several methods for association-based classification

– ARCS: Quantitative association mining and clustering of association rules
(Lent et al’97)
• It beats C4.5 in (mainly) scalability and also accuracy
– Associative classification: (Liu et al’98)
• It mines high support and high confidence rules in the form of
―cond_set => y‖, where y is a class label
– CAEP (Classification by aggregating emerging patterns) (Dong et al’99)
• Emerging patterns (EPs): the itemsets whose support increases
significantly from one class to another
• Mine Eps based on minimum support and growth rate

Pruning of decision trees

Discarding one or more subtrees and replacing them with leaves simplify a decision tree,
and that is the main task in decision-tree pruning. In replacing the subtree with a leaf, the
algorithm expects to lower the predicted error rate and increase the quality of a classification
model. But computation of error rate is not simple. An error rate based only on a training data
set does not provide a suitable estimate. One possibility to estimate the predicted error rate is
to use a new, additional set of test samples if they are available, or to use the cross-validation
techniques. This technique divides initially available samples into equal sized blocks and, for
each block, the tree is constructed from all samples except this block and tested with a given
block of samples. With the available training and testing samples, the basic idea of decision
tree-pruning is to remove parts of the tree (subtrees) that do not contribute to the
classification accuracy of unseen testing samples, producing a less complex and thus more
comprehensible tree. There are two ways in which the recursive-partitioning method can be
modified:

1. Deciding not to divide a set of samples any further under some conditions. The
stopping criterion is usually based on some statistical tests, such as the χ2 test: If there
are no significant differences in classification accuracy before and after division, then
represent a current node as a leaf. The decision is made in advance, before splitting,
and therefore this approach is called prepruning.
2. Removing restrospectively some of the tree structure using selected accuracy criteria.
The decision in this process of postpruning is made after the tree has been built.
C4.5 follows the postpruning approach, but it uses a specific technique to estimate the
predicted error rate. This method is called pessimistic pruning. For every node in a tree, the
estimation of the upper confidence limit ucf is computed using the statistical tables for
binomial distribution (given in most textbooks on statistics). Parameter Ucf is a function of
∣ Ti∣ and E for a given node. C4.5 uses the default confidence level of 25%, and compares
U25% (∣ Ti∣ /E) for a given node Ti with a weighted confidence of its leaves. Weights are the
total number of cases for every leaf. If the predicted error for a root node in a subtree is less
than weighted sum of U25% for the leaves (predicted error for the subtree), then a subtree will
be replaced with its root node, which becomes a new leaf in a pruned tree.

Let us illustrate this procedure with one simple example. A subtree of a decision tree
is given in Figure, where the root node is the test x1 on three possible values {1, 2, 3} of the
attribute A. The children of the root node are leaves denoted with corresponding classes and
(∣ Ti∣ /E) parameters. The question is to estimate the possibility of pruning the subtree and
replacing it with its root node as a new, generalized leaf node.

Figure : Pruning a subtree by replacing it with one leaf node

To analyze the possibility of replacing the subtree with a leaf node it is necessary to
compute a predicted error PE for the initial tree and for a replaced node. Using default
confidence of 25%, the upper confidence limits for all nodes are collected from statistical
tables: U25% (6, 0) = 0.206, U25%(9, 0) = 0.143, U25%(1, 0) = 0.750, and U25%(16, 1) = 0.157.
Using these values, the predicted errors for the initial tree and the replaced node are

PEtree = 6.0.206 + 9.0.143 + 1.0.750 = 3.257

PEnode = 16.0.157 = 2.512

Since the existing subtree has a higher value of predicted error than the replaced node,
it is recommended that the decision tree be pruned and the subtree replaced with the new leaf
node.

Rule Based Classification

Using IF-THEN Rules for Classification

 Represent the knowledge in the form of IF-THEN rules

R: IF age = youth AND student = yes THEN buys_computer = yes

  Rule antecedent/precondition vs. rule consequent
 Assessment of a rule: coverage and accuracy
 ncovers = # of tuples covered by R
 ncorrect = # of tuples correctly classified by R

coverage(R) = ncovers /|D| /* D: training data set */

accuracy(R) = ncorrect / ncovers

 If more than one rule is triggered, need conflict resolution

 Size ordering: assign the highest priority to the triggering rules that has the
―toughest‖ requirement (i.e., with the most attribute test)
 Class-based ordering: decreasing order of prevalence or misclassification cost
per class
 Rule-based ordering (decision list): rules are organized into one long priority
list, according to some measure of rule quality or by experts

Rule Extraction from a Decision Tree

 Rules are easier to understand than large trees

 One rule is created for each path from the root to a leaf
 Each attribute-value pair along a path forms a conjunction: the leaf holds the class
prediction
 Rules are mutually exclusive and exhaustive

 Example: Rule extraction from our buys_computer decision-tree

IF age = young AND student = no THEN buys_computer = no

IF age = young AND student = yes THEN buys_computer = yes
IF age = mid-age THEN buys_computer = yes
IF age = old AND credit_rating = excellent THEN buys_computer = yes
IF age = young AND credit_rating = fair THEN buys_computer = no

Rule Extraction from the Training Data

 Sequential covering algorithm: Extracts rules directly from training data

 Typical sequential covering algorithms: FOIL, AQ, CN2, RIPPER
 Rules are learned sequentially, each for a given class Ci will cover many tuples of Ci
but none (or few) of the tuples of other classes
 Steps:
 Rules are learned one at a time
 Each time a rule is learned, the tuples covered by the rules are removed
  The process repeats on the remaining tuples unless termination condition, e.g.,
when no more training examples or when the quality of a rule returned is
below a user-specified threshold
 Comp. w. decision-tree induction: learning a set of rules simultaneously
Classification by Backpropagation
 Backpropagation: A neural network learning algorithm
 Started by psychologists and neurobiologists to develop and test computational
analogues of neurons
 A neural network: A set of connected input/output units where each connection has a
weight associated with it
 During the learning phase, the network learns by adjusting the weights so as to be
able to predict the correct class label of the input tuples
 Also referred to as connectionist learning due to the connections between units
Neural Network as a Classifier

 Weakness
 Long training time
 Require a number of parameters typically best determined empirically, e.g.,
the network topology or ``structure."
 Poor interpretability: Difficult to interpret the symbolic meaning behind the
learned weights and of ``hidden units" in the network
 Strength
 High tolerance to noisy data
 Ability to classify untrained patterns
 Well-suited for continuous-valued inputs and outputs
 Successful on a wide array of real-world data
 Algorithms are inherently parallel
 Techniques have recently been developed for the extraction of rules from
trained neural networks

A Neuron (= a perceptron)

 The n-dimensional input vector x is mapped into variable y by means of the scalar
product and a nonlinear function mapping

A Multi-Layer Feed-Forward Neural Network

  The inputs to the network correspond to the attributes measured for each training
tuple
 Inputs are fed simultaneously into the units making up the input layer
 They are then weighted and fed simultaneously to a hidden layer
 The number of hidden layers is arbitrary, although usually only one
 The weighted outputs of the last hidden layer are input to units making up the output
layer, which emits the network's prediction
 The network is feed-forward in that none of the weights cycles back to an input unit
or to an output unit of a previous layer
 From a statistical point of view, networks perform nonlinear regression: Given
enough hidden units and enough training samples, they can closely approximate any
function

Backpropagation

 Iteratively process a set of training tuples & compare the network's prediction with the
actual known target value
 For each training tuple, the weights are modified to minimize the mean squared error
between the network's prediction and the actual target value
 Modifications are made in the ―backwards‖ direction: from the output layer, through
each hidden layer down to the first hidden layer, hence ―backpropagation‖
 Steps
 Initialize weights (to small random #s) and biases in the network
 Propagate the inputs forward (by applying activation function)
 Backpropagate the error (by updating weights and biases)
 Terminating condition (when error is very small, etc.)
 Efficiency of backpropagation: Each epoch (one interation through the training set)
takes O(|D| * w), with |D| tuples and w weights, but # of epochs can be exponential to
n, the number of inputs, in the worst case
 Rule extraction from networks: network pruning
 Simplify the network structure by removing weighted links that have the least
effect on the trained network
 Then perform link, unit, or activation value clustering
 The set of input and activation values are studied to derive rules describing the
relationship between the input and hidden unit layers
 Sensitivity analysis: assess the impact that a given input variable has on a network
output. The knowledge gained from this analysis can be represented in rules
SVM—Support Vector Machines
 A new classification method for both linear and nonlinear data
 It uses a nonlinear mapping to transform the original training data into a higher
dimension
 With the new dimension, it searches for the linear optimal separating hyperplane (i.e.,
―decision boundary‖)
 With an appropriate nonlinear mapping to a sufficiently high dimension, data from
two classes can always be separated by a hyperplane
 SVM finds this hyperplane using support vectors (―essential‖ training tuples) and
margins (defined by the support vectors)
 Features: training can be slow but accuracy is high owing to their ability to
model complex nonlinear decision boundaries (margin maximization)
 Used both for classification and prediction
 Applications:
 handwritten digit recognition, object recognition, speaker identification,
benchmarking time-series prediction tests

SVM—General Philosophy

SVM—Margins and Support Vectors

SVM—Linearly Separable
 A separating hyperplane can be written as
W●X+b=0
where W={w1, w2, …, wn} is a weight vector and b a scalar (bias)
 For 2-D it can be written as
w0 + w1 x1 + w2 x2 = 0
 The hyperplane defining the sides of the margin:
H1: w0 + w1 x1 + w2 x2 ≥ 1 for yi = +1, and
H2: w0 + w1 x1 + w2 x2 ≤ – 1 for yi = –1
 Any training tuples that fall on hyperplanes H1 or H2 (i.e., the
sides defining the margin) are support vectors
 This becomes a constrained (convex) quadratic optimization problem: Quadratic
objective function and linear constraints  Quadratic Programming (QP) 
Lagrangian multipliers
Why Is SVM Effective on High Dimensional Data?
 The complexity of trained classifier is characterized by the # of support vectors rather
than the dimensionality of the data
  The support vectors are the essential or critical training examples —they lie closest to
the decision boundary (MMH)
 If all other training examples are removed and the training is repeated, the same
separating hyperplane would be found
 The number of support vectors found can be used to compute an (upper) bound on the
expected error rate of the SVM classifier, which is independent of the data
dimensionality
 Thus, an SVM with a small number of support vectors can have good generalization,
even when the dimensionality of the data is high

Associative Classification
 Associative classification
 Association rules are generated and analyzed for use in classification
 Search for strong associations between frequent patterns (conjunctions of
attribute-value pairs) and class labels
 Classification: Based on evaluating a set of rules in the form of
P1 ^ p2 … ^ pl  ―Aclass = C‖ (conf, sup)
 Why effective?
 It explores highly confident associations among multiple attributes and may
overcome some constraints introduced by decision-tree induction, which
considers only one attribute at a time
In many studies, associative classification has been found to be more accurate than some
traditional classification methods, such as C4.

Typical Associative Classification Methods

 CBA (Classification By Association: Liu, Hsu & Ma, KDD’98)
 Mine association possible rules in the form of
 Cond-set (a set of attribute-value pairs)  class label
 Build classifier: Organize rules according to decreasing precedence based on
confidence and then support
 CMAR (Classification based on Multiple Association Rules: Li, Han, Pei, ICDM’01)
 Classification: Statistical analysis on multiple rules
 CPAR (Classification based on Predictive Association Rules: Yin & Han, SDM’03)
 Generation of predictive rules (FOIL-like analysis)
 High efficiency, accuracy similar to CMAR
 RCBT (Mining top-k covering rule groups for gene expression data, Cong et al.
SIGMOD’05)
 Explore high-dimensional classification, using top-k rule groups
 Achieve high classification accuracy and high run-time efficiency

Associative Classification May Achieve High Accuracy and Efficiency (Cong et al.
SIGMOD05)
Other Classification Methods
The k-Nearest Neighbor Algorithm

 All instances correspond to points in the n-D space

 The nearest neighbor are defined in terms of Euclidean distance, dist(X1, X2)
 Target function could be discrete- or real- valued
 For discrete-valued, k-NN returns the most common value among the k training
examples nearest to xq
 k-NN for real-valued prediction for a given unknown tuple
 Returns the mean values of the k nearest neighbors
 Distance-weighted nearest neighbor algorithm
 Weight the contribution of each of the k neighbors according to their distance
to the query xq
 Give greater weight to closer neighbors
 Robust to noisy data by averaging k-nearest neighbors
 Curse of dimensionality: distance between neighbors could be dominated by
irrelevant attributes
 To overcome it, axes stretch or elimination of the least relevant attributes

Genetic Algorithms

 Genetic Algorithm: based on an analogy to biological evolution

 An initial population is created consisting of randomly generated rules
 Each rule is represented by a string of bits
 E.g., if A1 and ¬A2 then C2 can be encoded as 100
 If an attribute has k > 2 values, k bits can be used
 Based on the notion of survival of the fittest, a new population is formed to consist of
the fittest rules and their offsprings
 The fitness of a rule is represented by its classification accuracy on a set of training
examples
 Offsprings are generated by crossover and mutation
  The process continues until a population P evolves when each rule in P satisfies a
prespecified threshold
 Slow but easily parallelizable

Rough Set Approach:

 Rough sets are used to approximately or ―roughly‖ define equivalent

classes
 A rough set for a given class C is approximated by two sets: a lower approximation
(certain to be in C) and an upper approximation (cannot be described as not belonging
to C)
Finding the minimal subsets (reducts) of attributes for feature reduction is NP-hard but a
discernibility matrix (which stores the differences between attribute values for each pair of
data tuples) is used to reduce the computation intensity

Figure: A rough set approximation of the set of tuples of the class C suing lower and
upper approximation sets of C. The rectangular regions represent equivalence classes

Fuzzy Set approaches

 Fuzzy logic uses truth values between 0.0 and 1.0 to represent the degree of
membership (such as using fuzzy membership graph)
 Attribute values are converted to fuzzy values
 e.g., income is mapped into the discrete categories {low, medium, high} with
fuzzy values calculated
 For a given new sample, more than one fuzzy value may apply
 Each applicable rule contributes a vote for membership in the categories
 Typically, the truth values for each predicted category are summed, and these sums
are combined
Prediction

 (Numerical) prediction is similar to classification

 construct a model
 use model to predict continuous or ordered value for a given input
 Prediction is different from classification
 Classification refers to predict categorical class label
 Prediction models continuous-valued functions
 Major method for prediction: regression
 model the relationship between one or more independent or predictor variables
and a dependent or response variable
 Regression analysis
 Linear and multiple regression
 Non-linear regression
 Other regression methods: generalized linear model, Poisson regression, log-
linear models, regression trees
Linear Regression
 Linear regression: involves a response variable y and a single predictor variable x
y = w0 + w1 x
where w0 (y-intercept) and w1 (slope) are regression coefficients
 Method of least squares: estimates the best-fitting straight line
 Multiple linear regression: involves more than one predictor variable
 Training data is of the form (X1, y1), (X2, y2),…, (X|D|, y|D|)
 Ex. For 2-D data, we may have: y = w0 + w1 x1+ w2 x2
 Solvable by extension of least square method or using SAS, S-Plus
 Many nonlinear functions can be transformed into the above
Nonlinear Regression
 Some nonlinear models can be modeled by a polynomial function
 A polynomial regression model can be transformed into linear regression model. For
example,
y = w0 + w1 x + w2 x2 + w3 x3
convertible to linear with new variables: x2 = x2, x3= x3
y = w0 + w1 x + w2 x2 + w3 x3
 Other functions, such as power function, can also be transformed to linear model
 Some models are intractable nonlinear (e.g., sum of exponential terms)
 possible to obtain least square estimates through extensive calculation on more
complex formulae

QUESTIONS

UNIT II
PART - A
1. What is association rule mining? Explain with example.
2. How association rules are mined in large databases?
3. Design a method that mines the complete set of frequent item sets without candidate
generation?
4. Explain iceberg queries with example.
 5. Differentiate mining quantitative association rules and distance based association rules.
6. Explain how to improve the efficiency of apriori algorithm?
7. List out different kinds of constraint based association mining.
8. How to transform from association analysis to correlation analysis?
9. What is the difference between classification and prediction?
10. Explain the issues involved in classification and prediction.
11. Explain classification related concepts in terms of association rule miming.
12. What is a “decision tree”.
13. Define Bayes Theorem.
14. Explain K-nearest neighbor classifiers.
15. What is linear regression.
16. What is SVM?
17. Define Lazy Learners.

PART - B
1. How classification can be done using decision tree induction?
2. Explain in detail the Bayesian classification techniques.
3. Give an overview of the different classification approaches.
4. Explain market basket analysis with an motivating example for association rule mining.
5. Explain all the ways to classify association rule mining.
6. Write an apriori algorithm with an example.
7. Explain mining multi level association rules from transactional database.
8. Explain how to mine multidimensional association rules from relational databases and
data warehouses?
9. Write an algorithm to mine frequent item set without candidate generation. Give
suitable example.
10. Explain: Classification by Back propagation and Associative Classification.
11. Explain in detail, the Decision Tree Induction.
12. Briefly describe how classification is done using Bayes Method.

REFERENCES

1. www.tutorialspoint.com/dwh/dwh_overview.htm

2. http://study.com/academy/lesson/data-warehousing-and-data-mining-information-for-business-
intelligence.html

3. http://www.dei.unipd.it/~capri/SI/MATERIALE/DWDM0405.pdf.

4. https://www.cse.iitb.ac.in/infolab/Data/Talks/krithi-talk-impact.ppt