Lazy Classifiers Using P-trees *

William Perrizo1, Qin Ding2, Anne Denton1

1 2
Department of Computer Science Department of Computer Science
North Dakota State University Penn State Harrisburg
Fargo, ND 58015 Middletown, PA 17057
{William.Perrizo, Anne.Denton}@ndsu.nodak.edu qding@psu.edu

Abstract a new sample needs to be classified. It differs

Lazy classifiers store all of the training samples from eager classifiers, such as decision tree
and do not build a classifier until a new sample induction, which build a general model (such as
needs to be classified. It differs from eager a decision tree) before receiving new samples.
classifiers, such as decision tree induction, which
build a general model (such as a decision tree) K-nearest neighbor (KNN) classification is a
before receiving new samples. K-nearest typical lazy classifier. Given a set of training
neighbor (KNN) classification is a typical lazy data, a k-nearest neighbor classifier predicts the
classifier. Given a set of training data, a k- class value for an unknown tuple X by searching
nearest neighbor classifier predicts the class the training set for the k nearest neighbors to X
value for an unknown tuple X by searching the and then assigning to X the most common class
training set for the k nearest neighbors to X and among its k nearest neighbors.
then assigning to X the most common class
among its k nearest neighbors. Lazy classifiers Lazy classifiers are simple and effective.
are faster at training time than eager classifiers, However, it’s slow at predicating time since all
but slower at predicating time since all computation is delayed to that time. In this
computation is delayed to that time. In this paper, we introduce two lazy classifiers, P-tree
paper, we introduce approaches to efficient based k-nearest neighbor classifier (PKNN), and
construction of lazy classifiers, using a data Podium Incremental Neighbor Evaluator (PINE).
structure, Peano Count Tree (P-tree)*. P-tree is a We use a data structure, Peano Count Tree (P-
lossless and compressed representation of the tree), for efficient construction of those lazy
original data that records the count information classifiers. P-tree is a lossless and compressed
to facilitate efficient data mining. With P-tree representation of the original data that records
structure, we introduced two classifiers, P-tree the counts to facilitate efficient data mining.
based k-nearest neighbor classifier (PKNN), and
Podium Incremental Neighbor Evaluator (PINE). The rest of the paper is organized as follows.
Performance analysis shows that our algorithms Section 2 reviews the P-tree structure. Section 3
outperform classical KNN methods. introduces two classifiers, PKNN and PINE.
Performance analysis is given in Section 4.
Section 5 concludes the paper.
Keywords:
Lazy classifier, classification, nearest neighbor
2. THE P-TREE STRUCTURE REVIEW

1. INTRODUCTION We use a structure, called a Peano Count

Tree (P-tree) [1, 2], to represent the spatial data.
Classifiers can be divided into two We first split each attribute value into bits. For
categories, lazy classifiers and eager classifiers. example, for image data, each band is an
In lazy classifiers, no general model is built until attribute, and the value is represented as a byte (8
bits). The file for each individual bit is called a
* bSQ file. For an attribute with m-bit values, we
Patents are pending on the P-tree technology. have m bSQ files. We organize each bSQ bit
This work is partially supported by GSA Grant file, Bij (the file constructed from the jth bits of
ACT#: K96130308.

1
ith attribute), into a P-tree. An example is given [3]. The NOT operation is a straightforward
in Figure 1. translation of each count to its quadrant-
complement. The AND and OR operations are
shown in Figure 2.
11 11 11 00 39
11 11 10 00 __________/ / \ \_______
11 11 11 00 / _____/ \ ___ \
16 ____8__ _15__ 0 P-tree-1: m P-tree-2: m
11 11 11 10 / / | \ / | \ \ ______/ / \ \______ ______/ / \ \_____
11 11 00 00 3 0 4 1 4 4 3 4 / / \ \ / / \ \
11 11 00 00 //|\ //|\ //|\ / / \ \ / / \ \
11 11 00 00 1 m m 1 1 0 m 0
1110 0010 1101
01 11 00 00 / / \ \ / / \ \ / / \ \
m 0 1 m 11 m 1 11 1 m
//|\ //|\ //|\ //|\
m
1110 0010 1101 0100
_____________/ / \ \____________
/ ____/ \ ____ \
1 ____m__ _m__ 0
/ / | \ / | \ \ AND-Result: m OR-Result: m
m 0 1 m 1 1 m 1 ________ / / \ \___ ________ / / \ \___
//|\ //|\ //|\ / ____ / \ \ / ____ / \ \
1110 0010 1101 / / \ \ / / \ \
1 0 m 0 1 m 1 1
/ | \ \ / / \ \
1 1 m m m 0 1 m
//|\ //|\ //|\ //|\
Figure 1. P-tree and PM-tree 1101 0100 1110 0010

In this example, 39 is the count of 1’s in the

entire image, called root count. The numbers at Figure 2. P-tree Algebra
the next level, 16, 8, 15 and 16, are the 1-bit
counts for the four major quadrants. Since the
first and last quadrant is made up of entirely 1- The basic P-trees can be combined using
bits and 0-bits respectively (called pure1 and simple logical operations to produce P-trees for
pure0 quadrant respectively), we do not need the original values (at any level of precision, 1-
sub-trees for these two quadrants. This pattern is bit precision, 2-bit precision, etc.). We let Pb,v
continued recursively. Recursive raster ordering denote the P-tree for attribute, b, and value, v,
is called the Peano or Z-ordering in the literature where v can be expressed in any bit precision.
– therefore, the name Peano Count trees. The Using the 8-bit precision for values, Pb,11010011
process will definitely terminate at the “leaf” can be constructed from the basic P-trees as:
level where each quadrant is a 1-row-1-column Pb,11010011 = Pb1 AND Pb2 AND Pb3′ AND Pb4
quadrant. For each band, assuming 8-bit data AND Pb5′ AND Pb6′ AND Pb7 AND Pb8
values, we get 8 basic P-trees, one for each bit where ′ indicates the NOT operation. The AND
position. For band Bi we will label the basic P- operation is simply the pixel-wise AND of the
trees, Pi,1, Pi,2, …, Pi,8, where Pi,j is a lossless bits.
representation of the jth bit of the values from the
ith band. However, the Pij provide much more Similarly, the data in the relational format
information and are structured to facilitate many can be represented as P-trees also. For any
important data mining processes. combination of values, (v1,v2,…,vn), where vi is
from attribute-i, the quadrant-wise count of
For efficient implementation, we use a occurrences of this combination of values is
variation of P-trees, called PM-tree (Pure Mask given by:
tree), in which mask instead of count is used. In P(v1,v2,…,vn) = P1,v1 AND P2,v2 AND …
the PM-tree, 3-value logic is used, i.e, 11 AND Pn,vn
represents a pure1 quadrant, 00 represents a
pure0 quadrant and 01 represents a mixed
quadrant. To simplify, we use 1 for pure1, 0 for 3. LAZY CLASSIFIERS USING P-TREES
pure0, and m for mixed. This is illustrated in the 3.1 Distance Metrics
3rd part of Figure 1. P-tree algebra contains
operators, AND, OR, NOT and XOR, which are The distance metric is required to determine
the pixel-by-pixel logical operations on P-trees the closeness of samples. Various distance

2
metrics have been proposed. For two data 2) Find the k nearest neighbors using the
points, X = <x1, x2, x3, …, xn-1> and Y = <y1, y2, selected distance metric.
y3, …, yn-1>, the Euclidean similarity function is 3) Find the plurality class of the k-nearest
n −1 neighbors.
defined as d 2 ( X ,Y ) = ∑ (x
i =1
i − yi ) . It can
2
4) Assign that class to the sample to be
classified.
be generalized to the Minkowski similarity
Database scans are performed to find the
n −1
d q ( X ,Y ) = q ∑w xi − y i . nearest neighbors, which is the bottleneck in the
q
function, i If
method. In PKNN, by using P-trees, we can
i =1
quickly calculate the nearest neighbor for the
q = 2, this gives the Euclidean function. If q = 1, given sample. For example, given a tuple <v1,
it gives the Manhattan distance, which is v2, …, vn>, we can get the exact count of this
n −1
d1 ( X , Y ) = ∑ xi − yi . If q = ∝, it gives the
tuple by calculating the root count of
P(v1,v2,…,vn) = P1,v1 AND P2,v2 AND …
i =1
AND Pn,vn
n −1
max function d ∞ ( X , Y ) = max xi − yi .
i =1 With HOBBit metric, we first calculate all
exact matches with the given data sample X, then
We proposed a metric using P-trees, called extend to data points with distance 1 (with at
HOBBit [3]. The HOBBit metric measures most one bit difference in each attribute) from X,
distance based on the most significant and so on, until we get k nearest neighbors.
consecutive bit positions starting from the left
(the highest order bit). The HOBBit similarity Instead of just getting exact k nearest
between two integers A and B is defined by neighbors, we take the closure of the k-NN set,
SH(A, B) = max{s | 0 ≤ i ≤ s ⇒ ai = bi}…(eq. 1 ) that is, we include all of the boundary neighbors
where ai and bi are the ith bits of A and B that have the same distance as the kth neighbor
respectively. to the sample point. Obviously closed-KNN is a
superset of KNN set. As we pointed out later in
The HOBBit distance between two tuples X the performance analysis, closed-KNN improves
and Y is defined by the accuracy obvisously.

d H ( X,Y ) = max{m - S H ( xi ,y i )} … (eq. 2)

n
3.3 PINE – Podium Nearest Neighbor
i =1
Classifier using P-trees
where m is the number of bits in binary
representations of the values; n is the number of In classical k-nearest neighbor (KNN)
attributes used for measuring distance; and xi and methods, each of the k nearest neighbors casts an
yi are the ith attributes of tuples X and Y. The equal vote for the class of X. We suggest that
HOBBit distance between two tuples is a the accuracy can be increased by weighting the
function of the least similar pairing of attribute vote of different neighbors. Based on this, we
values in them. propose an algorithm, called Podium Incremental
Neighbor Evaluator (PINE), to achieve high
3.2 PKNN – K-nearest Neighbor Classifier accuracy by applying a podium function on the
using P-trees neighbors.
The basic idea of k-nearest neighbor (KNN) The continuity assumption of KNN tells us
classification is that similar tuples most likely that tuples that are more similar to a given tuple
belongs to the same class (a continuity have more influence on classification than tuples
assumption). Based on some pre-selected that are less similar. Therefore giving more
distance metric, it finds the k most similar or voting weight to closer tuples than distant tuples
nearest training samples and assign the plurality increases the classification accuracy. Instead of
class of those k samples to the new sample. It considering the k nearest neighbors, we include
consists four steps: all of the points, using the largest weight, 1, for
those matching exactly, and the smallest weight,
1) Determine a suitable distance metric. 0, for those furthest away. Many weighting

3
functions which decreases with distance, can be operations instead of scanning the database. In
used (e.g., Gaussian, Kriging, etc). addition, with proposed closed-KNN, accuracy is
increased. In PINE method, podium function is
applied to different neighbors so accuracy is
4. PERFORMANCE ANALYSIS further improved.

We have performed experiments on the real Lazy classifiers are particularly useful for
image data sets, which consist of aerial TIFF classification on data streams. In data streams,
image (with Red, Green and Blue bands), new data keep arriving, so building a new
moisture, nitrate, and yield map of the Oaks area classifier each time can be very expensive. By
in North Dakota. In these datasets, yield is the using P-trees, we can build lazy classifiers
class label attribute. The data sets are available efficiently and effectively for stream data.
at [8]. We tested KNN with various metrics,
PKNN, and PINE. The accuracies of these
different implementations are given in Figure 3. 6. REFERENCES

Raw guessing KNN-Manhattan

[1] William Perrizo, Qin Ding, Qiang Ding,
KNN-Euclidean KNN-Max Amalendu Roy, “On mining satellite and other
KNN-HOBBit PKNN(closed-KNN-HOBBit) Remotely Sensed Images”, in Proceedings of
PINE (Ptree)
Workshop on Research Issues on Data Mining
Accuracy Comparison for KNN, PKNN and Knowledge Discovery, 2001, p 33-44.
and PINE [2] William Perrizo, Peano Count Tree
80 Technolgy, Tchnical Report NDSU-CSOR-TR-
70
01-1, 2001.
[3] Maleq Khan, Qin Ding, William Perrizo, “k-
60
Nearest Neighbor Classification on Spatial Data
Accuracy (%)

50
Streams Using P-Trees”, PAKDD 2002,
40 Springer-Verlag, LNAI 2336, 2002, pp. 517-528.
30
[4] Dasarathy, B.V., “Nearest-Neighbor
Classification Techniques”, IEEE Computer
20
Society Press, Los Alomitos, CA, 1991.
10 [5] M. James, “Classification Algorithms”, New
0 York: John Wiley & Sons, 1985.
256 1024 4096 16384 65536 262144
[6] M. A. Neifeld and D. Psaltis, “Optical
Training Set Size (number of tuples) Implementations of Radial Basis Classifiers”,
Applied Optics, Vol. 32, No. 8, 1993, pp. 1370-
Figure 3. Accuracy comparison for KNN, PKNN 1379.
and PINE [7] TIFF image data sets. Available at
http://midas-10.cs.ndsu.nodak.edu/data/images/
We see that PKNN performs better than [8] Jiawei Han and Micheline Kamber, Data
KNN using various metrics, including Euclidean, Mining: Concepts and Techniques, Morgan
Manhattan, Max and HOBBit. PINE further Kaufmann Publishers, 2001.
improves the accuracy. Especially when the
training set size increases, the improvement is
more apparent. Both PKNN and PINE work
well compared to raw guessing, which is 12.5%
in this data set with 8 class values.

5. CONCLUSION

In this paper, we introduce two lazy

classifiers using P-trees, PKNN and PINE. In
PKNN, P-tree structure facilitates efficient
computation of the nearest neighbors. The
computation is done by performing fast P-tree