International Journal of Engineering and Technical Research (IJETR)

ISSN: 2321-0869, Volume-2, Issue-12, December 2014

Automatic Document Clustering and Knowledge

Discovery
Ms. Khureshi Rukhsar Afreen Intizar Ahmed , Mrs. Vishwakarma Pinki

Abstract There always exists a need to organize a large set

II. LITERATURE SURVEY:
of documents into categories through clustering, so, document Document clustering is the task of arranging a set of
clustering is done such that intra cluster documents are similar documents into clusters so that intra cluster documents are
to each other than inter cluster documents. Once clusters are similar to each other than inter cluster documents. There are
formed now reading each document manually to identify what two common clustering algorithms. 1) Partitioning algorithms
the documents are actually all about is a time consuming task
in which clusters are computed directly. Clustering is done by
therefore knowledge can be discovered in the form of association
rules so that the topic for the cluster can be discovered by just iteratively swapping objects or groups of objects between the
having a glance at the rules. Hence, our paper basically deals clusters. 2) The hierarchical based algorithms in which a
with two phases : clustering and knowledge discovery. In the hierarchy of clusters is build. Every cluster is subdivided into
first phase of clustering, we introduce a algorithm where the child clusters, which form a partition of their parent cluster.
number of clusters are not to be given as input as in other Different clustering algorithms produce different results with
partitioning algorithms and in second phase of knowledge different features. Hierarchical algorithms are slower than
discovery, we have used GARW algorithm to generate partitioning algorithms but they give better accuracy.
association rules so that we can come to know what the cluster is Therefore, a clustering algorithm should be chosen based on
all about .
the applications, as the desirable features are application
Section 1 is introduction about the topic, section 2 gives
details about the literature survey, section 3 is about the dependent.
proposed system, section 4 is all about the experimental results.
The author of [1] has compared kmeans algorithm with his
Index Terms About four key words or phrases in proposed algorithm and proved that his algorithm is much
alphabetical order, separated by commas. better than k means because it shows better results on large
number of documents and also resolved zero clustering issue.
I. INTRODUCTION The author of [2] has proposed a novel k nearest
Internet is very important in todays life. Finding relevant neighbor algorithm As the name suggests the method is
information on the WWW is challenging. To respond to a user dependent on the parameter k which can drastically change
query, it is difficult to search through the large number of the output as we vary its values. When the training set
returned documents with the presence of todays search contains classes of unequal sizes, the test data is likely to get
engines. There is a need to organize a large set of documents classified to a class which has more samples than the actual
into categories through clustering. Document class it belongs to, if that actual class has less number of
clustering (or Text clustering) is automatic document samples. The parameter k in this method depends on the size
organization, and fast information retrieval . To read all the of the smallest class sample. The proposed algorithm tries to
available text documents in text databases and group them take care of this limitation by first considering the size of the
effectively and then extract knowledge from them on manual smallest class and then selecting the k nearest neighbors.
basis is a quite difficult and time consuming task and hence
text mining techniques such as clustering are used. After the The author of [3] has used term frequency inverse
documents are clustered into various categories , now its the document frequency weighting scheme. Here weighting
time to discover knowledge from them in the form of scheme plays very important role as it helps in removing
association rules. Finding association rules in text documents number of words which are less important leading to the
can be useful in a number of contexts . For example, criminal dimensionality reduction.
investigations, market trends, intrusion detection , etc. by just
having a glance at these association rules we can come to According to the author of [4], the disadvantage in k-means
know that the documents of a particular cluster are what all algorithm is that, the accuracy and efficiency is varied with
about i.e we can discover a topic for the cluster, and if we are the choice of initial clustering centers on choosing it
interested in the topic of that cluster we can have a detailed randomly. So in his paper, less similarity based clustering
view of those documents without concerning with the method is proposed for finding the better initial centroids and
documents of other clusters. to provide an efficient way of assigning the data points to
suitable clusters with reduced time complexity.

Existing algorithms for k-means clustering are slow and do

Manuscript received November 30, 2014. not scale to large number of data points and converge to
Ms.Khureshi Rukhsar Afreen Intizar Ahmed, Computer different local minima based on the initializations. A fast
Engineering, Shah and Anchor Kutchchi Engineering College, Mumbai greedy k-means algorithm can attack both these drawbacks,
University,Mumbai, India,8286203753.
but it is a limitation when the algorithm is used for large
Mrs.VishwakarmaPinki, Computer Engineering, Shah and Anchor
Kutchchi Engineering College, MumbaiUniversity number of data points, So the author of [5] introduced an

16 www.erpublication.org
 Automatic Document Clustering and Knowledge Discovery

efficient method to compute the distortion for this algorithm

and his experiment results show that the fast greedy algorithm TEXT DOCUMENTS
is superior to other methods and can help users to find the
relevant documents more easily than by relevance ranking. PRE PROCESSING

The author of [8][9] has introduced EART (extraction of FEATURE EXTRACTION

association rules from text) technique , which uses term
frequency-inverse document frequency as weighting scheme. VECTOR SPACE MODELLING
It helps in removing the words which are of least importance
thereby leading to important rules which helps in knowledge SIMILARITY MEASURE
discovery.
CLUSTERING ALGORITHM
The author of [10] has used Association rules discovery
techniques to compare the students performance in the Fig 1: Clustering Phase
subjects common at Graduation and Post Graduation level
and will predict the factors which can explain their success or a) Preprocessing :
failure. The mined association rules reveal various factors like Preprocessing steps take as input a plain text document and
students interest, curriculum design; teaching and assessment output a set of tokens to be included in the vector model. The
methodologies that can affect students who have failed to following are the steps of preprocessing [3][4]:
attain a satisfactory level of performance in the Post
Graduation level. Filtering: Remove special characters and punctuation marks
from the plain text document.
A new semantic-based model that analyzes documents Tokenization: Split sentences into individual tokens or words.
based on their meaning is introduced by the author of [11]. He Stop word removal: The words (e.g. "and", "the" etc.) which
proposed the model that analyzes terms and their do not convey any meaning as a dimension in the vector space
corresponding synonyms and/or hypernyms on the sentence are removed.
and document levels. In this model, if two documents contain Stemming: Reduce the words to their base form, or stem. For
different words and these words are semantically related, the example, the words computer, computing", are reduced to
proposed model can measure the semantic-based similarity the stem compute using Porter's algorithm.
between the two documents. The similarity Pruning: Remove the words with very low frequency in the
between documents relies on a new semantic-based similarity dataset.
measure which is applied to the matching concepts between
documents. b) Feature Vector:
To create Vector space model, a subset of the tokens from the
The author of [12] has discussed the structural and semantic dataset is required which is known as Feature vector. We have
similarity of various XML DTDs for efficient clustering used frequency based method for feature vector extraction.
mechanism. As weIl as by analysing the structure of DTDs,
the nested and repeated nodes in the tree has been eliminated c) Vector space Model and Similarity Measure:
in the pre-processing mode and efficient clusters has been
It is also known as TF-IDF model i.e. term frequency inverse
made based on similarities of XML trees. This study proves
document frequency model. It is the standard retrieval
group of structurally similar XML documents provides most
technique used in text mining area. In this model, each
precise results in large document collections and described an
document is represented as an n-dimensional vector using the
efficient clustering strategy which group most similar DTDs
feature vector. The value of each element in the vector reflects
where preliminary evaluation made with the synthetic data.
the importance of the corresponding feature in the document.
Text documents are converted into machine acceptable,
III. PROPOSED SYSTEM:
mathematical representations after the transformation. The
Our proposed system consists of 2 phases: similarity between documents can be measured by calculating
A. Clustering. the distance between document vectors [1]. Now that D is a
vector => Given a doc, find vectors (docs) near it.
B. Knowledge Discovery. Intuition: Documents that are close together in vector space
talk about the same things.

A. CLUSTERING:
Document or Text Clustering is an unsupervised technique in
which no input output patterns are pre - defined. The
clustering is done in an efficient manner if the documents of
intra cluster are more similar than the inter-cluster documents.
Clustering differs from categorization as the documents are
clustered on the fly instead of having trained datasets.

Fig 2 : Vector Space Model

17 www.erpublication.org
 International Journal of Engineering and Technical Research (IJETR)
ISSN: 2321-0869, Volume-2, Issue-12, December 2014
(where, ti= terms and di=documents..i = 1,2,3,) i. Start with document d[i,j] in the matrix with CS
A weighting scheme for document weights is defined by value 1, where (i=j).
considering the frequency of terms within a document. If the ii. Take the mean of row or column the obtained value
Documents contain the same keywords they are similar will be a threshold.
Therefore we can use the term frequency tf(i,j) i.e. number of iii. Now search for documents in the row or column
times a term i occurs in a document j. The term frequency is whose value satisfies the above threshold.
normalized with respect to the maximal frequency of all terms iv. Cluster the documents who satisfy the threshold in
occurring in a document [1]. the category Ci = {d1,d2, dm,. | where m is any
tf (i, j) = freq (i,j) number between 1 to n}.
max {f(k,j)} v. For all the documents that are clustered in first
We should also consider how frequent a term is in the category , their CS value will be changed to some
document collection of size n. The Document frequency (dfi) other value, and now these documents will not be
of a term is the number of documents in which term i occurs. considered further at any stage to avoid
If D is the number of documents in a database then, idf (i, j) = redundancy. Documents with CS value 1 are
log (D/ dfi) considered for the formation of the clusters.
Term weight is given by: Wi = tfi * log (D/dfi). vi. Repeat steps i to v for making more clusters.
Now, the Cosine similarity (the cosine of the angle of two 3. /* generation of number of clusters */
vectors) has to be calculated by the following formula: i. Initially count =0
Cosine similarity between two vectors x and y is calculated ii. After formation of each cluster count is
as: incremented by 1

n iii. Finally display count
x y xi yi 4. Display output in the form c1={d1, d2, d6,
cos( x , y ) i 1
d8..} , and so on.
x y

n n
x2
i 1 i i 1
yi2 B. KNOWLEDGE DISCOVERY:
Knowledge discovery from textual database refers generally
Hence, for any two given documents dj and dk, their similarity to the process of extracting interesting information from
is: unstructured text documents. Here, knowledge is discovered
in the form of association rules. These rules can be extracted

n
d j dk wi , j wi ,k from texts using GARW (Generation of association rules
sim(d j , d k ) i 1
based on weighting scheme) algorithm. These rules would

n n
d j dk wi2, j wi2,k give us the idea regarding the documents of the clusters, and
i 1 i 1 would help us in making correct choice for the topics in which
we are interested.
It has the following properties :
a) If two vectors coincide completely their similarity should This phase describe a way for finding from a collection of
be maximal, i.e. 1. documents by automatically extracting association rules from
b) If two vectors have no keywords in common, the similarity them . Given a set of keywords A ={ w1 ,w2 ,..., wn} and a
should be minimal, i.e. 0. collection of indexed documents D = {d1,d2,..., dm }, where
c) In other cases the similarity should be between 0 and 1. each document di is a set of keywords such that di A. Let
Wi be a set of keywords. A document di is said to contain Wi
if and only if Wi di . An association rule is an implication
d) Proposed Algorithm: of the form Wi Wj where Wi A , Wj A and Wi Wj
Input : Document set D = { d1, d2, d3, , dn} = . There are two important basic measures for association
Output : No of clusters and Set of cluster numbers C along rules, support(s) and confidence(c). The rule Wi Wj has
with documents associated. support s in the collection of documents D if s% of documents
1. Preprocessing in D contain Wi Wj . The support is calculated by the
following formula:
2. CS ixj = Support (Wi,Wj) =Support count of Wi Wj / Total number of
documents D.
and the confidence of a rule is defined as: conf(xy) =
1 d(1,2) d(1,3) ... d(1,n) supp(X u Y) / supp(X). [2][3]
d(2,1) 1 d(2,3)..d(2,n)
d(3,1) d(3,2) 1 d(3,n) An association rule-mining problem is broken into two steps:
. . . 1) generate all the keyword combinations (keywordsets)
. . . whose support is greater than the user specified minimum
d(n,1) d(n,2) d(n,3) 1 support (called minsup). Such sets are called the frequent
keywordsets .
2) use the identified frequent keywordsets to generate the
rules that satisfy a user specified minimum confidence (called
minconf). The frequent keywords generation requires more
3. /* clustering */ effort and the rule generation is straightforward.

18 www.erpublication.org
 Automatic Document Clustering and Knowledge Discovery

We have an algorithm for Generating Association Rules Therefore, it means time required to make clusters
based on Weighting scheme (GARW). and the space required in memory for storing these
The GARW algorithm is as follows: clusters is also less.
Let N be the number of keywords that satisfy the threshold
value of weight. 12
Store the n keywords in a hashmap along with their 10
frequencies in all document and their weight values
8
TF-IDF.
Scan the hashmap and find all keywords that satisfy the 6
threshold minimum support. k means
4
Generate candidate keywords from large frequent Proposed
2
keywordset..
Compare the frequencies of candidate keyword sets with 0

Cluster 8
Cluster 2

Cluster 4
Cluster 1

Cluster 3

Cluster 5
Cluster 6
Cluster 7
minimum support
Generate different association rules from candidate
keywordset , that satisfy the threshold minimum
confidence.
X axis : cluster no
The extracted association rules can be reviewed in textual Y axis : no of documents
format or tables. The extracted association rules contain
important features and describe the informative news 4
included in the documents collection. 3.5
The GARW algorithm reduces the execution time in 3
comparable to the Apriori algorithm because it does not make
multiple scanning on the original documents like Apriori but 2.5
it scan only the file or hashmap which contains all the 2
keywords that satisfy the threshold weight value and their 1.5
frequencies in each document [8]. Apriori-based system
generates all frequent keywordsets from all keywords in the 1
documents that are important and unimportant. This leads to 0.5
extract interesting and uninteresting rules. In contrast, the 0
system based on the GARW algorithm generates all frequent kmeans proposed
keywordsets from mostly important keywords in the
documents based on the weighting scheme. Here, the X axis: algorithm type
weighting scheme plays an important role for selecting Y axis : time in seconds for cluster generation
important keywords for association rules generation. This
leads to extract the more interesting rules in short time. Phase 2: Knowledge Discovery
Here, we have used GARW algorithm for generation of
IV. EXPERIMENTAL RESULTS: association rules, in this algorithm first the keywords who
For experimental purpose we have used the 20 newsgroup donot satisfy the threshold weight is removed thus, only
dataset, from that we retrieved some documents and used important keywords will be remained hence we get important
them for our experiments. rules.
For analysis of results, we applied k means and our However,if apriori algorithm is used for
proposed algorithm on these dataset. We got the generation of rules it would lead to all important and
following observations: unimportant rules thereby, consuming time and memory as
well.
Phase 1: Clustering So, on each cluster obtained in phase 1, we apply both these
a) K means: algorithm for analysis purpose and we got the following
If we give k = 10 , our output consists of 10 clusters observations.
whereas maximum clusters are 4 only (according to
the documents taken for experiment) , similarly if we
give k = 15, 15 clusters will be obtained. Some of the
clusters would be singleton( i.e. clusters with just
one document) as well.
Therefore, it means time required to make so many
clusters and the space required in memory for storing
so many clusters is also more.

b) Proposed algorithm:
Here, there is no need to specify k (the number of X axis : cluster no
clusters) and we get 4 clusters from the input text Y axis : no of rules generated.
dataset.

19 www.erpublication.org
 International Journal of Engineering and Technical Research (IJETR)
ISSN: 2321-0869, Volume-2, Issue-12, December 2014
V. CONCLUSION:
Our proposed system automatically groups unknown text
dataset in optimum clusters as it does not takes number of
clusters as input. It also generates association rules for each
cluster so that it would discover the knowledge or the topic
about the documents of the clusters and would give the idea to
the user about them, so that if the user is interested in that
topic he would have a detailed information about that cluster
without concerning the other clusters, thereby manual effort
and time will be reduced.

REFERENCES
[1] Madhura Phatak , Ranjana Agrawal , A Novel Algorithm for
Automatic Document Clustering , 2013 3rd IEEE International
Advance Computing Conference (IACC).
[2] Ms. Anjali Ganesh Jivani, The Novel k Nearest Neighbor Algorithm
, 2013 International Conference on Computer Communication and
Informatics (ICCCI -2013).
[3] Ms. Vaishali Bhujade, Prof. N. J. Janwe, Ms. Chhaya Meshram ,
Discriminative Features Selection in Text Mining Using TF-IDF
Scheme , International Journal of Computer Trends and Technology-
July to Aug Issue 2011.
[4] Manjot Kaur , Navjot Kaur Web Document Clustering Approaches
Using K-Means Algorithm , International Journal of Advanced
Research in Computer Science and Software Engineering , Volume
3, Issue 5, May 2013 ISSN: 2277 128X.
[5] Hongwei Yang , A Document Clustering Algorithm for Web Search
Engine Retrieval System , 2010 International Conference on
e-Education, e-Business, e-Management and e-Learning.
[6] http://pyevolve.sourceforge.net/wordpress/?p=97
[7] http://people.csail.mit.edu/jrennie/20Newsgroup.
[8] Vaishali Bhujade , N.J. janwe , Knowledge discovery in text mining
using association rules extraction , 2011 IEEE computer society.
[9] Hany Mahgoub, Dietmar Rsner, Nabil Ismail and Fawzy Torkey , A
Text Mining Technique Using Association Rules Extraction ,
International Journal of Information and Mathematical Sciences 4:1
2008.
[10] Dr. Varun Kumar1, Anupama Chadha2 , Mining Association Rules
in Students Assessment Data , IJCSI International Journal of
Computer Science Issues, Vol. 9, Issue 5, No 3, September 2012.
[11] Shady Shehata , AWordNet-based Semantic Model for Enhancing
Text Clustering , 2009 IEEE International Conference on Data
Mining Workshops .
[12] Mary Posonia A , Dr V LJyothi , Structural- based Clustering
Technique of XML
[13] Documents 2013 International Conference on Circuits, Power and
Computing Technologies [ICCPCT-2013].

Ms.Khureshi Rukhsar Afreen Intizar Ahmed, ME pursuing,

Computer Engineering, Shah and Anchor Kutchchi Engineering College,
Mumbai University,Mumbai, India.

Assistant Professor Mrs.Vishwakarma Pinki , Computer Engineering,

Shah and Anchor Kutchchi Engineering College, MumbaiUniversity

20 www.erpublication.org