International Journal of Computer Applications (0975 – 8887)

Volume 87 – No.3, February 2014

Algorithm for Tracing Visitors’ On-Line Behaviors for

Effective Web Usage Mining
S. Umamaheswari S. K. Srivatsa
Research Scholar Senior Professor
SCSVMV University St.Joseph College of Engineering,
Kanchipuram, India Chennai, India

ABSTRACT 1.1 Why Mine the Web?

User behavior identification is an important task in web usage There are enormous wealth of information on web such as
mining. Web usage mining is also called as web log mining. financial information like stock quotes, book/CD/video stores,
The web logs are mainly used to identify the user behavior. restaurant information and car prices. Even though it has
There are so many pattern mining methods which enable this many sort of information, the web poses great challenges for
user behavior identification. The preprocessing techniques effective resources and knowledge discovery. The web seems
will maximize the accurate and quality of pattern mining to be too huge for effective data warehousing and mining.
methodologies. In existing algorithms, the preprocessing Also the complexity of web pages is far greater than that of
concepts are applied to calculate the unique user’s count, to any old text documents. Only a small portion of the
minimize the log file size and to identify the sessions. The information on the web is truly relevant [4].It is possible to
newly proposed algorithm is Visitors’ Online Behavior get lots of data on user access patterns and also possible to
(VOB) which identifies user behavior, creates user cluster and mine interesting nuggets of information. The process of
page cluster, and tells the most popular web page and least searching the web is illustrated in the following “Fig. 1”.
popular web page. This paper brings into discussion about the
basic concepts of web mining, web usage mining, general data
preprocessing, how to preprocess the web data, what are the
various existing preprocessing techniques and the proposed
VOB algorithm.
Content
aggregators C
Keywords O
Data preprocessing methods, Web mining, Web usage mining,
Web usage data, Web log. msn N
1. INTRODUCTION S
World wide web is a very large, widely distributed, global W U
information service centre to facilitate the services such as Google
E M
news, advertisements, consumer information, financial
management, education, government, e-commerce, etc. It B E
consists of hyper-link information, access and usage
information. World wide web gives enough number of rich
R
Yahoo!
sources of data for data mining .Web Mining is one of the S
data mining technique used to automatically discover and
extract information from web documents or services. This
refers a process by which we can discover useful information
from the world wide web and it’s usage patterns. Here the
data objects are linked together for interactive access. The
subtasks of web mining are resource finding, information Web has recently become a powerful platform for retrieving
selection and preprocessing, generalization and analysis. information and discovering knowledge from web data. The
Resource finding refers the task of retrieving intended web idea of discovering useful patterns in data may have many
documents. Information selection and preprocessing is an names such as data mining, knowledge extraction,
automatic selection and preprocessing of particular information discovery, information harvesting, data
information from retrieved web resources. Also the archeology, and data pattern processing[12].
generalization represents an automatic discovery of patterns in
web sites and analysis is the validation and interpretation of 1.2 Web Mining Applications
mined patterns. Generally data mining techniques are used to Web mining applications are listed such as to target potential
make the web more useful and more profitable for some and customers for electronic commerce, to enhance the quality
to increase the efficiency of our interaction with the and delivery of internet information services to the end user,
web.Some of the data mining techniques are association to improve the web server program’s performance, to identify
rules, sequential patterns, classification, clustering and outlier the potential prime advertisement locations, to facilitate
discovery. Nowadays these techniques and concepts have adaptive sites, to improve site design, to do fraud detection
employed in many applications to the web like e-commerce, and to predict the user’s actions.
information retrieval and network management. 1.3 Web Mining Issues
Nowadays the web became so popular and used by many
categories of people which includes school students and the
business men also. The number of users who are employing

22
 International Journal of Computer Applications (0975 – 8887)
Volume 87 – No.3, February 2014

the web is increasing at exponential speed [3].On the web,

many different types of data such as images, text, audio/video,
XML and HTML are used. Web datasets can be very large. It
is in the range of tens to hundreds of tera bytes. So it cannot
Website
mine on a single server. There is a need of large forms of Reorganization
servers.

2. WEB USAGE MINING

Web usage mining is a category of web mining technique to
discover interesting usage patterns from the secondary data
derived from the interactions of the users while surfing the
web.
The web pages contain information. Here actually the links Prefetching Personalization
are ‘roads’. It tells how the people navigate the internet. The
information on navigation paths is available in log files. Logs web pages
can be mined from a client or a server perspective .It is aimed
to discover user ‘navigation patterns’ from web data and to
predict the user’s behavior while the user interacts with the Figure 3. Website usage analysis
web. Also it helps to improve large collection of resources
[5].The web usage mining techniques are to construct Many organizations have been supported by the analysis of
multidimensional view on the weblog database, to perform user’s browsing patterns for the purpose of giving
data mining on web log records and also to conduct studies personalized recommendations of web pages. Generally the
for analyzing the system performance, etc. Some of the usage-based personalized recommendation gives solution to
frequently used techniques are such as data collection, data many of the problems occurred in the web [13, 14, 15].It has
preparation and data cleaning. The web usage mining process created an interest between the researchers to do research. The
is given in the following “Fig. 2”. recommendation systems listen the information overload by
suggesting pages that fullfills the user’s requirement. In recent
days, the web usage mining has great potential and frequently
employed for the tasks like web personalization, web pages
pre-fetching and website reorganization, etc [16]. Data
Site sources for web usage mining are obtained in three ways [12].
Web
data In server level, the server keeps the client request details. At
serv the client level, the client itself forwards data about user’s
Usage behavior to a database .It can be accomplished by using either
er
pattern an ad-hoc browsing application or through client side
Log application which runs on the standard browsers. In the proxy
s level, the proxy side maintains user behavior information.
Even though the web data is taken from many users on
various web sites, only the users whose web clients pass
through the proxy.
2.2 Web usage data
Data Generally the web pages, intra page structures, inter page
Data Min structures and usage data are the input used in web usage
Prepar mining. Other forms of web data resides as profiles,
cleani ing registration information and cookies. Web usage data is
ation
ng referred as the collective data about how a user utilizes a web
site through his mouse and keyboard. This data can also be
available in form of web server logs, referral logs,
registration-files and index server logs and cookies.

2.1 What is the need for tracing visitors’ 2.3 Web log
The aim of web log file is to create user profile by allowing
on-line behaviors in web usage mining their browsing similarities with previous users. Before the
It must to trace the visitors’ on-line behaviors for website
data mining process, it is required to clean, condense and
usage analysis. Actually it is an analysis to get knowledge
transform the raw data of weblog before performing data
about how visitors use website which could provide
mining. Weblog information can be integrated with web
guidelines to web site reorganization and helps to prevent
content and web structure mining to help webpage ranking
disorientation. It also helps to the designers in placing the
and web document classification. The interaction details of
important information where the visitors look for it. It has to
users with website are recorded automatically in web servers
be done for pre-fetching and caching web pages. Also it
as the form of weblogs [2]. Weblogs are kept as in form of
provides adaptive website (personalization). This is
line of text in web server, proxy server and browser
represented in the figure “Fig 3.” given below.
[8].Various forms of logs are server access logs, server
referrer logs, agent logs, client-side cookies, user profiles,
search engine logs and database logs. These are considered as
input for knowing the end user behavior in web usage mining.
Log files are those files that list the actions that have been

23
 International Journal of Computer Applications (0975 – 8887)
Volume 87 – No.3, February 2014

occurred [18].Log files hold many parameters which have the unwanted data and shapes the required data by filling in
employed in recognizing user browsing patterns. Some of the missing values, smoothing noisy data, identifying or removing
parameters are user name, visiting path traversed, timestamp, outliers and resolving inconsistencies. Always the dirty data
page last visited, success rate, user agent, URL and request can make confusion while processing it in the mining process.
type [17].
3.2 Data integration
2.4 Transfer / Access Log Many categories of databases, data cubes or files have been
The information on user’s request from their web browsers is collected and integrated together in this step.
stored in transfer/access log.
3.3 Data transformation
Table 1. Transfer/Access Log It actually pointed by the process of normalization and
aggregation.
Amount Status
Host File
Time Date of data of
name requested
Transferred report 3.4. Data reduction
It leads to the reduced representation based on the volume of
data collected and processed. But it gives the same or similar
analytical results.
2.5 Referrer Log
The recorded two fields of referrer log are URL and referrer 3.5. Data discretization
URL. It is a section in the data reduction step. This is done only for
Table 2. Referrer Log the case of numerical data not for all types of data.
4. PREPROCESSING OF WEB USAGE
URL Referrer URL DATA
Generally in the web usage mining, the preprocessing [9] is
considered as an essential task and treated as an idea to reach
the goal. As it was suggested in referred paper [1], the
intelligent system web usage preprocessor splits the human
2.6 Error Log and search engine accesses before using the preprocessing
The list of errors and requests which have failed are collected techniques. In the recent days, it is not possible to get good
in error log. Not only for the page which holds links to a file quality data. Also there is no better result for mining the
that does not exist, but also for the user who is not permitted quality data. But the quality decisions have been taken
to access a particular page, the user request may fail. It is depending upon the quality of data. The duplicate or
depicted in the following “Fig 4.” missing data may create incorrect or even misleading
statistics. Also the data warehouse requires consistent
integration of quality data. Moreover the data extraction,
cleaning, and transformation take the maximum of the
Request work in building a data warehouse. It is depicted in the
SERVER CLIENT following “Fig 5.”

4.1 Data cleaning

Reply Data collection [6] is the initial step in weblog preprocessing.
After collecting the data, irrelevant records are removed in the
data cleaning process. Data cleaning [10] refers a process of
eliminating the noisy and irrelevant data which are disturbing
the process of mining the knowledge through weblogs.
Figure 4. Error Log
4.2 User and Session Identification
When cookies are used by the websites, the information will From the web access log, different user sessions can be
be in the cookies field of log file. Web traffic analysis identified by user as well as session identification. Session
software employs the cookies to track the repeat visitors. identification [7] is the process of dividing the individual user
access logs into sessions. To identify the various sessions, a
referrer based method is used.
3.DATA PREPROCESSING METHODS
The raw data may include noise, missing values, and 4 . 3 Path Completion
inconsistency mostly. The data mining results have been This is done in order to acquire the entire user access path.
affected by the data quality. So it must to preprocess the data The incomplete access path of every user session is
for the purpose of increasing the quality and efficiency. The recognized based on user session identification. In the start
process of preprocessing contains data preparation and of user session, the referrer has data values and delete this
transformation of the initial dataset. The preprocessing value of referrer by adding ‘_’.Also the web log preprocessing
methods are categorized such as data cleaning, data supports the unwanted click stream removal from log file and
integration, data transformation, data reduction and data to minimize by the original file size 40-50%.
discretization[4].
3.1Data cleaning
Data cleaning is an essential requirement of preprocessing
methodologies. It is done for duplicate tuples. It will remove

24
 International Journal of Computer Applications (0975 – 8887)
Volume 87 – No.3, February 2014

can be performed. The log file collected from different

Raw sources undergoes different preprocessing phases to make
Data cleaning actionable data source. It will help to automatic discovery of
usage
meaningful pattern and relationships from access stream of
data user. Swarm based web session clustering helps in many ways
to manage the web resources effectively such as web
personalization, schema modification, website modification
and web server performance. In this paper [2], they proposed
User/session a framework for web session clustering at preprocessing level
Usage of web usage mining. The framework will cover the data
identification
statistics preprocessing steps to prepare the weblog data and convert
the categorical weblog data into numerical data. A session
vector is obtained, so that appropriate similarity and swarm
optimization could be applied to cluster the weblog data. The
hierarchical cluster based approach will enhance the existing
Page view web session techniques for more structured information about
identification the user sessions. The paper [6] introduces an extensive
Site research framework which is capable of preprocessing web
structure log data completely and efficiently. The learning algorithm of
proposed research framework separates human user and
and search engine accesses intelligently, with less time. In order to
content Path create suitable target data, the further essential tasks of pre-
processing like data cleaning, user identification, session
completion
identification and path completion are designed collectively.
The framework reduces the error rate and improves significant
learning performance of the algorithm. The work ensures the
goodness of split by using popular measures like entropy and
gini index.

In UILP, data cleaning method is used to remove the noisy

Server and irrelevant information from the weblog. This is one of the
session features in identifying the user level of interest. The second
feature used is based on site topology and cookies. Frequency
file value, session identification, path completion are also
identified using this UILP algorithm [11]. In UILP
(i) During data cleaning process, explicit image and
multimedia requests from users are considered; those requests
are not removed from weblogs.
Episode Episode (ii) Users are identified based on site topology and cookies.
identification (iii) Session time is calculated based on the time spent on each
file website by a particular user.
(iv) Frequency value is calculated based on the number of
web pages visited by the user on particular website.

5. AN OVERVIEW OF EXISTING Here the site topology is used to identify the user and for
METHODOLOGIES completing the missing path .To label the session, the time
This research paper [2] studies and presents several data duration is calculated between two nearby website visited by
preparation techniques of access stream even before the the particular user. It is calculated each and every time when a
mining process can be started. These are used to improve the user switches from one website to another and the amount of
performance of the data preprocessing, to identify the unique time spent in each website.
sessions and unique users. The methods proposed will help to
discover meaningful pattern and relationships from the access 6. PROPOSED METHODOLOGY
stream of the user. These are proved to be valid and useful by The proposed method tells user behavior and it creates user
various research tests. Yang Bin et al. in [19] used negative cluster and site cluster. Also it gives the information like what
association rules in discovery of web visitor’s patterns. sites are the most and least popular, which website is most
Negative association rules have been deployed to solve the commonly used by visitors and from what search engine are
deficiencies in which positive rules are referred to. It is known visitors coming frequently. In this method, if IP address is
that the data preprocessing is an essential process for effective unique then similar user cluster is created; If IP address is
mining process. In paper [9], a novel pre-processing technique same and user name is not unique, agent log, operating system
is proposed by removing local and global noise and web and browser are different then distinguish user cluster is
robots. Anonymous microsoft web dataset and MSNBC.com created.
anonymous web dataset are used for estimating this
preprocessing technique.
The paper [1] describes the effective and complete
preprocessing of access stream before actual mining process

25
 International Journal of Computer Applications (0975 – 8887)
Volume 87 – No.3, February 2014

A. Steps followed By the proposed algorithm, web visitors are classified on the
1. Create similar user cluster and distinguish user cluster basis of user click history and similarity measure. The
based on IP address. processed dataset is given below.
2. Create site clusters based on frequently accessed sites.
3. If number of sites in current site cluster is greater Table 3. Processed Dataset
than previous site cluster then assign that is the most
popular site. Label Processed Value
4. Return the most popular site Total No. of users 5080
5. Otherwise assume that is the least popular site Similar user cluster 2279
6. Return the least popular site and repeat until all user &
site clusters are processed. Distinguish user cluster 2801
VOB algorithm
Input The following “Fig 6.” shows the creation of user cluster.
Web log files
Output
User Cluster Creation
User cluster, site cluster, most popular site, least popular
site. 6000
Algorithm 5000
If (IP address is unique) then 4000
Create similar_user_ cluster; 3000 Distinguish
Return similar_ user_ cluster; 2000 User Cluster
If (IP address is same and user name is not unique, agent log, 1000 Similar User
operating system and browsers are different ) then 0
Create distinguish_ user_ cluster. Cluster
Return distinguish_ user _cluster. Total No.of
For i =sitecluster_1 to sitecluster_n do
users
If (no. of. sites in current site cluster > previous site
cluster) then
Most Popular = current_ site_ cluster
return “most popular site”
else Figure 6. User cluster creation
Least Popular = current_ site_ cluster
The VOB algorithm identifies the users based on the data
return “least popular site”
collected from cookies. This algorithm takes all the users in
repeat until all user & site clusters are processed. count and their request for processing. By the result, the
In the proposed method VOB, clustering plays a key role to proposed VOB algorithm outperforms to classify the similar
classify web visitors on the basis of user click history and user cluster and distinguish user cluster.
similarity measure. This algorithm considers four entities
namely IP address, user name, website name, and frequency The total number of sites visited by the user is calculated as
of accessed sites. Cookies based weblogs are taken as the 12682. Among these sites, maximum number of visits has
input which mainly classify the unique users and helps to done for the educational websites. Totally it is of count 4700.
create user clusters. And the users have given next preference to the social
networking sites.
Here, the website and webpage navigation behavior are
considered as the basic source for tracing the visitors’ online The number of visits made to social networking sites is 3269.
behavior and also to identify the interest of the user in Also 3031 users have referred the research sites. Only from
accessing the various web sites. Based on the number of sites the month of APRIL and MAY, the 1230 users have used the
in the site clusters, it is concluded that it is the most popular electronic commerce websites.
website or the least one. Also the frequency is calculated by The number of visits for the case of entertainment is 452
taking the time difference and the total number of clicks on a which explicitly shows the minimum desirability of that kind
particular website given in a log file. Hence the VOB of sites. The given “Fig 7.” tells that site clusters are created
algorithm effectively traces the behavior of online users which based on frequently accessed sites.
supports the website usage analysis.
From the following “Fig 8” it is known that the maximum
7. EXPERIMENTAL SETUP AND weighttage has given to educational sites than other sites like
PERFORMANCE ANALYSIS entertainment, social, electronic commerce and research.
The weblog files are collected from college web server and The most popular website is identified based on the condition
browser machine for the period of 6 months from January that if no. of. sites in current site cluster is greater than
2013 to June 2013. For implementation, Java (jdk 1.6) is used previous site. Otherwise it was assumed that is the least
in the system which posses Intel core i3 processor with 4GB popular site This same procedure is repeated until all user and
RAM. site clusters have processed.
7.1 Performance evaluation “Fig 9.” shows that, the proposed algorithm proves it’s
The performance evaluation is done by analyzing the dataset efficiency for classifying the preference of users to various
taken. In the period of 6 months, the total no. of users is 5080. categories of websites.

26
 International Journal of Computer Applications (0975 – 8887)
Volume 87 – No.3, February 2014

Site Cluster Creation

5000 Identification of Most Popular and

4500 Least Popular Site
4000
3500 9.69 3.56
3000 25.77 Social
2500
2000 23.9
1500 37.06 Educational
1000
500 No.of visits Research
0
Ecommerce

Figure 9. Identification of Most Popular and Least

Figure 7. Site cluster creation Popular Site

Overall Performance of VOB Algorithm 9. FUTURE ENHANCEMENT

A number of further tasks could be added by demonstrating
5000 the utility of web mining. It can be done by making
4500 exploratory changes to web sites. The intelligent system web
usage preprocessor splits the human and search engine
4000 accesses before using the preprocessing techniques. This can
3500 be extended by using some other learning algorithms also[1].
3000 It can be further extended to user profiling and similar image
2500 retrieval by tracing the visitors’ on line behaviors for effective
Distinguish web usage mining[11]. Many preprocessing techniques can be
2000 user cluster effectively applied in web log mining[7]. The preprocessing
1500 of web log data for finding frequent patterns using weighted
1000 Similar user association rule mining technique can be extended to other
500 cluster industrial and social organizations too[6]. In recent days, the
web usage mining has great potential and frequently
0 employed for the tasks like web personalization, web pages
prefetching and website reorganization, etc[16]. So it is
required to know the users’ behavior when interaction is made
with the web.

10. REFERENCES
[1] V.V.R. Maheswara Rao and Dr. V. Valli Kumari, “An
Figure 8. Overall Performance of VOB algorithm Enhanced Pre-Processing Research Framework For Web
Log Data Using A Learning Algorithm”, Netcom
In this algorithm, user cluster and site cluster creation is 2010,Cscp 01, Pp. 01–15, 2011.
mainly considered as an important work and it helps to do
website usage analysis based on their website surfing [2] Mr. Sanjay Bapu Thakare and Prof. Sangram. Z. Gawali,
behavior. “A Effective and Complete Preprocessing for Web Usage
Mining”, (IJCSE) International Journal on Computer
8. CONCLUSION AND SUMMARY Science and Engineering Vol. 02, No. 03, 2010, 848-851.
Web usage mining has emerged as the essential tool for
[3] Hussain.T, Asghar.S and Masood.N, “Web usage
realizing more personalized user-friendly and business
mining: A survey on preprocessing of web log
optimal web services. The key is to use the user-click stream
file”,Information and emerging technologies,2010,
data for many mining purposes. Traditionally, web usage
ISBN: 978-1-4244-8001-2
mining is used by e-commerce sites to organize their sites and
to increase profits. The newly proposed algorithm is Visitors’ [4] Jiawai Han and Micheline Kamber,”Data mining-
Online Behavior (VOB) which identifies user behavior and Concepts and echniques”, secondedition, Elsevier,
creates user cluster, site cluster, most popular web site and the Reprint 2010.
least popular web site. It must to trace the visitors’ on-line
behaviors for website usage analysis. Actually it is an analysis [5] http://www.galeas.de/webmining.html.
to get knowledge about how visitors use website which could [6] M. Malarvizhi S. A. Sahaaya Arul Mary, “Preprocessing
provide guidelines to web site reorganization and helps to of Educational Institution Web Log Data for Finding
prevent disorientation. Frequent Patterns using Weighted Association Rule

27
 International Journal of Computer Applications (0975 – 8887)
Volume 87 – No.3, February 2014

Mining Technique”, European Journal of Scientific Conference on Tools with Artificial Intelligence,
Research ISSN 1450-216X Vol.74 No.4 ,617-633,2012. Newport Beach, CA., pp: 558-567. DOI:
10.1109/TAI.1997.632303
[7] Sheetal A. Raiyani and, Shailendra Jain, “Efficient
Preprocessing technique using Web log mining, [14] Srivatsava, J., R. Cooley, M. Deshpande and P.N. Tan,
International Journal of Advancements in Research & 2000. Web usage mining: discovery and applications of
Technology”, 1(6) ISSN 2278-7763, 2012. usage patterns from Web data. ACM SIGKDD Explorat.
Newsletter, 1: 12-23. DOI: 10.1145/846183.846188
[8] J.Srivatsava, R.Cooley, M.Deshpande, and P.N. Tan,
"Web Usage Mining: Discovery and Applications of [15] Agarwal, R. and R. Srikant, 1994. Fast algorithms for
Usage Patterns from Web Data." ACM SIGKDD mining association rules in large database. Proceeding of
Explorat. Newsletter, 2000. the 20th Conference on Very Large Data Bases, Sept. 12-
15, Morgan Kaufmann Publishers Inc., San Francisco,
[9] V.Chitraa, Dr.Antony Selvadoss Devamani, “A Novel CA. USA., pp: 487-499. DOI: 10.1234/12345678
Technique for Sessions Identification in Web Usage
Mining Preprocessing”, International Journal of [16] C.P. Sumathi et. al., Automatic Recommendation of
Computer Applications, Volume 34– No.9, 2012 Web Pages in Web Usage Mining, (IJCSE) International
Journal on Computer Science and Engineering, Vol. 02,
[10] Vijayashri Losarwar and Dr. Madhuri Joshi, Data No. 09, 2010, 3046-3052.
Preprocessing in Web Usage Mining, International
Conference on Artificial Intelligence and Embedded [17] Nanhay Singh1, Achin Jain1, Ram Shringar Raw,
Systems (ICAIES'2012) July 15-16, Singapore, 2012. Comparison Analysis Of Web Usage Mining Using
Pattern Recognition Techniques, International Journal of
[11] R. Suguna et.al,”User interest level based preprocessing Data Mining & Knowledge Management Process
algorithms using web usage mining”, International (IJDKP) Vol.3, No.4, July 2013
Journal on Computer Science and Engineering.
[18] L.K Joshila Grace, V. Maheswari, Dhinaharan
[12] Navin Kumar Tyagi1, A.K. Solanki2& Sanjay Tyagi3, Nagamalai, “Analysis of Weblogs and Web User in Web
An algorithmic approach to data preprocessing in web Mining,” International Journal of Network Security & Its
usage mining, International Journal of Information Applications (IJNSA), Vol. 3, No. 1, January 2011.
Technology and Knowledge Management July-
December 2010, Volume 2, No. 2, pp. 279-283 [19] Yang Bin, Dong Xianguin, Shi Fufu, “Research of Web
Usage Mining based on Negative Association Rules”
[13] Cooley, R., B. Mobasher and J. Srivatsava, 1997. Web International Forum on Computer Science-Technology
mining: Information and pattern discovery on the World and Applications, 2009.
Wide Web. Proceeding of the 9th IEEE International

IJCATM : www.ijcaonline.org 28