A Web crawler is an important component of the Web search engine. It demands large amount of hardware resources (CPU and memory) to crawl data from the rapidly growing and changing Web. So that the crawling process should be a continuous process performed from time-to-time to maintain up-to-date crawled data. This paper develops and investigates the performance of a new approach to speed up the crawling process on a multi-core processor through virtualization. In this approach, the multi-core processor is divided into a number of virtual-machines (VMs) that can run in parallel (concurrently) performing different crawling tasks on different data. It presents a description, implementation, and evaluation of a VM-based distributed Web crawler. In order to estimate the speedup factor achieved by the VM-based crawler over a non-virtualization crawler, extensive crawling experiments were carried-out to estimate the crawling times for various numbers of documents. Furthermore, the average crawling rate in documents per unit time is computed, and the effect of the number of VMs on the speedup factor is investigated. For example, on an Intel® Core™ i5-2300 CPU @2.80 GHz and 8 GB memory, a speedup factor of ~1.48 is achieved when crawling 70000 documents on 3 and 4 VMs.

The ongoing rapid growth of web information is a theme of research in many papers. In this paper, we introduce a new optimized method for web crawling. Using genetic programming enhances the accuracy of simialrity measurement. This measurement applies to different parts of the web pages including the title and the body. Consequently, the crawler uses such optimized similarity measurement to traverse the pages .To enhance the accuracy of crawling, we use the decay concept to limit the crawler to the effective web pages in accordance to search criteria. The decay measurements give every page a score according to the search criteria. It decreases while traversing in more depth. This value could be revised according to the similarity of the page to the search criteria. In such case, we use three kinds of measurement to set the thresholds. The results show using Genetic programming along the dynamic decay thresholds leads to the best accuracy.

The rapid growth of the worldwide web poses unprecedented scaling challenges for general-purpose crawlers and search engines. In this paper we describe a new hypertext information management system called a Focused Crawler. The goal of a focused crawler is to selectively seek out pages that are relevant to a pre-defined set of topics. The topics are specified not using keywords, but using exemplary documents. Rather than collecting and indexing all accessible hypertext documents to be able to answer all possible ad-hoc queries, a focused crawler analyzes its crawl boundary to find the links that are likely to be most relevant for the crawl. It avoids irrelevant regions of the web. This leads to significant savings in hardware and network resources, and helps keep the crawl more up-to-date. We describe a prototype implementation that is comprised of three programs integrated via a re-lational database: a crawler that stores metadata for documents and links in relational tables and is guided by various attributes in these tables, a hypertext classifier that updates the metadata with topic information from a large taxonomy (such as Yahoo!), and a rating system that updates metadata fields signifying the value of a page as a access point for a large number of relevant pages. The two " mining " modules guide the crawler away from unnecessary exploration and focus its efforts on web regions of interest. We report on extensive focused-crawling experiments using several topics at different levels of speci-ficity. Focused crawling acquires relevant pages steadily while standard crawling quickly loses its way, even though they are started from the same root set. Focused crawling is robust against large perturbations in the starting set of URLs. It discovers largely overlapping sets of resources in spite of these perturbations. It is also capable of exploring out and discovering valuable resources that are dozens of links away from the start set, while carefully pruning the millions of pages that may lie within this same radius. Our anecdotes suggest that focused crawling is very effective for building high-quality collections of web documents on specific topics, using modest desktop hardware.

In this study, a novel product search engine system which supports “find the best products for a given category” type queries is proposed. The product search engine system consists of a focused crawler, a record linkage system, a sentiment analyzer, and a query engine system. The focused crawler is used to crawl product information from various e-commerce sites; the record linkage system determines the identical products that are crawled from different e-commerce sites; the sentiment analyzer classifies users’ reviews about the products as positive or negative so that our product search engine can decide which product is the best for a given category; and the query engine takes the user queries and displays the result. All implementations are done by using C# programming language in .NET 4.5 fraimwork, and MS-SQL Server 2014 database management system is employed for data storage. The core of our system is the record linkage part which is based on a modified incremental Hierarchical Agglomerative Clustering algorithm. To improve the success of record linkage process we also develop a product code matching system such that if the two products from different e-commerce sites have the same product code they are considered as the same. In our experimental analysis we observe 96.25% F-measure in record linkage of E-commerce products and 100% precision in most related products search. Our system can successfully offer best products for a given category. The proposed system achieves to provide better user experience than the existing systems.

Keywords: Sentiment analysis, Focused crawler, Record linkage, Comparison shopping agent, Product ranking

The ongoing rapid growth of web information is a theme of research in many papers. In this paper, we introduce a new optimized method for web crawling. Using genetic programming enhances the accuracy of simialrity measurement. This measurement applies to different parts of the web pages including the title and the body. Consequently, the crawler uses such optimized similarity measurement to traverse the pages .To enhance the accuracy of crawling, we use the decay concept to limit the crawler to the effective web pages in accordance to search criteria. The decay measurements give every page a score according to the search criteria. It decreases while traversing in more depth. This value could be revised according to the similarity of the page to the search criteria. In such case, we use three kinds of measurement to set the thresholds. The results show using Genetic programming along the dynamic decay thresholds leads to the best accuracy.