An Architectural Framework of a Personalized Web Crawler based on User Interests

Abstract

The World Wide Web (WWW) is overwhelmed with information which can not be assimilated by the normal users without the use of search tools. The traditional search returns thousands of results for a single search query making the search and surfing experience cumbersome. This drawback has triggered the need for implementing personalized search tools. In this paper, a novel architecture is proposed to gather pages that are relevant to a particular user or group of users. The system consists of three modules: input, crawling and feedback. The input module is integrated with topic suggestion component extracting search query terms and representative documents from different sources. The crawling module is realized with intelligent multi-agent system for prioritizing the download of appropriate URLs. The relevance of the documents is computed based on interests of the users. While rendering the results, the user gives feedback and the system is compared to different crawler implementations. The empirical results clearly suggest the advantage of using topic suggestion component and computation of personalized relevance score in terms of harvest ratio and coverage.