a focussed web crawler learns to recognize web pages that are relevant to the interest of one or more users , from a set of examples provided by the users . it then explores the web starting from the example set , using the statistics collected from the examples and other analysis on the link graph of the growing crawl database , to guide itself towards relevant , valuable resources and away from irrelevant and / or low quality material on the web . thereby , the web crawler builds a comprehensive topic - specific library for the benefit of specific users .