Your website may be invaded by robots at any time. If you’re lucky that is – at least if you want people to visit you from places like Google or Yahoo or Bing. And, if the visiting robots are polite.
In the early days
Your website may be invaded by robots at any time. If you’re lucky that is – at least if you want people to visit you from places like Google or Yahoo or Bing. And, if the visiting robots are polite.
In the early days of the Web, automated programs known as robots, or bots, were created to find information on the Web, and to create indexes of that information. They would do this regardless of whether you wanted them to visit your pages or not, and you had no way to tell them not to go through your web site.
If you search through Usenet message boards from the early days of the Web, you might come across a document such as the World Wide Web Frequently Asked Questions (FAQ), Part 1/2 (December, 1994), which describes robots in those days:
4.10: Hey, I know, I’ll write a WWW-exploring robot! Why not?
Programs that automatically traverse the web can be quite useful, but have the potential to make a serious mess of things. Robots have been written which do a “breadth-first” search of the web, exploring many sites in a gradual fashion instead of aggressively “rooting out” the pages of one site at a time. Some of these robots now produce excellent indexes of information available on the web.
But others have written simple depth-first searches which, at the worst, can bring servers to their knees in minutes by recursively downloading information from CGI script-based pages that contain an infinite number of possible links. (Often robots can’t realize this!) Imagine what happens when a robot decides to “index” the CONTENTS of several hundred mpeg movies. Shudder.
A Google patent granted today describes how it might schedule the crawling of web sites so that it doesn’t bring servers to their knees.
It tells us that Google might use crawlers that are focused upon finding different kinds of content, such as a robot that looks specifically for images, another for news, one for shopping or sports, and yet another that is more general and will try to discover new URLs for pages, and new content on already discovered pages. These different types of robots might all decide to visit the same web site, or the same host server which might be home to more than one site. If too many robots try to visit the same server, they might use up all of the resources of the site being visited, and keep other people from seeing the pages on that host server.
This kind of fragmentation of web crawling with different robots organized by type of information sought was intended to put less stress, and less of a load on web pages being crawled. But segmenting crawling by type created new problems in managing which crawlers visited different servers, and when.
The patent was originally filed back in 2003, and chances are very good that Google’s crawling of web sites has evolved in the years since this document was created. But the basic concepts about using multiple crawlers, scheduling the crawling of pages, prioritizing different kinds of pages to be crawled based upon metrics like PageRank, might still be very similar almost 7 years later.
Search Engine Push Towards Page Speed
When someone visits your pages, if they have to wait while the content of the page appears, it’s possible that they may move on to another site. If they find a link to your page in Google’s (or Yahoo’s or Bing’s) search results, and click upon it, and nothing happens for a couple of seconds because your page is slow, they might click on another search result and visit another site.
I’ve written in the past on the topic of Does Page Load Time influence SEO?, pa