title: Is it possible to control the crawl speed by robots.txt?
body: <p>We can tell bots to crawl or not to crawl our website in robot.txt. On the other hand, we can control the crawling speed in Google Webmasters (how much Google bot crawls the website). I wonder if it is possible to limit the crawler activities by robots.txt</p>
  
  <p>I mean accepting bots to crawl pages but limit their presence by time or pages or size!</p>