title: How are crawl delays from robots.txt interpreted?
body: <p>I am building a python based webscraper that scrapes price and specification data for products from multiple sites.  I want to be respectful and follow the robots.txt as much as I can. </p>
  
  <p>Lets say the crawl delay defined in the robots.txt file is 10 seconds.  How is this interpreted? I built my scraper to go to each product category page and then take the list of all products from each category and then go into each individual product page and scrape price and specifications.  </p>
  
  <p>Does each page request need to be delayed by 10 seconds? Or is the act of running my script once considered one action and I just need to wait 10s each time I run it?  </p>
  
  <p>If it is the former then how does anyone scrape large amounts of data from a site? If there are 5000 product pages and I delay each one by 10 seconds then my script will take 14hrs for a single run.</p>
  
  <p>What if I split the work between multiple scripts? Does each separate script need to follow the rule by itself or do all requests from a certain IP need to follow the rule collectively?</p>
  
  <p>I don't want to get my IP banned or accidentally take down anyone's site. Thanks in advance for any answers.</p>