title: Slow down spidering of website
body: <p>Is there a way to force a spider to slow down its spidering of a website?  Anything that can be put in headers or robots.txt?</p>
  
  <p>I thought i remembered reading something about this being possible but cannot find anything now.</p>