title: Is it important to set some sort of timer to avoid detection when web scraping?
body: <p>I've set up a program in an attempt to do some scraping of a real estate site in order to get some statistical data about the market.</p>
  
  <p>My program will probably call the web site about 150 times. And I want to do this once per day. I imagine the web is big enough, they will possibly get about 10,000 - 20,000 hits per day (estimate).</p>
  
  <p>But if I sent these all together, would they think that they're putting flooded with requests? would they notice that I'm web scraping and block my IP?</p>
  
  <p>If so, is it important to set a timer in? At the moment, I've put a timer that waits 3 to 5 seconds before each call, I'm just calling if that's necessary.</p>