title: price comparison service, how to effectively use a sitemap file?
body: <p>Many online shops provide a sitemap file which contains their product information in the form of:</p>
  
  <pre><code>...
      &lt;url&gt;
            &lt;loc&gt;http://blabla.com/tbcart/pc/-DOOR-GYM-Full-Body-Exerciser-256p34168.htm&lt;/loc&gt;
            &lt;lastmod&gt;2010-11-26&lt;/lastmod&gt;
            &lt;changefreq&gt;weekly&lt;/changefreq&gt;
      &lt;/url&gt;
  ...
  </code></pre>
  
  <p>But for an online price comparison service to work, it needs the actual product prices in addition to their URL. Assuming that a typical sitemap for an online shop contains 20'000 URLs, how would you proceed in getting the actual prices for each product ? Is this how the sitemap should be used for getting product prices ? </p>
  
  <p>It is highly likely that, performing 20'000 Http Get requests would cause the online shop to block the IP of the crawler :)</p>
  
  <p>Thanks,</p>
  
  <p>PS - How would this scale ? Like a sitemap with 50'000 links, Let's say one needs to reindex every Sunday, this implies sending 1 request every 2 seconds during the whole day, How can one avoid getting blocked in this situation ? </p>