title: Scrapy and respect of robots.txt
body: <p>I discovered yesterday that Scrapy respects the robots.txt file by default (<code>ROBOTSTXT_OBEY = True</code>). </p>
  
  <p>If I request an URL with <code>scrapy shell url</code>, and if I have a response, does it mean that <code>url</code> is not protected by robots.txt?</p>