title: Avoid getting banned on sites using scrapy
body: <p>I'm trying to download data from gsmarena. A sample code to download HTC one me spec is from the following site "<a href="http://www.gsmarena.com/htc_one_me-7275.php" rel="nofollow">http://www.gsmarena.com/htc_one_me-7275.php</a>" as mentioned below:</p>
  
  <p>The data on the website is classified in form of tables and table rows. 
  The data is of the format:</p>
  
  <pre><code>table header &gt; td[@class='ttl'] &gt; td[@class='nfo']
  </code></pre>
  
  <p>Items.py file:</p>
  
  <pre><code>import scrapy
  
  class gsmArenaDataItem(scrapy.Item):
      phoneName = scrapy.Field()
      phoneDetails = scrapy.Field()
      pass
  </code></pre>
  
  <p>Spider file:</p>
  
  <pre><code>from scrapy.selector import Selector
  from scrapy import Spider
  from gsmarena_data.items import gsmArenaDataItem
  
  class testSpider(Spider):
      name = "mobile_test"
      allowed_domains = ["gsmarena.com"]
      start_urls = ('http://www.gsmarena.com/htc_one_me-7275.php',)
  
      def parse(self, response):
          # extract whatever stuffs you want and yield items here
          hxs = Selector(response)
          phone = gsmArenaDataItem()
          tableRows = hxs.css("div#specs-list table")
          for tableRows in tableRows:
              phone['phoneName'] = tableRows.xpath(".//th/text()").extract()[0]
              for ttl in tableRows.xpath(".//td[@class='ttl']"):
                  ttl_value = " ".join(ttl.xpath(".//text()").extract())
                  nfo_value = " ".join(ttl.xpath("following-sibling::td[@class='nfo']//text()").extract())
                  colonSign = ": "
                  commaSign = ", "
                  seq = [ttl_value, colonSign, nfo_value, commaSign]
                  seq = seq.join(seq)
          phone['phoneDetails'] = seq
          yield phone
  </code></pre>
  
  <p>However, I'm getting banned as soon  as I try to even load the page in scrapy shell using:</p>
  
  <pre><code>"http://www.gsmarena.com/htc_one_me-7275.php"
  </code></pre>
  
  <p>I've even tried using DOWNLOAD_DELAY = 3 in settings.py.</p>
  
  <p>Kindly suggest how should I go about it.</p>