title: Scrapy Error : HTTP status code is not handled or not allowed
body: <p>i have a problem when run the spider . when i crawl it , it's shown eror like this : "HTTP status code is not handled".</p>
  
  <pre><code>2018-08-27 14:30:58 [scrapy.core.engine] DEBUG: Crawled (404) &lt;GET http://id.priceprice.com/harga-hp/%3Ca%20href=%22/harga-hp/?maker=OPPO-182%22%3EOPPO%3C/a%3E&gt; (referer: http://id.priceprice.com/harga-hp/)
  2018-08-27 14:30:58 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;404 http://id.priceprice.com/harga-hp/%3Ca%20href=%22/harga-hp/?maker=OPPO-182%22%3EOPPO%3C/a%3E&gt;: HTTP status code is not handled or not allowed
  2018-08-27 14:30:58 [scrapy.core.engine] DEBUG: Crawled (404) &lt;GET http://id.priceprice.com/harga-hp/%3Ca%20href=%22/harga-hp/?maker=Smartfren-178%22%3ESmartfren%3C/a%3E&gt; (referer: http://id.priceprice.com/harga-hp/)
  2018-08-27 14:30:58 [scrapy.core.engine] DEBUG: Crawled (404) &lt;GET http://id.priceprice.com/harga-hp/%3Ca%20href=%22/harga-hp/?maker=Evercoss-184%22%3EEvercoss%20Evercoss%3C/a%3E&gt; (referer: http://id.priceprice.com/harga-hp/)
  2018-08-27 14:30:58 [scrapy.core.engine] DEBUG: Crawled (404) &lt;GET http://id.priceprice.com/harga-hp/%3Ca%20href=%22/harga-hp/?maker=Nokia-109%22%3ENokia%3C/a%3E&gt; (referer: http://id.priceprice.com/harga-hp/)
  2018-08-27 14:30:58 [scrapy.core.engine] DEBUG: Crawled (404) &lt;GET http://id.priceprice.com/harga-hp/%3Ca%20href=%22/harga-hp/?maker=HUAWEI-69%22%3EHUAWEI%3C/a%3E&gt; (referer: http://id.priceprice.com/harga-hp/)
  2018-08-27 14:30:58 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;404 http://id.priceprice.com/harga-hp/%3Ca%20href=%22/harga-hp/?maker=Smartfren-178%22%3ESmartfren%3C/a%3E&gt;: HTTP status code is not handled or not allowed
  2018-08-27 14:30:58 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;404 http://id.priceprice.com/harga-hp/%3Ca%20href=%22/harga-hp/?maker=Evercoss-184%22%3EEvercoss%20Evercoss%3C/a%3E&gt;: HTTP status code is not handled or not allowed
  </code></pre>
  
  <p>i have follow another instruction for edit setting.py and add code : </p>
  
  <pre><code>user_agent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1"
  </code></pre>
  
  <p>but its still not working .</p>
  
  <p>this is my code : </p>
  
  <pre><code>import scrapy
  from handset.items import HandsetItem
  from scrapy.linkextractors import LinkExtractor
  from scrapy.spiders import Rule, CrawlSpider
  
  
  class HandsetpriceSpider(scrapy.Spider):
      name = 'price'
      allowed_domains = ['id.priceprice.com']
      start_urls = ['http://id.priceprice.com/harga-hp/']
  
      def parse(self, response):
  
          rules = (
                  Rule(LinkExtractor(allow='div.listCont:nth-child(2) &gt; ul:nth-child(1)'), callback='parse_details'),
                  Rule(LinkExtractor(restrict_css='ul &gt; li &gt; a[href*="maker"]'), follow =True)                
                 )
          for url in  response.xpath('//ul[1]//li/a[contains(@href, "maker")]').extract() :
              url = response.urljoin(url)
              yield scrapy.Request(url, callback = self.parse_details)
  
          next_page_url = response.css('li.last &gt; a::attr(href)').extract_first()
          if next_page_url:
             next_page_url = response.urljoin(next_page_url)
             yield scrapy.Request(url=next_page_url, callback=self.parse)
  
      def parse_details(self, response):
          yield {
         'Name' : response.css('div.itmName h3:nth-child(1) &gt; a:nth-child(1) ::text').extract_first(),
         'Price' : response.css('div.itmPrice &gt; a.price ::text').extract_first(),
          }
  </code></pre>