title: Scrapy - Handling a page which loads with 404 status code
body: <p><a href="https://www.classifiedads.com/search.php?keywords=&amp;cid=468&amp;lid=rx10&amp;lname=India&amp;from=s&amp;page=1" rel="nofollow noreferrer">This</a> is the website I'm scraping. The ads in the pages are loading without any problem, but they are always loading with 404 status code, so scrapy doesn't yield items from those links. </p>
  
  <p>If I send a request from shell to an ad, it retries 10 times and a valid response is returned. However, when I run the script with <code>scrapy crawl myspider</code> command, the ads don't return valid responses, scrapy tries to send request single time.</p>
  
  <p>This is the error code from random items.</p>
  
  <pre><code>2019-07-30 15:33:51 [scrapy] DEBUG: Retrying &lt;GET https://www.classifiedads.com/homes_for_sale/57c10snzt1wzz&gt; (failed 1 times): 404 Not Found
  2019-07-30 15:33:51 [scrapy] DEBUG: Retrying &lt;GET https://www.classifiedads.com/homes_for_sale/49zbgqvx21wzz&gt; (failed 1 times): 404 Not Found
  2019-07-30 15:33:51 [scrapy] DEBUG: Retrying &lt;GET https://www.classifiedads.com/homes_for_sale/49482b3hq1wzz&gt; (failed 1 times): 404 Not Found
  </code></pre>
  
  <p>This is my spiders code. How can I handle this problem?</p>
  
  <pre><code>class MySpider(CrawlSpider):
  
      name = 'myspider'
  
      start_urls = [
          'https://www.classifiedads.com/search.php?keywords=&amp;cid=468&amp;lid=rx10&amp;lname=India&amp;from=s&amp;page=1',
          'https://www.classifiedads.com/search.php?keywords=&amp;cid=18&amp;lid=rx10&amp;lname=India&amp;page=1'
      ]
  
      rules = (
          Rule(LinkExtractor(allow=(r'https://www.classifiedads.com/search.php\?keywords=&amp;cid=468&amp;lid=rx10&amp;lname=India&amp;from=s&amp;page=\d+',)), callback='parse_page', follow=True),
          Rule(LinkExtractor(allow=(r'https://www.classifiedads.com/search.php\?keywords=&amp;cid=18&amp;lid=rx10&amp;lname=India&amp;page=\d+',)), callback='parse_page', follow=True)
      )
  
      headers = {
          'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36',
          'upgrade-insecure-requests': 1,
  
      }
  
      def parse_page(self, response):
          items = response.css('div#results div.resultitem div a::attr(href)').getall()
  
          if items:
              for item in items:
                  if item.startswith('//www.classifiedads.com/'):
                      yield scrapy.Request(
                          url='https:{}'.format(item),
                          method='GET',
                          headers=self.headers,
                          callback=self.parse_items
                      )
  
      def parse_items(self, response):
          # scraping the items
  </code></pre>