title: Scrapy crawler extract urls but miss half callbacks
body: <p>I'm facing a strange issue while trying to scrape this <a href="http://www.ikea.com/fr/fr/catalog/productsaz/8/" rel="nofollow">URL</a>:</p>
  
  <p>To perform the crawling, I designed this:</p>
  
  <pre><code>class IkeaSpider(CrawlSpider) :
  
      name = "Ikea"
      allower_domains = ["http://www.ikea.com/"]
      start_urls = ["http://www.ikea.com/fr/fr/catalog/productsaz/8/"]
  
      rules = (
          Rule(SgmlLinkExtractor(allow=[r'.*/catalog/products/\d+']),
              callback='parse_page',
              follow=True),
              )
  
      logging.basicConfig(filename='example.log',level=logging.ERROR)
  
          def parse_page(self, response):
  
              for sel in response.xpath('//div[@class="rightContent"]'):
  
                   Blah blah blah
  </code></pre>
  
  <p>I launch the spider from the command-line, and I can see urls normally scraped, but, for some of them, the callback doesn't work (about half of them are normally scrapped). </p>
  
  <p>As there is more than 150 links on this page, it may explain why the crawler is missing callbacks (too many jobs). Does some of you have any idea regarding this?</p>
  
  <p>This is the log :</p>
  
  <p>2015-12-25 09:02:55 [scrapy] INFO: Stored csv feed (107 items) in: test.csv
  2015-12-25 09:02:55 [scrapy] INFO: Dumping Scrapy stats:
   'downloader/request_bytes': 68554,
   'downloader/request_count': 217,
   'downloader/request_method_count/GET': 217,
   'downloader/response_bytes': 4577452,
   'downloader/response_count': 217,
   'downloader/response_status_count/200': 216,
   'downloader/response_status_count/404': 1,
   'dupefilter/filtered': 107,
   'file_count': 106,
   'file_status_count/downloaded': 106,
   'finish_reason': 'finished',
   'finish_time': datetime.datetime(2015, 12, 25, 8, 2, 55, 548350),
   'item_scraped_count': 107,
   'log_count/DEBUG': 433,
   'log_count/ERROR': 2,
   'log_count/INFO': 8,
   'log_count/WARNING': 1,
   'request_depth_max': 2,
   'response_received_count': 217,
   'scheduler/dequeued': 110,
   'scheduler/dequeued/memory': 110,
   'scheduler/enqueued': 110,
   'scheduler/enqueued/memory': 110,
   'start_time': datetime.datetime(2015, 12, 25, 8, 2, 28, 656959)
  2015-12-25 09:02:55 [scrapy] INFO: Spider closed (finished</p>