title: Why LinkExtractor does not catch the links that was generated after AJAX requests?
body: <p>I'm crawling a page that generates data with infinite scrolling.
  I'm using CrawlSpider, and the <code>rules</code> are defined like this:</p>
  
  <pre><code>rules = (
      Rule(LinkExtractor(restrict_xpaths = ('//*some/xpaths')), callback = 'parse_first_itmes', follow = True),
      Rule(LinkExtractor(restrict_xpaths = ('//*some/other/xpaths')), callback = 'parse_second_itmes'),
  )
  </code></pre>
  
  <p>In the <code>parse_item</code> function, I have a <code>Request</code> makes the <code>AJAX</code> requests:</p>
  
  <pre><code>def parse_first_items(self, response):
      l = ItemLoader(item = AmazonCnCustomerItem(), response = response)
  
      l.add_xpath('field1', '//p[@class="field1")]/text()')
      l.add_xpath('field2', '//p[@class="field2")]/text()')
  
      r_url = l.get_xpath('//*/url/xpath/@href')
  
      r = Request(url = req_url,
                      headers = {"Referer": "the/same/page/url",
                          "X-Requested-With": "XMLHttpRequest"},
                      callback = self.parse_first_items)
  
      return r, l.load_item()
  </code></pre>
  
  <p>I get the desired data just fine, but the <code>LinkExtractor</code> in the <code>Second Rule</code> does not catch the <code>urls</code> from the data generated by the <code>Request</code> inside the the <code>parse_first_itmes</code> function.</p>
  
  <p>How can I make the <code>LinkExtractor</code> in the <code>Second Rule</code> extract those links and use them to parse the <code>parse_second_itmes</code> function?</p>