title: Scrapy CrawlSpider for AJAX content
body: <p>I am attempting to crawl a site for news articles. My start_url contains:</p>
  
  <p>(1) links to each article: <a href="http://example.com/symbol/TSLA" rel="noreferrer">http://example.com/symbol/TSLA</a></p>
  
  <p>and </p>
  
  <p>(2) a "More" button that makes an AJAX call that dynamically loads more articles within the same start_url: <a href="http://example.com/account/ajax_headlines_content?type=in_focus_articles&amp;page=0&amp;slugs=tsla&amp;is_symbol_page=true" rel="noreferrer">http://example.com/account/ajax_headlines_content?type=in_focus_articles&amp;page=0&amp;slugs=tsla&amp;is_symbol_page=true</a></p>
  
  <p>A parameter to the AJAX call is "page", which is incremented each time the "More" button is clicked. For example, clicking "More" once will load an additional n articles and update the page parameter in the "More" button onClick event, so that next time "More" is clicked, "page" two of articles will be loaded (assuming "page" 0 was loaded initially, and "page" 1 was loaded on the first click). </p>
  
  <p>For each "page" I would like to scrape the contents of each article using Rules, but I do not know how many "pages" there are and I do not want to choose some arbitrary m (e.g., 10k). I can't seem to figure out how to set this up. </p>
  
  <p>From this question, <a href="https://stackoverflow.com/questions/6566322/scrapy-crawl-urls-in-order">Scrapy Crawl URLs in Order</a>, I have tried to create a URL list of potential URLs, but I can't determine how and where to send a new URL from the pool after parsing the previous URL and ensuring it contains news links for a CrawlSpider. My Rules send responses to a parse_items callback, where the article contents are parsed. </p>
  
  <p>Is there a way to observe the contents of the links page (similar to the BaseSpider example) before applying Rules and calling parse_items so that I may know when to stop crawling? </p>
  
  <p>Simplified code (I removed several of the fields I'm parsing for clarity):</p>
  
  <pre><code>class ExampleSite(CrawlSpider):
  
      name = "so"
      download_delay = 2
  
      more_pages = True
      current_page = 0
  
      allowed_domains = ['example.com']
  
      start_urls = ['http://example.com/account/ajax_headlines_content?type=in_focus_articles&amp;page=0'+
                        '&amp;slugs=tsla&amp;is_symbol_page=true']
  
      ##could also use
      ##start_urls = ['http://example.com/symbol/tsla']
  
      ajax_urls = []                                                                                                                                                                                                                                                                                                                                                                                                                          
      for i in range(1,1000):
          ajax_urls.append('http://example.com/account/ajax_headlines_content?type=in_focus_articles&amp;page='+str(i)+
                        '&amp;slugs=tsla&amp;is_symbol_page=true')
  
      rules = (
               Rule(SgmlLinkExtractor(allow=('/symbol/tsla', ))),
               Rule(SgmlLinkExtractor(allow=('/news-article.*tesla.*', '/article.*tesla.*', )), callback='parse_item')
              )
  
          ##need something like this??
          ##override parse?
          ## if response.body == 'no results':
              ## self.more_pages = False
              ## ##stop crawler??   
          ## else: 
              ## self.current_page = self.current_page + 1
              ## yield Request(self.ajax_urls[self.current_page], callback=self.parse_start_url)
  
  
      def parse_item(self, response):
  
          self.log("Scraping: %s" % response.url, level=log.INFO)
  
          hxs = Selector(response)
  
          item = NewsItem()
  
          item['url'] = response.url
          item['source'] = 'example'
          item['title'] = hxs.xpath('//title/text()')
          item['date'] = hxs.xpath('//div[@class="article_info_pos"]/span/text()')
  
          yield item
  </code></pre>