title: Scraping breadcrumb links recursively with scrapy
body: <p>I am a newbie in python and scrapy. For a university project, I need to scrape car classifieds on the autoscout24.de domain (in german).</p>
  
  <p>Here is an example site: <a href="http://ww3.autoscout24.de/classified/268487991?asrc=fa|as&amp;testvariant=articles_t20&amp;tierlayer=fa" rel="nofollow">http://ww3.autoscout24.de/classified/268487991?asrc=fa|as&amp;testvariant=articles_t20&amp;tierlayer=fa</a></p>
  
  <p>I want the CrawlSpider to scrape the 'next offer' ('nächstes Angebot') recursively, but could not seem to extract the link from the breadcrumb. The interesting part of the site should be:</p>
  
  <pre><code>&lt;div class="floatRight"&gt;
                      &lt;span class="linkInvisible"&gt;
                          &lt;span class="floatLeft"&gt;‹&lt;/span&gt;
                          &lt;span class="floatLeft marginLeftXS"&gt;
                              vorheriges Angebot
                          &lt;/span&gt;
                      &lt;/span&gt;
  
  
                  &lt;p class="floatLeft marginLeftXS"&gt;|&lt;/p&gt;   
  
  
                      &lt;a id="breadCrumbNextOffer" ng-click="breadCrumb.nextOffer('http://ww3.autoscout24.de/classified/266709231')"&gt;
                          &lt;span class="floatLeft marginLeftXS"&gt;
                              nächstes Angebot
                          &lt;/span&gt;
                          &lt;p class="floatLeft marginLeftXS"&gt;›&lt;/p&gt;
                      &lt;/a&gt;
  
              &lt;/div&gt;
  </code></pre>
  
  <p>I would like to get the link inside breadCrumb.nextOffer(...)</p>
  
  <p>I have tried this:</p>
  
  <pre><code>import scrapy
  from scrapy.contrib.linkextractors import LinkExtractor
  from scrapy.contrib.spiders import CrawlSpider, Rule
  
  from autocrawler.items import AutocrawlerItem
  
  
  class AutocrawlerRecursiveSpider(CrawlSpider):
      name = 'autocrawler_recursive'
      allowed_domains = ['autoscout24.de']
      start_urls = ['http://ww3.autoscout24.de/classified/268348256asrc=pl|as&amp;testvariant=articles_t20&amp;tierlayer=pl']
  
  rules = (
      Rule(LinkExtractor(allow=(r'/^http:\/\/ww3.autoscout24.de\/classified\/\w+$/'), restrict_xpaths=('//*[@id="ng-app"]/div[2]/div/div[1]/div')),  callback='parse_item', follow=True),
  )
  
  def parse_item(self, response):
      item = AutocrawlerItem()
      item['brand'] = response.xpath('//*[@id="detailsContent"]/div[1]/h1/text()').extract()
      item['price'] = response.xpath('//*[@id="detailsContent"]/div[2]/div/div[2]/div/div[1]/text()').extract()
      return item
  </code></pre>
  
  <p>But this would only scrape the start_url and can't seem to find the next links.</p>
  
  <p>I would be very thankful of any input. :)</p>