title: Scrapy callback function, how to parse several pages?
body: <p>I want to make a crawler that starts at a url (page1), and follows a link to a new page, page2. On page2 it should follow a link to page3. Then I want to scrape some data on page3.</p>
  
  <p>However, I'm a noob at scraping and can't get the callback function to work. Here's my code:</p>
  
  <pre><code>class allabolagnewspider(CrawlSpider):
  name="allabolagnewspider"
  # allowed_domains = ["byralistan.se"]
  start_urls = [
      "http://www.allabolag.se/5565794400/befattningar"
  ]
  
  rules = (
      Rule(LinkExtractor(allow = "http://www.allabolag.se",
                         restrict_xpaths=('//*[@id="printContent"]//a[1]'),
                         canonicalize=False),
           callback='parse_link1'),
  )
  
  def parse_link1(self, response):
      hxs = HtmlXPathSelector(response)
      return Request(hxs.xpath('//*[@id="printContent"]/div[2]/table/tbody/tr[4]/td/table/tbody/tr/td[2]/a').extract(), callback=self.parse_link2)
  
  def parse_link2(self, response):
      for sel in response.xpath('//*[@id="printContent"]'):
          item = AllabolagnewItem()
          item['Byra'] = sel.xpath('/div[2]/table/tbody/tr[3]/td/h1').extract()
          item['Namn'] = sel.xpath('/div[2]/table/tbody/tr[3]/td/h1').extract()
          item['Gender'] = sel.xpath('/div[2]/table/tbody/tr[3]/td/h1').extract()
          item['Alder'] = sel.xpath('/div[2]/table/tbody/tr[3]/td/h1').extract()
          yield item
  </code></pre>
  
  <p>However, when I run it I get the following error message: 
  "TypeError: Request url must be str or unicode, got list:"</p>
  
  <p>If I got it correctly I mess up when I try to return my request for parse_link1. What should I do?</p>
  
  <p>Edit:</p>
  
  <p>Here's the working code (still got a few issues though but the specific problem was solved):</p>
  
  <pre><code>class allabolagnewspider(CrawlSpider):
  name="allabolagnewspider"
  # allowed_domains = ["byralistan.se"]
  start_urls = [
      "http://www.allabolag.se/5565794400/befattningar"
  ]
  
  rules = (
      Rule(LinkExtractor(allow = "http://www.allabolag.se",
                         restrict_xpaths=('//*[@id="printContent"]//a[2]'),
                         canonicalize=False),
           callback='parse_link1'),
  )
  
  def parse_link1(self, response):
      for href in response.xpath('''//*[@id="printContent"]/div[2]/table//tr[4]/td/table//tr/td[2]/a/@href''').extract():
          print "hey"
          yield Request(response.urljoin(href), callback=self.parse_link2)
  
  def parse_link2(self, response):
      for sel in response.xpath('//*[@id="printContent"]'):
          print "hey2"
          item = AllabolagnewItem()
          item['Byra'] = sel.xpath('./div[2]/table//tr[3]/td/h1/text()').extract()
          item['Namn'] = sel.xpath('./div[2]/table//tr[3]/td/h1/text()').extract()
          item['Gender'] = sel.xpath('./div[2]/table//tr[7]/td/table[1]//tr[1]/td/text()').extract()
          item['Alder'] = sel.xpath('./div[2]/table//tr[3]/td/h1/text()').extract()
          yield item
  </code></pre>