title: Scrapy, Parse items data from page then follow link to get additional items data
body: <p>I have a problem to scrape additional fields which are on other pages after scraped data from first page e.g:</p>
  
  <p>Here is my code:</p>
  
  <pre><code>from scrapy.selector import HtmlXPathSelector
  from scrapy.http import HtmlResponse
  from IMDB_Frompage.items import ImdbFrompageItem
  from scrapy.http import Request
  from scrapy.contrib.spiders import CrawlSpider, Rule
  from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
  
  URL = "http://www.imdb.com/search/title?count=100&amp;ref_=nv_ch_mm_1&amp;start=1&amp;title_type=feature,tv_series,tv_movie"
  
  class MySpider(CrawlSpider):
      name = "imdb"
      allowed_domains = ["imdb.com"]
      start_urls = [URL]
      DOWNLOAD_DELAY = 0.5
  
      rules = (Rule(SgmlLinkExtractor(allow=('100&amp;ref'), restrict_xpaths=('//span[@class="pagination"]/a[contains(text(),"Next")]')), callback='parse_page', follow=True),)
  
      def parse_page(self, response):
          hxs = HtmlXPathSelector(response)
          item = ImdbFrompageItem()
          links = hxs.select("//td[@class='title']")
          items=[]
          for link in links:
              item = ImdbFrompageItem()
              item['link'] = link.select("a/@href").extract()[0]
              item['new_link'] ='http://www.imdb.com'+item['link']
              new_links = ''.join(item['new_link'])
              request = Request(new_links, callback=self.parsepage2)
              request.meta['item'] = item
              yield request
              yield item
  
      def parsepage2(self, response):
          item = response.meta['item']
          hxs = HtmlXPathSelector(response)
          blocks = hxs.select("//td[@id='overview-top']")
          for block in blocks:
              item = ImdbFrompageItem()
              item["title"] = block.select("h1[@class='header']/span[@itemprop='name']/text()").extract()
              item["year"] = block.select("h1[@class='header']/span[@class='nobr']").extract()
              item["description"] = block.select("p[@itemprop='description']/text()").extract()
              yield item
  </code></pre>
  
  <p>Part of results is:</p>
  
  <pre><code>{"link": , "new_link": }
  {"link": , "new_link": }
  {"link": , "new_link": }
  {"link": , "new_link": }
  ....
  {"link": , "new_link": }
  {"title": , "description":}
  {"title": , "description":}
  next page
  {"link": , "new_link": }
  {"link": , "new_link": }
  {"link": , "new_link": }
  {"title": , "description":}
  </code></pre>
  
  <p>My results don't contain all data ({"title": , "description":}) for each link</p>
  
  <p>But I want something like that:</p>
  
  <pre><code>{"link": , "new_link": }
  {"title": , "description":}
  {"link": , "new_link": }
  {"title": , "description":}
  {"link": , "new_link": }
  {"title": , "description":}
  {"link": , "new_link": }
  ....
  {"link": , "new_link": }
  {"title": , "description":}
  next page
  {"link": , "new_link": }
  {"title": , "description":}
  {"link": , "new_link": }
  {"title": , "description":}
  {"link": , "new_link": }
  {"title": , "description":}
  </code></pre>
  
  <p>Any suggestions as to what I am doing wrong?</p>