title: Multi-page scrapy yielding my items too quickly to finish - functions not chaining and waiting for completion
body: <p>I'm making a soccer app trying to wrap my head around how multi page scraping works.</p>
  
  <p>For example, on the first page (<a href="http://footballdatabase.com/ranking/world/1" rel="nofollow">http://footballdatabase.com/ranking/world/1</a>) is 2 sets of links I want to scrape: the club name link, and pagination links</p>
  
  <p>I want to go through <strong>a)</strong> every page (pagination), then <strong>b)</strong> go through each club and grab its current eu ranking.</p>
  
  <p>The code I've written works, somewhat. However I end up with only around 45 results instead of the 2000+ clubs. --Note: There are 45 pages of pagination. So it appears as soon as it has looped through that everything finishes and my item is yielded.</p>
  
  <p>How can I make it all chain together so I end up with more like 2000+ results?</p>
  
  <p>Here is my code</p>
  
  <pre><code># get Pagination links
  def parse(self, response):
      for href in response.css("ul.pagination &gt; li &gt; a::attr('href')"):
          url = response.urljoin(href.extract())
          yield scrapy.Request(url, callback=self.parse_club)
  
  # get club links on each of the pagination pages
  def parse_club(self, response):
  
  
      # loop through each of the rows
      for sel in response.xpath('//table/tbody/tr'):
  
          item = rankingItem()
  
              item['name'] = sel.xpath('td/a/div[@class="limittext"]/text()').extract()
  
              # get more club information
              club_href = sel.xpath('td[2]/a[1]/@href').extract_first()  
              club_url = response.urljoin(club_href) 
              request = scrapy.Request(club_url,callback=self.parse_club_page_2)
  
              request.meta['item'] = item
              return request
  
  # get the EU ranking on each of the club pages
  def parse_club_page_2(self,response):
  
      item = response.meta['item']
      item['eu_ranking'] = response.xpath('//a[@class="label label-default"][2]/text()').extract() 
  
      yield item
  </code></pre>