title: Scrapy: Unsuccessful iterating over a list and pagination
body: <p>My goal is to extract all 25 rows ( 6 items per row) per page then iterate over each of the 40 pages.</p>
  
  <p>Currently, my spider extracts the first row from page 1-3 (see CSV output image).   </p>
  
  <p>I assumed the, <code>list_iterator()</code> function would iterate over each row; however,  there appears to be an error in either my <code>rules</code> or <code>list_iterator()</code> function that is not allowing all rows per page to be scrapped.   </p>
  
  <p>Any assistance or advice is greatly appreciated!</p>
  
  <p>propub_spider.py:</p>
  
  <pre><code>import scrapy 
  from scrapy.contrib.spiders import CrawlSpider, Rule
  from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor 
  from propub.items import PropubItem
  from scrapy.http import Request
  
  class propubSpider(CrawlSpider):
      name = 'prop$'
      allowed_domains = ['https://projects.propublica.org']
      max_pages = 40
      start_urls = [
          'https://projects.propublica.org/docdollars/search?state%5Bid%5D=33',
          'https://projects.propublica.org/docdollars/search?page=2&amp;state%5Bid%5D=33',
          'https://projects.propublica.org/docdollars/search?page=3&amp;state%5Bid%5D=33']
  
      rules = (Rule(SgmlLinkExtractor(allow=('\\search?page=\\d')), 'parse_start_url', follow=True),)
  
      def list_iterator(self):
          for i in range(self.max_pages):
              yield Request('https://projects.propublica.org/docdollars/search?page=d' % i, callback=self.parse)
  
      def parse(self, response):
          for sel in response.xpath('//*[@id="payments_list"]/tbody'):
              item = PropubItem()
              item['payee'] = sel.xpath('tr[1]/td[1]/a[2]/text()').extract()
              item['link'] = sel.xpath('tr[1]/td[1]/a[1]/@href').extract()
              item['city'] = sel.xpath('tr[1]/td[2]/text()').extract()
              item['state'] = sel.xpath('tr[1]/td[3]/text()').extract()
              item['company'] = sel.xpath('tr[1]/td[4]').extract()
              item['amount'] =  sel.xpath('tr[1]/td[7]/span/text()').extract()
              yield item 
  </code></pre>
  
  <p>pipelines.py:</p>
  
  <pre><code>import csv
  
  class PropubPipeline(object):
  
      def __init__(self):
          self.myCSV = csv.writer(open('C:\Users\Desktop\propub.csv', 'wb'))
          self.myCSV.writerow(['payee', 'link', 'city', 'state', 'company', 'amount'])
  
      def process_item(self, item, spider):
          self.myCSV.writerow([item['payee'][0].encode('utf-8'), 
          item['link'][0].encode('utf-8'), 
          item['city'][0].encode('utf-8'), 
          item['state'][0].encode('utf-8'),
          item['company'][0].encode('utf-8'),
          item['amount'][0].encode('utf-8')])
          return item
  </code></pre>
  
  <p>items.py:</p>
  
  <pre><code>import scrapy
  from scrapy.item import Item, Field
  
  class PropubItem(scrapy.Item):
      payee = scrapy.Field()
      link = scrapy.Field()
      city = scrapy.Field()
      state = scrapy.Field()
      company = scrapy.Field()
      amount =  scrapy.Field()
      pass
  </code></pre>
  
  <p>CSV output:</p>
  
  <p><img src="https://i.stack.imgur.com/sTdZn.png" alt="enter image description here"></p>