title: Scrapy pagination timing error
body: <p>So I set up a spider very similarly to the example on scrapy.</p>
  
  <p>I want the spider to crawl all of the quotes BEFORE going to the next page. I also want it to parse only 1 quote per second. So if there were 20 quotes on a page, it would take 20 seconds to scrape the quotes then 1 second to go to the next page.</p>
  
  <p>As of right now, my current implementation is iterating through each page first before actually getting the quote information. </p>
  
  <pre><code>import scrapy
  
  class AuthorSpider(scrapy.Spider):
  name = 'author'
  
  start_urls = ['http://quotes.toscrape.com/']
  
  def parse(self, response):
      # follow links to author pages
      for href in response.css('.author+a::attr(href)').extract():
          yield scrapy.Request(response.urljoin(href),
                               callback=self.parse_author)
  
      # follow pagination links
      next_page = response.css('li.next a::attr(href)').extract_first()
      if next_page is not None:
          next_page = response.urljoin(next_page)
          yield scrapy.Request(next_page, callback=self.parse)
  
  def parse_author(self, response):
      def extract_with_css(query):
          return response.css(query).extract_first().strip()
  
      yield {
          'name': extract_with_css('h3.author-title::text'),
          'birthdate': extract_with_css('.author-born-date::text'),
          'bio': extract_with_css('.author-description::text'),
      }
  </code></pre>
  
  <p>Here are the basics of my settings.py file</p>
  
  <pre><code>ROBOTSTXT_OBEY = True
  CONCURRENT_REQUESTS = 1
  DOWNLOAD_DELAY = 2
  </code></pre>