title: How to paginate once number of pages has been scraped? (Scrapy)
body: <p>I'm trying to paginate through a review site that doesn't have hyperlinked pagination buttons. I've written logic to paginate and hardcoded the number of pages for each link. However, I'm wondering if it's possible to use information that I scrape as the number of pages for a given link in <code>start_requests</code>.</p>
  
  <p>Spider code here (with 2 links to paginate through):</p>
  
  <pre><code>class TareviewsSpider(scrapy.Spider):
      name = 'tareviews'
      allowed_domains = ['tripadvisor.com']
      # start_urls = []
  
      def start_requests(self):
          for page in range(0,395,5):
              yield self.make_requests_from_url('https://www.tripadvisor.com/Hotel_Review-g60795-d102542-Reviews-or{}-Courtyard_Philadelphia_Airport-Philadelphia_Pennsylvania.html'.format(page))
          for page in range(0,1645,5):
              yield self.make_requests_from_url('https://www.tripadvisor.com/Hotel_Review-g60795-d122332-Reviews-or{}-The_Ritz_Carlton_Philadelphia-Philadelphia_Pennsylvania.html'.format(page))
  
      def parse(self, response):
          for idx,review in enumerate(response.css('div.review-container')):
              item = {
                  'num_reviews': response.css('span.reviews_header_count::text')[0].re(r'\d{0,3}\,?\d{1,3}'),
                  'hotel_name': response.css('h1.heading_title::text').extract_first(),
                  'review_title': review.css('span.noQuotes::text').extract_first(),
                  'review_body': review.css('p.partial_entry::text').extract_first(),
                  'review_date': review.xpath('//*[@class="ratingDate relativeDate"]/@title')[idx].extract(),
                  'num_reviews_reviewer': review.css('span.badgetext::text').extract_first(),
                  'reviewer_name': review.css('span.scrname::text').extract(),
                  'bubble_rating': review.xpath("//div[contains(@class, 'reviewItemInline')]//span[contains(@class, 'ui_bubble_rating')]/@class")[idx].re(r'(?&lt;=ui_bubble_rating bubble_).+?(?=0)')
              }
              yield item
  </code></pre>
  
  <p><code>'num_reviews'</code> is the value for the number of the last page for each link. In the <code>for loop</code> in <code>start_requests</code> it is <code>395</code> and <code>1645</code>.</p>
  
  <p>Is this possible? I'd like to avoid a headless browser if possible. Thanks!</p>