title: Scraping infinite scroll page using Scrapy Python
body: <p>I am trying to scrape all the reviews from steam games, but steam has both infinite scroll, till it reaches a specific number of reviews, then a button for loading more content. How do I navigate that with scrapy?</p>
  <p>My code:</p>
  <pre><code>i = 1
  url_ = 'https://steamcommunity.com/app/1506830/homecontent/?userreviewscursor=AoIIPwYYan/HzbUD&amp;userreviewsoffset={}0&amp;p={}&amp;workshopitemspage={}&amp;readytouseitemspage={}&amp;mtxitemspage={}&amp;itemspage={}&amp;screenshotspage={}&amp;videospage={}&amp;artpage={}&amp;allguidepage={}&amp;webguidepage={}&amp;integratedguidepage={}&amp;discussionspage={}&amp;numperpage=10&amp;browsefilter=trendweek&amp;browsefilter=trendweek&amp;l=english&amp;appHubSubSection=10&amp;filterLanguage=default&amp;searchText=&amp;maxInappropriateScore=50&amp;forceanon=1'
  
  #url_ is url gathered from inspecting Network while scrolling 
  
  class ReviewSpider(scrapy.Spider):
      name = 'steam_test'
      allowed_domains = ['store.steampowered.com']
      start_urls = ['https://steamcommunity.com/app/1517290/reviews/']
  
      
      def start_requests(self):
          yield Request(url, callback=self.parse)
  
  
      def parse(self, response):
  
          page = get_page(response)
          product_id = get_product_id(response)
      
          # Load all reviews on current page.
          reviews = response.css('div.apphub_CardContentMain')
          for review in (reviews):
                   
              yield{
                  &quot;text&quot;:review.css('div.apphub_CardTextContent::text').get(),
                  &quot;Recommend&quot;: review.css('div.title::text').get(), 
                  &quot;date&quot;: review.css('div.date_posted::text').get()
                  }
  </code></pre>
  <p>Pagination section that's not working:</p>
  <pre><code>
          if response.status == 200:    
              global i 
              i+= 1
              SteamFifaSpider.start_urls = url_.format(i,i,i,i,i,i,i,i,i,i,i,i,i)
  
  </code></pre>
  <p>How do I fix the infinite scroll for Steam?</p>