title: I have started learning Scrapy and was trying to scrape LetterBoxd. I am unable to integrate Splash and Scrapy. How do I scrape the first 2000 pages?
body: <pre><code>import scrapy
  from scrapy_splash import SplashRequest
   
  class Test4basicSpider(scrapy.Spider):
      name = 'test4Basic'
      allowed_domains = ['letterboxd.com']
      start_urls = ['https://letterboxd.com/films/popular/size/small/page/1/']
      # start_urls = ['https://letterboxd.com/film/tales-from-the-darkside-the-movie/']
   
      script1 = '''
      function main(splash, args)
          splash.private_mode_enabled = false
          url = args.url
          assert(splash:go(args.url))
          assert(splash:wait(0.5))
          return {
              html = splash:html()
          }
      end
      '''
   
      def start_requests(self):
          yield SplashRequest(url='https://letterboxd.com/films/popular/size/small/page/1/', callback=self.parse, endpoint=&quot;execute&quot;, args={
              'lua_source': self.script1
          })
   
      def parse(self, response):
          for movie in response.xpath(&quot;//li[@class='listitem poster-container']&quot;):
              movie_url = movie.xpath(&quot;(//a[@class='frame'])[1]/@href&quot;).get()
              yield scrapy.Request(
                  url=f'https://letterboxd.com{movie_url}',
                  callback=self.parse_movie,
              )
   
          next_page = response.xpath(&quot;//a[@class='next']/@href&quot;).get()
          if next_page:
              yield SplashRequest(
                  url=f'https://letterboxd.com{next_page}',
                  endpoint='execute',
                  args={
                      'lua_source': self.script1
                  },
                  callback=self.parse
              )
   
      def parse_movie(self, response):
   
          yield {
              'title': response.xpath('//section[@id=&quot;featured-film-header&quot;]/h1/text()').get(),
              'year': response.xpath('//small[@class=&quot;number&quot;]/a/text()').get(),
              'duration': response.xpath('(//p[@class=&quot;text-link text-footer&quot;]/text())[1]').get(),
              'genre': response.xpath('//div[@class=&quot;text-sluglist capitalize&quot;]/p/a/text()').getall(),
              'rating': response.xpath('//a[contains(@class, &quot;tooltip display-rating&quot;)]/text()').get(),
              'language': response.xpath('((//span[contains(text(), &quot;Language&quot;)]/parent::node()/following::node())/p/a/text())[1]').get()
          }
  </code></pre>
  <p>I am successfully able to <strong>scrape</strong> all fields apart from Ratings (which needs Javascript) if I <strong>scrape</strong> directly from any movie URL. I have tried to apply some logic for pagination as well but I get nothing when I try to crawl. Where am I going wrong with this code? Could it be prohibited by robots.txt file of Letterboxd, I don't see how.</p>