title: Recursive crawling same page using javascript with scrapy and splash
body: <p>I am crawling a site which have javascript to go to next page. I am using splash to execute my javascript code on first page. But I was able to go to 2nd page. But I am unable to go to the 3,4,5.... pages. crawling is stopped after only one page.</p>
  
  <p>the link I am crawling:
  <a href="http://59.180.234.21:8788/user/viewallrecord.aspx" rel="nofollow noreferrer">http://59.180.234.21:8788/user/viewallrecord.aspx</a></p>
  
  <p>The code:</p>
  
  <pre><code>import scrapy
  from scrapy_splash import SplashRequest
  from time import sleep
  
  
  class MSEDCLSpider(scrapy.Spider):
      name = "msedcl_spider"
      scope_path = 'body &gt; table:nth-child(11) tr &gt; td.content_area &gt; table:nth-child(4) tr:not(:first-child)'
      ref_no_path = "td:nth-child(1) ::text"
      title_path = "td:nth-child(2) ::text"
      end_date_path = "td:nth-child(5) ::text"
      fee_path = "td:nth-child(6) ::text"
      start_urls = ["http://59.180.234.21:8788/user/viewallrecord.aspx"]
  
      lua_src = """function main(splash)
          local url = splash.args.url
          splash:go(url)
          splash:wait(2.0)
          splash:runjs("document.querySelectorAll('#lnkNext')[0].click()")
          splash:wait(4.0)
          return {
              splash:html(),
          }
          end
          """
  
      def start_requests(self):
          for url in self.start_urls:
              yield SplashRequest(
                  url,
                  self.parse,
                  endpoint='execute',
                  method='POST',
                  dont_filter=True,
                  args={
                      'wait': 1.0,
                      'lua_source': self.lua_src,
                  },
              )
  
  
      def parse(self, response):
          print response.status
          scopes = response.css('#page-info').extract()[0]
          print(response.url)
          print(scopes)
  </code></pre>
  
  <p>I am newbie to both scrapy and splash. please be gentle. Thank you</p>