title: Scrapy, Selenium, Python - problem with pagination (missing pages)
body: <p>I have a problem with running scrapy. Seems like scrapy is skiping last pages. For example I've set 20 pages to scrap but Scrapy is missing last 10 or 7 pages. It does not have any problem when Im setting one single page &quot;for page in range(6,7)&quot;. Terminal shows that it is scraping all pages from 1 to 100 but output in my database is ending at random pages. Any ideas why is that heppening?</p>
  <p>Mayber there is a way to run Scrapy synchronously. To scrap every item in first page -&gt; second page -&gt; third page and so on</p>
  <pre><code>class SomeSpider(scrapy.Spider):
      name = 'default'
      urls = [f'https://www.somewebsite.com/pl/c/cat?page={page}' for page in range(1, 101)]
  
      service = Service(ChromeDriverManager().install())
      options = Options()
      options.add_argument('--ignore-certificate-errors')
      options.add_argument(&quot;--headless&quot;)
      options.add_argument(&quot;--allow-running-insecure-content&quot;)
      options.add_argument(&quot;--enable-crash-reporter&quot;)
      options.add_argument(&quot;--disable-popup-blocking&quot;)
      options.add_argument(&quot;--disable-default-apps&quot;)
      options.add_argument(&quot;--incognito&quot;)
  
      driver = webdriver.Chrome(service=service, options=options)
  
      def start_requests(self):
          for url in self.urls:
              yield scrapy.Request(
                  url=url,
                  callback=self.parse
              )
  
      def parse(self, response):
          for videos in response.css('div.card-img'):
              item = WebsitescrapperItem()
              
              link = f'https://www.somewebsite.com{videos.css(&quot;a.item-link&quot;).attrib[&quot;href&quot;]}'
              SomeSpider.driver.get(link)
              domain_name = SomeSpider.driver.current_url
              SomeSpider.driver.back()
  
              item['name'] = videos.css('span.title::text').get().strip()
              item['duration'] = videos.css('span.duration::text').get().strip()
              item['image'] = videos.css('img.thumb::attr(src)').get()
              item['url'] = domain_name
              item['hd'] = videos.css('span.hd-icon').get()
  
              yield item
  </code></pre>