title: Scrapy crawler pagination for indeed website
body: <pre><code># -*- coding: utf-8 -*-
  import scrapy
  
  
  class SearchSpider(scrapy.Spider):
      name = 'search'
      allowed_domains = ['www.indeed.com/']
      start_urls = ['https://www.indeed.com/jobs?q=data%20analyst&amp;l=united%20states']
  
      def parse(self, response):
          listings = response.xpath('//*[@data-tn-component=&quot;organicJob&quot;]')
          for listing in listings:
              title = listing.xpath('.//a[@data-tn-element=&quot;jobTitle&quot;]/@title').extract_first()
              link = listing.xpath('.//h2[@class=&quot;title&quot;]//a/@href').extract_first()
              company = listing.xpath('normalize-space(.//span[@class=&quot;company&quot;]//a/text())').extract_first()
  
              yield {'title':title,
              'link':link,
              'company':company}
              
          next_page = response.xpath('//ul[@class=&quot;pagination-list&quot;]//a/@href').extract_first()
          if next_page:
              yield scrapy.Request(response.urljoin(next_page),callback=self.parse)
  </code></pre>
  <p>I am trying to extract all the job titles and company for every job posting in all the indeed pages. However, I am stuck at a point, because the forward button on the indeed page does not have a fixed link which my scraper could follow instead the next page url is the same as the numbered button. Which means that even after requesting the next page url, the numbers at the end change which does not allow me to extract the next page. I am trying to refrain from using selenium or splash, since I am trying to get my results through only Scrapy or Beautifull Soup. However, any help would be greatly appreciated.</p>