title: How do I go to the next page with Scrapy in Python
body: <p>I am trying to scrape job vacancies from indeed. Everything in my scraper works, except for the fact that it only scrapes the first page. Does anybody know what might be the problem.</p>
  <pre><code>class IndeedSpider(scrapy.Spider):
      name = 'indeed'
      allowed_domains = ['nl.indeed.com']
      start_urls = ['https://nl.indeed.com/vacatures?l=Woerden&amp;limit=50&amp;lang=en&amp;start=0']
  
  
      def parse(self, response):
          urls= response.xpath('//h2[contains(@class, &quot;jobTitle&quot;)]/a/@href').extract()
          for url in urls:
              url = response.urljoin(url)
              yield scrapy.Request(url=url, callback=self.parse_details)
  
  
          next_page_url = response.css('ul.pagination-list li:nth-child(7) a::attr(href)').get()
          if next_page_url is not None:
              next_page_url = response.urljoin(next_page_url)
  
              yield scrapy.Request(url=next_page_url, callback=self.parse) 
  
  
      def parse_details(self, response):
          Page = response.url
          Title = response.css('h1.icl-u-xs-mb--xs.icl-u-xs-mt--none.jobsearch-JobInfoHeader-title ::text').extract_first()
          Company = response.css('div.icl-u-lg-mr--sm.icl-u-xs-mr--xs ::text').extract_first() 
          Location =  response.css('.jobsearch-DesktopStickyContainer-companyrating+ div div ::text').extract_first()
          Description = response.xpath('normalize-space(//div[contains(@class, &quot;jobsearch-jobDescriptionText&quot;)])').extract_first()
          Date= response.css('span.jobsearch-HiringInsights-entry--text ::text').extract_first()
  
          yield {
          'Page': Page,
          'Title': Title,
          'Company': Company,
          'Location': Location,
          'Description': Description,
          'Date':Date
          }
  </code></pre>
  <p>Anybody that can help me?</p>