title: Unable to go next page
body: <p>Trying to scrape the internet archive website <strong>(Wayback Machine)</strong>: <a href="https://web.archive.org/web/20150906222155mp_/https://www.zalando.co.uk/womens-clothing/" rel="nofollow noreferrer">https://web.archive.org/web/20150906222155mp_/https://www.zalando.co.uk/womens-clothing/</a>.</p>
  <p>I am succesful in scraping the 1st page content, but can't move to the next page. I have tried multiple <code>xpath</code> to move to next pages:</p>
  <pre><code># 1
   next_page_url = response.xpath(&quot;//li[a[contains(.,'&gt;')]]//@href&quot;).extract_first() # does not work
  # 2
  next_page_url = response.xpath(//a[@class='catalogPagination_page' and text() ='&gt;'])[1]//@href).get() # does not work
  </code></pre>
  <p>I have tried converting to <code>absolute url</code> (and without) but again with no luck.</p>
  <p>Can anyone help with new <code>xpath</code> or <code>css selectors</code> that I can finally <code>scrape</code> the next pages?</p>
  <p>Below you can see my full code:</p>
  <pre><code># -*- coding: utf-8 -*-
  import scrapy
  class ZalandoWomenSpider(scrapy.Spider):
      name = 'zalando_women_historic_2015'
      allowed_domains = ['www.web.archive.org']
      start_urls = ['https://web.archive.org/web/20150906222155mp_/https://www.zalando.co.uk/womens-clothing/']
  
      def parse(self, response):
          products = response.xpath(&quot;//a[@class='catalogArticlesList_productBox']&quot;)
          for product in products:
              link = product.xpath(&quot;.//@href&quot;).get()
              absolute_url = f&quot;https://web.archive.org{link}&quot;
              yield scrapy.Request(url=absolute_url,callback=self.parse_product,dont_filter=True,meta={'link':link})
          # process next page 
          next_page_url = response.xpath(&quot;//li[a[contains(.,'&gt;')]]//@href&quot;).extract_first() #(//a[@class='catalogPagination_page' and text() ='&gt;'])[1]//@href
          absolute_next_page_url = f&quot;https://web.archive.org{next_page_url}&quot;
          #absolute_next_page_url = next_page_url
          #absolute_next_page_url = response.urljoin(next_page_url)
          if next_page_url:
              yield scrapy.Request(url=absolute_next_page_url,callback=self.parse)
  
      def parse_product(self, response):
          link = response.request.meta['link']
          brand = response.xpath(&quot;//span[@itemprop='brand']/text()&quot;).get()
          price = response.xpath(&quot;//span[@class='price oldPrice nowrap']/text()&quot;).get()
          price1 = response.xpath(&quot;//span[@itemprop='price']/text()&quot;).get()
          price2 = response.xpath(&quot;//div[@class='boxPrice']//span[contains(@class,'price')]/text()&quot;).get()
          disc_price = response.xpath(&quot;//span[@class='price specialPrice nowrap']/text()&quot;).get()
          
          product_type = response.xpath(&quot;//span[@itemprop='name']/text()&quot;).get() 
          material = response.xpath(&quot;//div[@class='content']//li[contains(.,'material')]/text()&quot;).get()
          yield {   
              'brand_name': brand,
              'product_price':price,
              'product_price1':price1,
              'product_price2':price2,
              'product_price_b4_disc':disc_price,
              'link':link,
              'product_type':product_type,
              'material':material}
  </code></pre>