title: Problems getting next page when scraping with scrapy
body: <p>I have a scrapy code which doesn't crawl pagination links and i'm stuck.</p>
  
  <p>The source of the page is:</p>
  
  <p><a href="https://www.levenhuk.bg/katalog/teleskopi/?page=1" rel="nofollow noreferrer">https://www.levenhuk.bg/katalog/teleskopi/?page=1</a></p>
  
  <p>My code is:</p>
  
  <pre><code>
  
  
  import scrapy
  
  
  class TelescopesSpider(scrapy.Spider):
      name = 'telescopes'
      allowed_domains = ['https://www.levenhuk.bg/']
      start_urls = ['https://www.levenhuk.bg/katalog/teleskopi/?page=1']
      download_delay = 3
  
      def parse(self, response):
          for product in response.xpath('//div[@class="catalog-item"]'):
              yield {
  
                  # 'name': product.xpath('.//span[@itemprop="name" and contains(text(), "Levenhuk")]/text()').get(),
                  'name': product.xpath('.//span[@itemprop="name"]/text()').get(),
                  # 'price': product.xpath('.//div[@class="price"]/span/text()').get(),
                  'price': product.xpath('.//span[@itemprop="price"]/text()').re_first(r'[0-9]+,[0-9]+'),
                  'short_discr': product.xpath('.//div[@class="opis-item"]/p/strong/text()').get()
  
              }
  
              next_page_url = response.xpath('//*[@class="pagesCount"][1]//@href').get()
              if next_page_url is not None:
                  yield scrapy.Request(response.urljoin(next_page_url))
  
  
  
  </code></pre>