title: How to follow links in Scrapy if there is no href?
body: <p>I am trying to follow links in Scrapy when I already parsed one page and extract information from there. The problem is, webpage has no href, so I can't just follow it with ease. I have managed to expand my XPath query with <strong>@data-param</strong> and finally got something: <strong>page=2</strong>.</p>
  
  <p>The problem is I am not sure how to follow this link as I want to pass <code>listName["listLinkMaker"]</code> to my URL generator or composer.</p>
  
  <p>Should I make another "def" and call it say, def parse_pagination to follow links?</p>
  
  <p>JSON used in code is really simple:</p>
  
  <pre><code>[
  {"storeName": "Interspar", "storeLinkMaker": "https://popusti.njuskalo.hr/trgovina/Interspar"}
  ]
  </code></pre>
  
  <p>Code below:</p>
  
  <pre><code># -*- coding: utf-8 -*-
  import scrapy
  import json
  
  
  class LoclocatorTestSpider(scrapy.Spider):
      name = "loclocator_test"
      start_urls = []
  
      with open("test_one_url.json", encoding="utf-8") as json_file:
          data = json.load(json_file)
          for store in data:
              storeName = store["storeName"]
              storeLinkUrl = store["storeLinkMaker"]
              start_urls.append(storeLinkUrl)
  
      def parse(self, response):
          selector = "//div[@class='mainContentWrapInner cf']"
  
          store_name_selector = ".//h1[@class='title']/text()"
          store_branches_selector = ".//li/a[@class='xiti']/@href"
  
          for basic_info in response.xpath(selector):
              store_branches = {}
  
              store_branches["storeName"] = basic_info.xpath(store_name_selector).extract_first()
              # This specific XPath extracts 1st part of link needed to crawl all of store branches
              store_branches["storeBranchesLink"] = basic_info.xpath(store_branches_selector).extract_first() + "?"
  
              store_branches_url = basic_info.xpath(store_branches_selector).extract_first()
              yield response.follow(store_branches_url, self.parse_pagination, meta={"store_branches": store_branches})
  
  
      def parse_branches(self, response):
          store_branches_name_selector = "//li[@class='xiti']"
          store_branches = response.meta["store_branches"]
  
          for store_branch in response.xpath(store_branches_name_selector):
              store_branches["storeBranchName"] = store_branch.xpath(".//span[@class='title']/text()").extract_first()
  
              yield store_branches
  
          # This specific XPath extracts 2nd part of link needed to crawl all of store branches
          # URL should look like: https://popusti.njuskalo.hr/trgovina/Interspar?page=n where n&gt;0
          links = response.selector.xpath("//li[@class='next']/button[@class='nBtn link xiti']/@data-param").extract()
          for link in links:
              absolute_url = #LIST FROM FIRST PARSE (ie. store_branches["storeBranchesLink"]) + link
              yield scrapy.Request(absolute_url, callback=self.parse_branches)
  </code></pre>
  
  <p>Thank you.</p>