title: How can I convert relative paths to absolute paths with my scrapy CrawlSpider?
body: <p>I am new to Scrapy and I am currently trying to write a CrawlSpider that will crawl a forum on the Tor darknet. Currently my CrawlSpider code is:</p>
  
  <pre><code>import scrapy
  from scrapy.spiders import CrawlSpider, Rule
  from scrapy.linkextractors import LinkExtractor
  
  class HiddenAnswersSpider(CrawlSpider):
      name = 'ha'
      start_urls = ['http://answerstedhctbek.onion/questions']
      allowed_domains = ['http://answerstedhctbek.onion', 'answerstedhctbek.onion']
      rules = (
              Rule(LinkExtractor(allow=(r'answerstedhctbek.onion/\d\.\*', r'https://answerstedhctbek.onion/\d\.\*')), follow=True, process_links='makeAbsolutePath'),
              Rule(LinkExtractor(allow=()), follow=True, process_links='makeAbsolutePath')
  
              )
  
   def makeAbsolutePath(links):
      for i in range(links):
            links[i] = links[i].replace("../","")
      return links
  </code></pre>
  
  <p>Because the forum uses relative path, I have tried to create a custom process_links to remove the "../" however when I run my code I am still recieving:</p>
  
  <pre><code>2017-11-11 14:46:46 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;400 http://answerstedhctbek.onion/../badges&gt;: HTTP status code is not handled or not allowed
  2017-11-11 14:46:46 [scrapy.core.engine] DEBUG: Crawled (400) &lt;GET http://answerstedhctbek.onion/../general-guidelines&gt; (referer: http://answerstedhctbek.onion/questions)
  2017-11-11 14:46:47 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;400 http://answerstedhctbek.onion/../general-guidelines&gt;: HTTP status code is not handled or not allowed
  2017-11-11 14:46:47 [scrapy.core.engine] DEBUG: Crawled (400) &lt;GET http://answerstedhctbek.onion/../contact-us&gt; (referer: http://answerstedhctbek.onion/questions)
  2017-11-11 14:46:47 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;400 http://answerstedhctbek.onion/../contact-us&gt;: HTTP status code is not handled or not allowed
  2017-11-11 14:46:48 [scrapy.core.engine] DEBUG: Crawled (400) &lt;GET http://answerstedhctbek.onion/../questions?sort=hot&gt; (referer: http://answerstedhctbek.onion/questions)
  2017-11-11 14:46:48 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response &lt;400 http://answerstedhctbek.onion/../questions?sort=hot&gt;: HTTP status code is not handled or not allowed
  2017-11-11 14:46:48 [scrapy.core.engine] DEBUG: Crawled (400) &lt;GET http://answerstedhctbek.onion/../questions?sort=votes&gt; (referer: http://answerstedhctbek.onion/questions)
  </code></pre>
  
  <p>As you can see, I am still getting 400 errors due to the bad path. Why isn't my code removing the "../" from the links?</p>
  
  <p>Thanks!</p>