title: Created my first webcrawler, how do I get a "URL stack-trace"/history of URL's for each endpoint?
body: <p>I created a web crawler that, given a base_url, will spider out and find all possible endpoints. While I am able to get all the endpoints, I need a way to figure out how I got there in the first place -- a 'url stack-trace' persay or breadcrumbs of url's leading to each endpoint. </p>
  
  <p>I first start by finding all url's given a base url. Since the sublinks I'm looking for are within a json, I thought the best way to do this would be using a variation of a recurisve dictionary example I found here: <a href="http://www.saltycrane.com/blog/2011/10/some-more-python-recursion-examples/" rel="nofollow">http://www.saltycrane.com/blog/2011/10/some-more-python-recursion-examples/</a>:</p>
  
  <pre><code>import requests
  import pytest
  import time
  
  BASE_URL = "https://www.my-website.com/"
  
  def get_leaf_nodes_list(base_url):
      """
      :base_url: The starting point to crawl
      :return: List of all possible endpoints
      """
  
      class Namespace(object):
          # A wrapper function is used to create a Namespace instance to hold the ns.results variable
          pass
      ns = Namespace()
      ns.results = []
  
      r = requests.get(BASE_URL)
      time.sleep(0.5)  # so we don't cause a DDOS?
      data = r.json()
  
      def dict_crawler(data):
          # Retrieve all nodes from nested dict
          if isinstance(data, dict):
              for item in data.values():
                  dict_crawler(item)
          elif isinstance(data, list) or isinstance(data, tuple):
              for item in data:
                  dict_crawler(item)
          else:
              if type(data) is unicode:
                  if "http" in data:  # If http in value, keep going
                      # If data is not already in ns.results, don't append it
                      if str(data) not in ns.results:  
                          ns.results.append(data)
                          sub_r = requests.get(data)
                          time.sleep(0.5)  # so we don't cause a DDOS?
                          sub_r_data = sub_r.json()
                          dict_crawler(sub_r_data)
  
      dict_crawler(data)
      return ns.results
  </code></pre>
  
  <p>To reiterate, the get_leaf_nodes_list does a get request and looks for any values within the json for a url (if 'http' string is in the value for each key) to recursively do more get requests until there's no url's left. </p>
  
  <p>So to reiterate here are the questions I have:</p>
  
  <ul>
  <li><p>How do I get a linear history of all the url's I hit to get to each endpoint? </p></li>
  <li><p>Corollary to that, how would I store this history? As the leaf nodes list grows, my process gets expontentially slower and I am wondering if there's a better data type out there to store this information or a more efficient process to the code above. </p></li>
  </ul>