title: Crawl a whole website with selenium in python recursive
body: <p>I'm new in python and i try to crawl a whole website recursive with selenium.
  I would like to do this with selenium because i want get all cookies which the website is used. I know that other tools can crawl a website easier and faster but other tools can't give me all cookies (first and third party).</p>
  
  <p>Here my code:</p>
  
  <pre><code>from selenium import webdriver
  import os, shutil
  
  
  url = "http://example.com/"
  links = set()
  
  
  def crawl(start_link):
      driver.get(start_link)
      elements = driver.find_elements_by_tag_name("a")
      urls_to_visit = set()
  
      for el in elements:
          urls_to_visit.add(el.get_attribute('href'))
  
      for el in urls_to_visit:
          if url in el:
              if el not in links:
                  links.add(el)
                  crawl(el)
              else:
                  return
  
  
  dir_name = "userdir"
  if os.path.isdir(dir_name):
      shutil.rmtree(dir_name)
  co = webdriver.ChromeOptions()
  co.add_argument("--user-data-dir=userdir")
  driver = webdriver.Chrome(options = co)
  crawl(url)
  print(links)
  driver.close();
  </code></pre>
  
  <p>My problem is that the crawl function not open all pages from the website apparently. On some websites i can navigate to pages by hand that the function not reached. Why?</p>