title: How to get all requests from selenium webdriver in Python
body: <p>I am trying to build an automated tool to validate a web analytics tag implementation. I am using Python 3 + Selenium and PhantomJS to automate browsing and collect the tags that are firing.
  I am using the code below to crawl the web pages and get the har log with all requests, similar to what the network tab in Google Chrome does. The issue is, for some websites it works perfectly and I can see the google-analytics.com/collect? calls. For others I can see it on Google Chrome network tab, but my crawler wont register it. I am sure I am waiting the page to load, but still it does not work. </p>
  
  <p>Example page where I find the analytics/collect? call in the har log:
  <a href="https://www.stackoverflow.com">https://www.stackoverflow.com</a></p>
  
  <p>Example of page where I don't find it the analytics/collect? call in the har log, but find it on the chrome network tab.
  <a href="https://www.nike.com.br" rel="nofollow noreferrer">https://www.nike.com.br</a></p>
  
  <p><strong>Code to instantiate a webdriver</strong></p>
  
  <pre><code>def get_driver():
    desired_capabilities = DesiredCapabilities.PHANTOMJS.copy()
    service_args = ['--ignore-ssl-errors=yes']
    desired_capabilities['phantomjs.page.customHeaders.User-Agent'] = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36'
    driver = webdriver.PhantomJS(service_args=service_args,
                                 desired_capabilities=desired_capabilities)
    driver.set_page_load_timeout(60)
    return driver
  </code></pre>
  
  <p><strong>Code to parse the har log</strong></p>
  
  <pre><code>def get_requests(url):
    driver = get_driver()
    try:
      driver.get(url)
          print(j)
    except Exception as e:
      print(e)
      return None
    log = driver.get_log('har')
    log = json.loads(log[0]['message'])
    log_list = log['log']['entries']
    return log_list
  </code></pre>