title: How to save all the network traffic (both request and response headers) from a website using python
body: <p>I am trying to find an object that is downloaded into the browser during the loading of a website.</p>
  <p>This is the website, <a href="https://epco.taleo.net/careersection/alljobs/jobsearch.ftl?lang=en" rel="nofollow noreferrer">https://epco.taleo.net/careersection/alljobs/jobsearch.ftl?lang=en</a>,</p>
  <p>I'm not very good with web technology and such.</p>
  <p>I am trying to save the request and response headers and the actual response using only the link to the website.</p>
  <p>If you look at the network traffic, you can see an object <code>jobsearch.ftl?lang=en</code> that loads towards the end and you can see the reponse and headers.</p>
  <p>Here are the screenshots, of the network event log showing the request and response headers.</p>
  <p><a href="https://i.stack.imgur.com/q9U2O.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/q9U2O.png" alt="network event log" /></a></p>
  <p>And the actual response.</p>
  <p><a href="https://i.stack.imgur.com/beeLt.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/beeLt.png" alt="response" /></a></p>
  <p>These are the objects that I want to save. How can I do that?</p>
  <p>I have tried</p>
  <pre><code>import json
  from selenium import webdriver
  from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
  from selenium.webdriver.common.by import By
  from selenium.common.exceptions import NoSuchElementException, TimeoutException, StaleElementReferenceException
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC
  from selenium.webdriver.chrome.options import Options
  
  chromepath = &quot;~/chromedriver/chromedriver&quot;
  
  caps = DesiredCapabilities.CHROME
  caps['goog:loggingPrefs'] = {'performance': 'ALL'}
  driver = webdriver.Chrome(executable_path=chromepath, desired_capabilities=caps)
  driver.get('https://epco.taleo.net/careersection/alljobs/jobsearch.ftl?lang=en')
  
  def process_browser_log_entry(entry):
      response = json.loads(entry['message'])['message']
      return response
  
  browser_log = driver.get_log('performance') 
  events = [process_browser_log_entry(entry) for entry in browser_log]
  events = [event for event in events if 'Network.response' in event['method']]
  
  </code></pre>
  <p>But I only get some of the headers, they look like this,</p>
  <pre><code>
  {'method': 'Network.responseReceivedExtraInfo',
    'params': {'blockedCookies': [],
     'headers': {'Cache-Control': 'private',
      'Connection': 'Keep-Alive',
      'Content-Encoding': 'gzip',
      'Content-Security-Policy': &quot;frame-ancestors 'self'&quot;,
      'Content-Type': 'text/html;charset=UTF-8',
      'Date': 'Mon, 27 Sep 2021 18:18:10 GMT',
      'Keep-Alive': 'timeout=5, max=100',
      'P3P': 'CP=&quot;CAO PSA OUR&quot;',
      'Server': 'Taleo Web Server 8',
      'Set-Cookie': 'locale=en; path=/careersection/; secure; HttpOnly',
      'Transfer-Encoding': 'chunked',
      'Vary': 'Accept-Encoding',
      'X-Content-Type-Options': 'nosniff',
      'X-UA-Compatible': 'IE=edge',
      'X-XSS-Protection': '1'},
     'headersText': 'HTTP/1.1 200 OK\r\nDate: Mon, 27 Sep 2021 18:18:10 GMT\r\nServer: Taleo Web Server 8\r\nCache-Control: private\r\nP3P: CP=&quot;CAO PSA OUR&quot;\r\nContent-Encoding: gzip\r\nVary: Accept-Encoding\r\nX-Content-Type-Options: nosniff\r\nSet-Cookie: locale=en; path=/careersection/; secure; HttpOnly\r\nContent-Security-Policy: frame-ancestors \'self\'\r\nX-XSS-Protection: 1\r\nX-UA-Compatible: IE=edge\r\nKeep-Alive: timeout=5, max=100\r\nConnection: Keep-Alive\r\nTransfer-Encoding: chunked\r\nContent-Type: text/html;charset=UTF-8\r\n\r\n',
     'requestId': '1E3CDDE80EE37825EF2D9C909FFFAFF3',
     'resourceIPAddressSpace': 'Public'}},
   {'method': 'Network.responseReceived',
    'params': {'frameId': '1624E6F3E724CA508A6D55D556CBE198',
     'loaderId': '1E3CDDE80EE37825EF2D9C909FFFAFF3',
     'requestId': '1E3CDDE80EE37825EF2D9C909FFFAFF3',
     'response': {'connectionId': 26,
  
  </code></pre>
  <p>They don't contain all the information I can see from the web inspector in chrome.</p>
  <p>I want to get the whole response and request headers and the actual response. Is this the correct way? Is there another better way which doesn't use selenium and only requests instead?</p>