title: python requests.get(url) times out but works in browser (chrome); how can I tailor the request headers for a certain host?
body: <p>I am trying to download a file using the python requests module, my code works for some urls/hosts but I've come across one that does not work.</p>
  <p>Based on other similar questions it may be related to the User-Agent request header, I have tried to remedy by adding the chrome user-agent but the connection still times out for this particular url (it does work for others).</p>
  <p>I have tested opening the url in chrome browser (which works all OK) and inspecting the request headers, but I still can't figure out why my code is failing:</p>
  <pre><code>import requests
  url = 'http://publicdata.landregistry.gov.uk/market-trend-data/house-price-index-data/Indices-2020-03.csv'
  headers = {'Upgrade-Insecure-Requests': '1', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36'}
  
  session = requests.Session()
  session.headers.update(headers)
  response = session.get(url, stream=True)
  # !!! code fails here for this particular url !!!
  
  with open('test.csv', &quot;wb&quot;) as fh:
      for x in response.iter_content(chunk_size=1024):
          if x: fh.write(x)
  </code></pre>
  <p><strong>Update 2020-08-14</strong>
  <em>I have figured out what was wrong; on the instances where the code was working the urls were using https protocol. This url is http protocol, and my proxy settings were not configured for http only https. After providing a http proxy to requests my code did work as written.</em></p>