title: How to add a proxy and a header to my web scraping script in Python/BS4?
body: <p>I used BeautifulSoup4 and Python to parse local html files a few times in the past. Now I would like to scrape a website using a proxy. (400 Request needed in total / after 100 request the IP does get blocked)</p>
  
  <p>After slowing down my script with an ordinary 'sleep', I want to use a proxy, but I have never done this before and do need some help here. I tried two methods, with help from Stack Overflow questions:</p>
  
  <p><strong>Method 1</strong><br>
  This method does work with another website, but it doesn't download data. When I 'print' the data I received, it does print "Response [200]". When I try this method with the real website, it does return an error: "Max retries exceeded with url:" I suspect the proxy is not being handled correct. When I try to read the html,  I get the following error.</p>
  
  <blockquote>
    <p>page_html = response.read()<br>
    AttributeError: 'Response' object has no attribute 'read'</p>
  
  <pre><code>response = requests.get(URL, proxies=PROXY, headers=HEADER)
  </code></pre>
  </blockquote>
  
  <p><strong>Method 2</strong><br>
  I was able to download another webpage, but I wasn't able to download from the original webpage (which blocked me). I assume there is a mistake with the script and the proxy isn't handled correct. Either the real IP is sent to the website, or I can't connect to the proxy:</p>
  
  <pre><code>response = urllib.request.urlopen(urllib.request.Request(url, None, header, proxy))
  </code></pre>
  
  <p>My script does look like this: </p>
  
  <pre><code>HEADER = {'User-Agent': 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7'}
  URL = "https://www.website.php"
  PROXY = {"https": "https//59.110.7.190:1080"}
  
  #response.close() Is this even necessary
  
  page_html = response.read()  # With Method 1 I also tried response.text which resulted in "str is not callable"
  response.close()
  page_soup = soup(page_html, "html.parser")
  
  adresses = page_soup.findAll("li", {"class":"list-group-item"})
  
  for address in adresses:
      try:
          #parsing the html
      except (TypeError):
          f.write("invalid data" + "\n")
  time.sleep(random.randint(1, 10))
  </code></pre>
  
  <p>The error I usually get is the following:</p>
  
  <blockquote>
    <p>requests.exceptions.ProxyError: HTTPSConnectionPool(host='www.firmendb.de', port=443): Max retries exceeded with url: /[website.php] (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError(': Failed to establish a new connection: [Errno 11001] getaddrinfo failed',)))</p>
    
    <p>Process finished with exit code 1</p>
  </blockquote>
  
  <p>I assume I messed up the proxy part of the script. It did work before I tried to implement it.  Because I have never done this before my main question is, is the proxy part correct?  I got the proxy from the following website: <a href="https://free-proxy-list.net/" rel="nofollow noreferrer">https://free-proxy-list.net/</a></p>
  
  <ul>
  <li>How to choose a proxy from these lists?  </li>
  <li>How to connect to the proxies?  </li>
  <li>Any suggestions on proxy-providers to use?</li>
  <li>Any proposal for my script?</li>
  </ul>