title: Errno 11001 getaddrinfo failed when looping over urls
body: <p>I want to extract a list of documents from some pages.</p>
  
  <p>I have problem when I try to loop over a list of urls as I keep getting </p>
  
  <pre><code>Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))
  </code></pre>
  
  <p>Appreciate clarification on why is this the case. </p>
  
  <p>If I run against one url page, there should be no problem though.</p>
  
  <p>I have a separate code which uses Selenium/Webdriver however the problem with using Selenium is the download behavior for different file types.</p>
  
  <p>For example, if a url brings you to a pdf file, it will open up a new page showing the full pdf file. The behavior is different if the url is linked to an Excel file.</p>
  
  <p>More details can be found here <a href="https://stackoverflow.com/questions/57566692/how-do-i-control-selenium-pdf-and-excel-files-download-behavior/57568732">How do I control Selenium PDF and Excel files download behavior?</a></p>
  
  <p>I ended up with the suggested code below which while it may not use Selenium, it does the job at getting all the files.</p>
  
  <p>Thanks!</p>
  
  <pre><code>import requests
  from bs4 import BeautifulSoup
  import re
  import pandas as pd
  import os
  
  doc_urls = ['http://www.ha.org.hk/haho/ho/bssd/18G042Pc.htm'
  'http://www.ha.org.hk/haho/ho/bssd/HKWCT03018A2Pa.htm',
  'http://www.ha.org.hk/haho/ho/bssd/19D070Pa.htm',
  'http://www.ha.org.hk/haho/ho/bssd/NTECT6AQ011Pa.htm',
  'http://www.ha.org.hk/haho/ho/bssd/T18G052Pa.htm',
  ]
  
  base_url = "http://www.ha.org.hk"
  
  
  for doc in doc_urls:
      with requests.Session() as session:
          r = session.get(doc)
          # get all documents links
          docs = BeautifulSoup(r.text, "html.parser").select("a[href]")
          print('Visiting:',doc)
          for doc in docs:
              href = doc.attrs["href"]
              name = doc.text
              print(f"&gt;&gt;&gt; Downloading file name: {name}, href: {href}")
              # open document page
              r = session.get(href)
              # get file path
              # check for attibute, if not, file doesn't exist: contact admin. but how to contact the hospital admin?
              if hasattr(re.search("(?&lt;=window.open\\(')(.*)(?=',)", r.text), 'group'):
                  file_path = re.search("(?&lt;=window.open\\(')(.*)(?=',)", r.text).group(0)
                  print(file_path)
                  file_name = file_path.split("/")[-1]
                  # get file and save
                  r = session.get(f"{base_url}/{file_path}")
                  with open('C:\\Users\\tender_documents\\'+ today_yyMMddhh + '\\' + file_name, 'wb') as f:
                      f.write(r.content)
              else:
                  print(f"&gt;&gt;&gt; File name: {name}, href: {href}", " is missing")
                  continue
  
  </code></pre>