title: Is it safe to use urllib to determine if a site is unresponsive?
body: <p>I was assigned a task of taking a list of 25k websites and removing the ones which are down/unresponsive. I figured the easiest way to do this:</p>
  <pre><code>with website in websites:
      try:
          req = Request(test, headers={&quot;User-Agent&quot;: &quot;Mozilla/5.0 (Linux i686)&quot;})
          with contextlib.closing(urlopen(req)) as response:
              new_list.add(response.geturl())   
      except:
          print(website + &quot; removed&quot;) 
  </code></pre>
  <p>How safe is this? The list contains many spam sites, and I don't think it's a good idea to send my IP address to them.</p>
  <p>Is there an safer option I can take?</p>