title: HEAD and GET http request return different status code for the same URL
body: <p>I'm trying to check if some URLs that are stored in my DB are still valid links or not. To achieve this, I am using httplib2 to request the HEAD status in order to avoid downloading the entire content of the page. And I was quite happy with the results.</p>
  
  <p>But then I discovered some cases where the status code returned when using a HEAD request is not similar to the one returned with a GET request.</p>
  
  <p>So, just in case of a bug in the library, I made some tests with different libraries (below is my "requests" lib test):</p>
  
  <pre><code>&gt; import requests    
  
  &gt; rg = requests.get("https://fr.news.yahoo.com/chemin-dames-l-hommage-personnel-pr%C3%A9sident-121005844.html")
  &gt; rh = requests.head("https://fr.news.yahoo.com/chemin-dames-l-hommage-personnel-pr%C3%A9sident-121005844.html")
  
  &gt; print("GET status code:", rg.status_code)
    ('GET status code:', 200)
  
  &gt; print("HEAD status code:", rh.status_code)
    ('HEAD status code:', 404) 
  </code></pre>
  
  <p>But whatever lib I use, I still have a different GET &amp; HEAD status for the same URL.</p>
  
  <p>So, obviously the site maintainer decided to not return an identical status code for both HEAD and GET request... and that seems legit even if not recommended.</p>
  
  <p>Is there a way to avoid this problem and still know if the link is a valid one without having to download the entire content of the almost 2 millions url that I need to verify?</p>
  
  <p>I can double check with a GET request whenever a >400 status code is returned on a HEAD request but that seems like a dirty work to me.</p>