title: Python: urllib2.HTTPError: HTTP Error 401: authenticationrequired
body: <p>I was trying to get a web page, but got into this problem. I've looked up some references, and this is what I've done so far:</p>
  
  <pre class="lang-py prettyprint-override"><code>import sys
  import urllib2
  from bs4 import BeautifulSoup
  
  user = 'myuserID'
  password = "mypassword"
  
  ip = sys.argv[1]
  url = "http://www.websites.com/" + ip
  
  passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
  passman.add_password(None, url, user, password)
  handler = urllib2.HTTPBasicAuthHandler(passman)
  opener = urllib2.build_opener(handler)
  urllib2.install_opener(opener)
  
  header = {
      'Connection' : 'keep-alive',
      'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0',
      'Accept-Language' : 'en-US,en;q=0.5',
      'Accept-Encoding' : 'gzip, deflate'
      }
  html = urllib2.urlopen(urllib2.Request(url, None, header))
  soup = BeautifulSoup(html, 'html.parser')
  # some if else function afterwards #
  </code></pre>
  
  <p>When I try to run the script, it shows this kind of error:
  </p>
  
  <pre><code>python checker.py 8.8.8.8
  Traceback (most recent call last):
    File "checker.py", line 34, in &lt;module&gt;
      html = urllib2.urlopen(urllib2.Request(url, None, header))
    File "C:\Python27\lib\urllib2.py", line 154, in urlopen
      return opener.open(url, data, timeout)
    File "C:\Python27\lib\urllib2.py", line 437, in open
      response = meth(req, response)
    File "C:\Python27\lib\urllib2.py", line 550, in http_response
      'http', request, response, code, msg, hdrs)
    File "C:\Python27\lib\urllib2.py", line 469, in error
      result = self._call_chain(*args)
    File "C:\Python27\lib\urllib2.py", line 409, in _call_chain
      result = func(*args)
    File "C:\Python27\lib\urllib2.py", line 656, in http_error_302
      return self.parent.open(new, timeout=req.timeout)
    File "C:\Python27\lib\urllib2.py", line 437, in open
      response = meth(req, response)
    File "C:\Python27\lib\urllib2.py", line 550, in http_response
      'http', request, response, code, msg, hdrs)
    File "C:\Python27\lib\urllib2.py", line 475, in error
      return self._call_chain(*args)
    File "C:\Python27\lib\urllib2.py", line 409, in _call_chain
      result = func(*args)
    File "C:\Python27\lib\urllib2.py", line 558, in http_error_default
      raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
  urllib2.HTTPError: HTTP Error 401: authenticationrequired
  </code></pre>
  
  <p>But if I opened the page or other web page, and manually enter my credential, this script works fine after that. Am I missing something?</p>
  
  <p>Just to add, my current network are using McAfee web gateway device. So sometimes we need to enter our credential to proceed browsing the net. Our user/pass are integrated with Active Directory. Is that may cause the issue?</p>