Take the 2-minute tour ×

After much reading here on Stackoverflow as well as the web I'm still struggling with getting things to work.

My challenge: to get access to a restricted part of a website for which I'm a member using Python and urllib2.

From what I've read the code should be like this:

mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()

url = 'http://www.domain.com'

mgr.add_password(None, url, 'username', 'password')
handler = urllib2.HTTPBasicAuthHandler(mgr)
opener = urllib2.build_opener(handler)


    response = urllib2.urlopen('http://www.domain.com/restrictedpage')
    page = response.read()
    print page.geturl()
except IOError, e:
    print e

The print doesn't print "http://www.domain.com/restrictedpage", but shows "http://www.domain.com/login" so my credentials aren't stored/processed and I'm being redirected.

How can I get this to work? I've been trying for days and keep hitting the same dead ends. I've tried all the examples I could find to no avail.

My main question is: what's needed to authenticate to a website using Python and urllib2? Quick question: what am I doing wrong?

share|improve this question
does the site you basic Http Authentication?? many sites do not, you will be required to find out what variables are posted to what URL, and do everything manually. –  dm03514 Mar 3 '12 at 19:55
How can I find out? I know about the 401 headers, but I can't seem to capture them. Do you have an example on how to do it manually? –  Roland Mar 3 '12 at 20:03

2 Answers 2

For HTTP Basic Auth you can refer this : http://www.voidspace.org.uk/python/articles/authentication.shtml

share|improve this answer
Thanks for the link, but unfortunately that didn't help. I'm now looking at Requests as it's a lib aimed at making these things more easy. –  Roland Mar 5 '12 at 9:39

Check first manually what is really happening when you are successfully authenticated (instructions with Chrome):

  • Open develper tools in Chrome (Ctrl + Shift + I)
  • Click Network tab
  • Go and do the authentication manually (go the the page, type user + passwd + submit)
  • check the POST method in the Network tab of the developer tools
  • check the Request Headers, Query String Parameters and Form Data. There you find all the information needed what you need to have in your own POST.

Then install "Advanced Rest Client (ARC)" Chrome extension

Use the ARC to construct a valid POST for authentication.

Now you know what to have in your headers and form data. Here's a sample code using Requests that worked for me for one particular site:

import requests

USERNAME = 'user' # put correct usename here
PASSWORD = 'password' # put correct password here

LOGINURL = 'https://login.example.com/'
DATAURL = 'https://data.example.com/secure_data.html'

session = requests.session()

req_headers = {

formdata = {
    'UserName': USERNAME,
    'Password': PASSWORD,
    'LoginButton' : 'Login'

# Authenticate
r = session.post(LOGINURL, data=formdata, headers=req_headers, allow_redirects=False)
print r.headers
print r.status_code
print r.text

# Read data
r2 = session.get(DATAURL)
print "___________DATA____________"
print r2.headers
print r2.status_code
print r2.text
share|improve this answer

Your Answer


