title: Python web scraping requests follow redirect
body: <p>I'm trying to scrape a web site with the requests module. </p>
  
  <p>Using chrome and inspect elements, I go to the url, fill in a form and click the continue button. Chrome's inspect elements (network documents) shows what chrome sent with post. It also shows multiple cookies.  The site redirects to a url with among other things a session ID.</p>
  
  <p>To simulate this, I try using requests. I take the form data from inspect elements and reformat it to a dictionary. I use requests.session to include the cookies.</p>
  
  <pre><code>    import requests
  
  form_data = 'currentCalForm=dep&amp;currentCodeForm=&amp;tripType=oneWay&amp;searchCategory=award&amp;originAirport=JFK&amp;flightParams.flightDateParams.travelMonth=5&amp;flightParams.flightDateParams.travelDay=14&amp;flightParams.flightDateParams.searchTime=040001&amp;destinationAirport=LHR&amp;returnDate.travelMonth=-1000&amp;returnDate.travelDay=-1000&amp;adultPassengerCount=2&amp;adultPassengerCount=1&amp;serviceclass=coach&amp;searchTypeMode=matrix&amp;awardDatesFlexible=true&amp;originAlternateAirportDistance=0&amp;destinationAlternateAirportDistance=0&amp;discountCode=&amp;flightSearch=award&amp;dateChanged=false&amp;fromSearchPage=true&amp;advancedSearchOpened=false&amp;numberOfFlightsToDisplay=10&amp;searchCategory=&amp;aairpassSearchType=false&amp;moreOptionsIndicator=oneWay&amp;seniorPassengerCount=0&amp;youngAdultPassengerCount=0&amp;childPassengerCount=0&amp;infantPassengerCount=0&amp;passengerCount=2'.split('&amp;')
  
  payload = {}
  for item in form_data:
      key, value = item.split('=')
      if value:
          payload[key] = value
  
  with requests.session() as s:    
      r = s.post('https://www.aa.com/homePage.do', params = payload, allow_redirects=True)
      print r.headers
      print r.history
      print r.url
      print r.status_code
      with open('x.htm', 'wb') as f:
          f.write(r.text.encode('utf8'))
  </code></pre>
  
  <p>requests, however, does not appear to follow the redirect. history is empty and the url appears to be the data I sent rather than  what the site returned. x.htm shows a web page, but does not contain the info I expected.</p>
  
  <p>From <a href="http://docs.python-requests.org/en/latest/user/quickstart/#redirection-and-history" rel="nofollow">http://docs.python-requests.org/en/latest/user/quickstart/#redirection-and-history</a> I expected r.url to contain the redirected url and r.history to contain an http response code.</p>
  
  <p>What am I doing wrong?</p>