title: Downloading web content from bash/curl to python
body: <p>I wrote a bash script to download content of some webpage. To make it work I need to catch a cookie, then send some special data request and then i am able to get to proper link to download its content.
  My script look like this:</p>
  
  <pre><code>#!/bin/bash  
  for ((i=1;i&lt;=$NB;++i)); do  
  cookie=`curl -I "http://example.com/index.php" | grep Set-Cookie: | awk '{print $2}' |         cut -d ';' -f 1\`  # cath a cookie  
  curl -b $cookie --data "a_piece_of_data" "http://example.com/index.php"  
  curl -b $cookie "http://example.com/proper_link_$i" &amp;&gt; $i.html  
  sleep 3  
  done
  </code></pre>
  
  <p>As i need to parse it later on to get rid of all html/xhtml tags ( just extract pure text ) and then convert it to XML i find out Python and its lib's will be just perfect to do that.<br>So I ask you for hints how to rewrite my script to python?<br>
  Here is what i come up with so far, but its still is far away from my bash example :</p>
  
  <pre><code>import mechanize
  import urllib2
  import BeautifulSoup
  import lxml
  
  request = mechanize.Request("http://example.com/index.php")
  response = mechanize.urlopen(request)
  cj = mechanize.CookieJar()
  cj.extract_cookies(response, request)
  print cj
  </code></pre>
  
  <p>Any help/hints appreciated.</p>