title: Of scraping data, headless browsers, and Python
body: <p>So i'm a CS student trying to learn web scraping and all the do's and dont's that come along with it. After messing about with iMacros and a few other data scraping 'tools', I turned to Python, a language I was not familiar with at the time. I learned about BeautifulSoup and urllib2, and blundered my way through learning it through stackoverflow and a few other forums.</p>
  
  <p>Now, using the knowledge ive gained so far, I can scrape most static web pages. However, we all know that the era of static pages is over, as JS reigns supreme on even mediocre websites now. </p>
  
  <p>I would like someone to please guide me in the right direction here. I want to learn a method to load Javascript-laden webpages, load all the content, and then somehow get this data into the BeautifulSoup function. Urllib2 sucks at that. I would also like the ability to fill in forms and navigate through button clicks.</p>
  
  <p>Mostly the websites im interested in consist of a long list of results that load as you scroll down. Loading them all and then downloading the page doesnt seem to help(Dont know why that is). I'm using Windows 7, and have Python 2.7.5 installed. </p>
  
  <p>I've been told that headless browsers such as zombie or Ghost would help me, but I really dont know much about those. I tried using libraries such as mechanize but they dont really cater for what I need, i.e, loading the results, fetching the webpage, and feeding into BS4.</p>
  
  <p>Bearing in mind my minimal knowledge of Python, could anyone help me out here?</p>
  
  <p>Thanks</p>