title: autogenerate HTTP screen scraping Java code
body: <p>I need to screen scrape some data from a website, because it isn't available via their web service. When I've needed to do this previously, I've written the Java code myself using Apache's HTTP client library to make the relevant HTTP calls to download the data. I figured out the relevant calls I needed to make by clicking through the relevant screens in a browser while using the <a href="http://www.charlesproxy.com/" rel="nofollow noreferrer">Charles web proxy</a> to log the corresponding HTTP calls.</p>
  
  <p>As you can imagine this is a fairly tedious process, and I'm wodering if there's a tool that can actually generate the Java code that corresponds to a browser session. I expect the generated code wouldn't be as pretty as code written manually, but I could always tidy it up afterwards. Does anyone know if such a tool exists? Selenium is one possibility I'm aware of, though I'm not sure if it supports this exact use case.</p>
  
  <p>Thanks,
  Don</p>