title: Web crawling - Identifying the main content on a webpage in Java
body: <p>Working on a little hobby project, scraping a few webpages, mostly blogs and news sites that our users post.</p>
  
  <p>We have a little script that simply takes a list of URLs and scrapes the pages pulling in all content, basically the hello world version of using Apache HTTP components... </p>
  
  <p>Anyway, we would like to do some analysis on the text that appears on these pages. But as any URL can appear, in any format, we need some reasonably reliable way to identify the main content on a web page (a few errors here and there are ok). </p>
  
  <p>Is anyone aware of any open source frameworks or little scripts in Java that can be used to Identify the major text content in a web page? </p>
  
  <p>There are a few methods that come to mind like simply looking for the largest DOM element with the most visible text, or comparing the number of chars used to describe some text (i.e. meta-data) vs. actual text data. I was hoping someone may have release something to do this in Java, but if not, as I'm about to go the trouble of doing this, is there a need out there for such a utility?</p>
  
  <p>Cheers!</p>