title: Extracting pure content / text from HTML Pages by excluding navigation and chrome content
body: <p>I am crawling news websites and want to extract News Title, News Abstract (First Paragraph), etc</p>
  
  <p>I plugged into the webkit parser code to easily navigate webpage as a tree. To eliminate navigation and other non news content I take the text version of the article (minus the html tags, webkit provides api for the same). Then I run the diff algorithm comparing various article's text from same website this results in similar text being eliminated. This gives me content minus the common navigation content etc.</p>
  
  <p>Despite the above approach I am still getting quite some junk in my final text. This results in incorrect News Abstract being extracted. The error rate is 5 in 10 article i.e. 50%. Error as in   </p>
  
  <p>Can you </p>
  
  <ol>
  <li><p>Suggest an alternative strategy for extraction of pure content,</p></li>
  <li><p>Would/Can learning Natural Language rocessing help in extracting correct abstract from these articles ? </p></li>
  <li><p>How would you approach the above problem ?.</p></li>
  <li><p>Are these any research papers on the same ?.</p></li>
  </ol>
  
  <p>Regards</p>
  
  <p>Ankur Gupta</p>