title: Python strategy for extracting text from malformed html pages
body: <p>I'm trying to extract text from arbitrary html pages. Some of the pages (which I have no control over) have malformed html or scripts which make this difficult. Also I'm on a shared hosting environment, so I can install any python lib, but I can't just install anything I want on the server. </p>
  
  <p>pyparsing and html2text.py also did not seem to work for malformed html pages. </p>
  
  <p>Example URL is <a href="http://apnews.myway.com/article/20091015/D9BB7CGG1.html" rel="nofollow noreferrer">http://apnews.myway.com/article/20091015/D9BB7CGG1.html</a></p>
  
  <p>My current implementation is approximately the following:</p>
  
  <pre><code># Try using BeautifulSoup 3.0.7a
  soup = BeautifulSoup.BeautifulSoup(s) 
  comments = soup.findAll(text=lambda text:isinstance(text,Comment))
  [comment.extract() for comment in comments]
  c=soup.findAll('script')
  for i in c:
      i.extract()    
  body = bsoup.body(text=True)
  text = ''.join(body) 
  # if BeautifulSoup  can't handle it, 
  # alter html by trying to find 1st instance of  "&lt;body" and replace everything prior to that, with "&lt;html&gt;&lt;head&gt;&lt;/head&gt;"
  # try beautifulsoup again with new html 
  </code></pre>
  
  <p>if beautifulsoup  still does not work, then I resort to using a heuristic of looking at the 1st char, last char (to see if they looks like its a code line # &lt; ; and taking a sample of the line and then check if the tokens are english words, or numbers. If to few of the tokens are words or numbers, then I guess that the line is code. </p>
  
  <p>I could use machine learning to inspect each line, but that seems a little expensive and I would probably have to train it (since I don't know that much about unsupervised learning machines), and of course write it as well. </p>
  
  <p>Any advice, tools, strategies would be most welcome. Also I realize that the latter part of that is rather messy since if I get a line that is determine to contain code, I currently throw away the entire line, even if there is some small amount of actual English text in the line. </p>