title: How to make Beautiful Soup output HTML entities?
body: <p>I'm trying to sanitize and XSS-proof some HTML input from the client. I'm using Python 2.6 with Beautiful Soup. I parse the input, strip all tags and attributes not in a whitelist, and transform the tree back into a string.</p>
  
  <p>However...</p>
  
  <pre><code>&gt;&gt;&gt; unicode(BeautifulSoup('text &lt; text'))
  u'text &lt; text'
  </code></pre>
  
  <p>That doesn't look like valid HTML to me. And with my tag stripper, it opens the way to all sorts of nastiness:</p>
  
  <pre><code>&gt;&gt;&gt; print BeautifulSoup('&lt;&lt;script&gt;&lt;/script&gt;script&gt;alert("xss")&lt;&lt;script&gt;&lt;/script&gt;script&gt;').prettify()
  &lt;
  &lt;script&gt;
  &lt;/script&gt;
  script&gt;alert("xss")&lt;
  &lt;script&gt;
  &lt;/script&gt;
  script&gt;
  </code></pre>
  
  <p>The <code>&lt;script&gt;&lt;/script&gt;</code> pairs will be removed, and what remains is not only an XSS attack, but even valid HTML as well.</p>
  
  <p>The obvious solution is to replace all <code>&lt;</code> characters by <code>&amp;lt;</code> that, after parsing, are found not to belong to a tag (and similar for <code>&gt;&amp;'"</code>). But the <a href="http://www.crummy.com/software/BeautifulSoup/documentation.html" rel="noreferrer">Beautiful Soup documentation</a> only mentions the parsing of entities, not the producing of them. Of course I can run a replace over all <code>NavigableString</code> nodes, but since I might miss something, I'd rather let some tried and tested code do the work.</p>
  
  <p><strong>Why doesn't Beautiful Soup escape <code>&lt;</code> (and other magic characters) by default, and how do I make it do that?</strong></p>
  
  <hr>
  
  <p>N.B. I've also looked at <code>lxml.html.clean</code>. It seems to work on the basis of blacklisting, not whitelisting, so it doesn't seem very safe to me. Tags can be whitelisted, but attributes cannot, and it allows too many attributes for my taste (e.g. <code>tabindex</code>). Also, it gives an <code>AssertionError</code> on the input <code>&lt;SCRIPT SRC=http://ha.ckers.org/xss.js&gt;&lt;/SCRIPT&gt;</code>. Not good.</p>
  
  <p>Suggestions for other ways to clean HTML are also very welcome. I'm hardly the only person in the world trying to do this, yet there seems to be no standard solution.</p>