title: HTML Purifier selectively eating special characters
body: <p>Using PHP against a UTF-8 compliant database. Here's how input goes in.</p>
  
  <ol>
  <li>user types input into textarea</li>
  <li>textarea encoded with javascript escape()</li>
  <li>passed via HTTP post</li>
  <li>decoded with PHP rawurldecode()</li>
  <li>passed through HTMLPurifier with default settings</li>
  <li>escaped for MySQL and stored in database</li>
  </ol>
  
  <p>And it comes out in the usual way and I run unescape() on page load. This is to allow people to, say, copy and paste directly from a word document and have the smart quotes show up.</p>
  
  <p>But HTMLPurifier seems to be clobbering non-UTF-8 special characters, ones that escape() to a simple % expression, like Ö, which escapes to %D6, whereas smartquotes escape to %u2024 or something and go into the database that way. It takes out both the special character and the one immediately following.</p>
  
  <p>I need to change something in this process. Perhaps I need to change multiple things.</p>
  
  <p>What can I do to not get special characters clobbered?</p>