title: How can I filter Emoji characters from my input so I can save in MySQL <5.5?
body: <p>I have a Django app that takes tweet data from Twitter's API and saves it in a MySQL database. As far as I know (I'm still getting my head around the finer points of character encoding) I'm using UTF-8 everywhere, including MySQL encoding and collation, which works fine except when a tweet contains <strong>Emoji</strong> characters, which I understand use a four-byte encoding. Trying to save them produces the following warnings from Django:</p>
  
  <blockquote>
    <p>/home/biggleszx/.virtualenvs/myvirtualenv/lib/python2.6/site-packages/django/db/backends/mysql/base.py:86: Warning: Incorrect string value: '\xF0\x9F\x98\xAD I...' for column 'text' at row 1
      return self.cursor.execute(query, args)</p>
  </blockquote>
  
  <p>I'm using MySQL 5.1, so using <a href="http://dev.mysql.com/doc/refman/5.5/en/charset-unicode-utf8mb4.html" rel="nofollow noreferrer">utf8mb4</a> isn't an option unless I upgrade to 5.5, which I'd rather not just yet (also from what I've read, Django's support for this isn't quite production-ready, though this might no longer be accurate). I've also seen <a href="https://stackoverflow.com/questions/8676005/in-php-how-to-remove-emoji-code">folks</a> advising the use of BLOB instead of TEXT on affected columns, which I'd also rather not do as I figure it would harm performance.</p>
  
  <p>My question is, then, assuming I'm not too bothered about 100% preservation of the tweet contents, is there a way I can filter out all Emoji characters and replace them with a non-multibyte character, such as the venerable <code>WHITE MEDIUM SMALL SQUARE (U+25FD)</code>? I figure this is the easiest way to save that data given my current setup, though if I'm missing another obvious solution, I'd love to hear it!</p>
  
  <p>FYI, I'm using the stock Python 2.6.5 on Ubuntu 10.04.4 LTS. <code>sys.maxunicode</code> is 1114111, so it's a UCS-4 build.</p>
  
  <p>Thanks for reading.</p>