title: fixing invalid XML with Python or Perl: UTF-16 surrogates for emoji encoded in UTF-8
body: <p>I'm trying to do a numerical analysis of my text messages using all the old backups I could gather. Ideally, emoji will be included in the analysis. I'm using a mix of Python and Perl to get things all into one place, and will likely use R once that's done. </p>
  
  <p>However, I've run into trouble with emoji encoding. Some of my backups were created using the SMS Backup and Restore app on Android to pull my texts as a XML file. I started throwing my XML into <a href="https://github.com/t413/SMS-Tools" rel="nofollow noreferrer">this Python SMS module available on github</a> by t413. When the module threw an error in the parser, I put my messages through a validator to see what was up, and the XML wasn't valid due to invalid characters. For an example, part of a text I received that doesn't play well with the XML::Validate module on Perl:</p>
  
  <pre><code>So if we get it out hang out will be short &amp;#55357;&amp;#56852;...
  </code></pre>
  
  <p>I don't know all the particulars of Unicode, but from what I can tell my text messages include HTML escape characters in UTF-8 for the high/low surrogates. Individually they're invalid characters, but together they do encode . (The XML header does specify UTF-8.)</p>
  
  <p>A lot of these texts are already deleted from my phone (some of these backups are almost a year old) so I can't simply pull them again and see if I can fix the formatting like that.</p>
  
  <p>My question: before I start digging into the particulars of Unicode and HTML escape characters and taking the time to write something to fix this myself (I know from <a href="https://stackoverflow.com/questions/35670238/emoji-in-r-utf-8-encoding">this question</a> that there is a formula to use to convert the surrogates, and that there are encode/decode methods for strings in Python, and various bits and pieces to help out with HTML entities), is there any existing module/built-in function in Python or Perl that can help me fix my file's encoding, or at least get me part of the way there? (Or even a Unix/Linux command line tool that I'm missing.)</p>