title: Trouble with strings with <U+0092> Unicode characters
body: <p>I have a very large dataset (70k rows, 2600 columns, CSV format) that I have created by web scraping. Unfortunately, doing the pre-processing, processing etc. at some point some problematic characters have become encoded in an odd way and I have problems dealing with them.</p>
  
  <p>I have strings like the following:</p>
  
  <pre><code>x = "but it doesn&lt;U+0092&gt;t matter"
  </code></pre>
  
  <p><a href="http://www.fileformat.info/info/unicode/char/92/index.htm" rel="noreferrer">Looking up the code</a>, we can see that it should be the character <code>’</code>, which actually should be <code>'</code> (the data are user-generated so may contain all kinds of odd characters). Although from looking that character, it seems that others also have problems with it (<a href="https://stackoverflow.com/questions/8571648/how-do-i-get-rid-of-this-unicode-character">1</a>, <a href="https://stackoverflow.com/questions/11030851/jquery-parsejson-u0092-character-is-not-parsed">2</a>, <a href="https://github.com/tenderlove/psych/issues/271" rel="noreferrer">3</a>). It's labelled a control character, not sure what that is, but perhaps that's why it's so hard to deal with.</p>
  
  <p>Most of the other questions about Unicode in R concern Unicode in the format like this <code>\u0092</code>.</p>
  
  <h2>Just use <code>Encoding()</code></h2>
  
  <p>Let's try:</p>
  
  <pre><code>#&gt; x = "but it doesn&lt;U+0092&gt;t matter"
  #&gt; Encoding(x)
  #[1] "unknown"
  #&gt; Encoding(x) = "UTF-8"
  #&gt; Encoding(x)
  #[1] "unknown"
  #&gt; x
  #[1] "but it doesn&lt;U+0092&gt;t matter"
  </code></pre>
  
  <p>So this does not seem to do anything.</p>
  
  <h2>Use the hack functions from these previous questions</h2>
  
  <p>There are a few prior questions that concern this Unicode format and try to convert them:</p>
  
  <ul>
  <li><a href="https://stackoverflow.com/questions/33866570/display-unicode-in-r">Display unicode in R</a></li>
  <li><a href="https://stackoverflow.com/questions/28248457/gsub-in-r-with-unicode-replacement-give-different-results-under-windows-compared?rq=1">gsub in R with unicode replacement give different results under Windows compared with Unix?</a></li>
  </ul>
  
  <p>Oddly, the example they give work, but mine doesn't.</p>
  
  <pre><code>#&gt; test.string &lt;- "This is a &lt;U+03B1&gt; &lt;U+03B2&gt; &lt;U+03B2&gt; &lt;U+03B3&gt; test &lt;U+03B4&gt; string."
  #&gt; Encoding(test.string)
  #[1] "unknown"
  #&gt; to_true_unicode(test.string)
  #[1] "This is a α β β γ test δ string."
  </code></pre>
  
  <p>But:</p>
  
  <pre><code>#&gt; x2 = to_true_unicode(x)
  #&gt; x2
  #[1] "but it doesn\u0092t matter"
  #&gt; cat(x2)
  #but it doesnt matter
  #&gt; Encoding(x2)
  #[1] "UTF-8"
  </code></pre>
  
  <p>So, it managed to convert to the <code>\u</code> format from the &lt;<code>U+....&gt;</code> format, and using <code>cat()</code> prints the character without that symbol (or a bugged symbol on SO).</p>
  
  <h2>Just search and replace them manually</h2>
  
  <p>I only have a limited number of these problems, so I could perhaps just use search-replace to solve it. However:</p>
  
  <pre><code>#&gt; #base-r
  #&gt; gsub(x = x, pattern = "&lt;U+0092&gt;", replacement = "'")
  #[1] "but it doesn&lt;U+0092&gt;t matter"
  #&gt; #stringr/stringi
  #&gt; library(stringr)
  #&gt; str_replace(x, pattern = "&lt;U+0092&gt;", "'")
  #[1] "but it doesn&lt;U+0092&gt;t matter"
  </code></pre>
  
  <p>So replacement does not seem to work, but it does work on the <code>\u</code> versions:</p>
  
  <pre><code>#&gt; #base-r
  #&gt; gsub(x = x2, pattern = "\u0092", replacement = "'")
  #[1] "but it doesn't matter"
  #&gt; #stringr/stringi
  #&gt; library(stringr)
  #&gt; str_replace(x2, pattern = "\u0092", "'")
  #[1] "but it doesn't matter"
  </code></pre>
  
  <p>So, this suggests a working method: 1) convert <code>&lt;U+&gt;</code> format to <code>\u</code> format, then use search-replace.</p>
  
  <h2>Unescape with <code>stringi::stri_unescape_unicode()</code></h2>
  
  <p>Does not seem to work with either version:</p>
  
  <pre><code>#&gt; stringi::stri_unescape_unicode(x)
  #[1] "but it doesn&lt;U+0092&gt;t matter"
  #&gt; stringi::stri_unescape_unicode(x2)
  #[1] "but it doesn\u0092t matter"
  </code></pre>
  
  <p>Is there some generally applicable way to deal with problems like this?</p>
  
  <h2>My setup</h2>
  
  <p>My sessionInfo is:</p>
  
  <pre><code>&gt; sessionInfo()
  R version 3.2.3 (2015-12-10)
  Platform: x86_64-w64-mingw32/x64 (64-bit)
  Running under: Windows &gt;= 8 x64 (build 9200)
  
  locale:
  [1] LC_COLLATE=Danish_Denmark.1252  LC_CTYPE=Danish_Denmark.1252    LC_MONETARY=Danish_Denmark.1252
  [4] LC_NUMERIC=C                    LC_TIME=Danish_Denmark.1252    
  
  attached base packages:
  [1] stats     graphics  grDevices utils     datasets  methods   base     
  
  other attached packages:
  [1] stringr_1.0.0
  
  loaded via a namespace (and not attached):
  [1] magrittr_1.5  tools_3.2.3   stringi_1.0-1
  </code></pre>
  
  <p>Running R via RStudio (0.99.893, preview) on Windows 8.1, 64-bit. Keyboard and time-units are Danish, but everything else is in English.</p>