title: Handling Encoding Errors in a UTF-8 File with Python3
body: <p>I'm trying to use a corpus for training an ML model but I'm running into some encoding errors that where likely caused by someone else's conversion/annotation of the file. I can visually see the errors when opening the file in <code>vim</code> but python doesn't seem to notice them when reading.  The corpus is fairly large so I need to find a way to get python to detect them and hopefully a method to correct them.</p>
  <p>Here's a sample line as viewed in <code>vim</code>...</p>
  <p><code># ::snt That&lt;92&gt;s what we&lt;92&gt;re with&lt;85&gt;You&lt;92&gt;re not sittin&lt;92&gt; there in a back alley and sayin&lt;92&gt; hey what do you say, five bucks?</code></p>
  <p>The &lt;92&gt; should be an apostrophe and the &lt;85&gt; should probably be 3 dots.  There are a number of other values that appear on other lines.  Doing some googling, I'm thinking the original encoding was probably CP1252 but currently the <code>file</code> command under Linux list this file as UTF-8.  I've tried a few ways to open this but no luck...</p>
  <p><code>with open(fn) as f:</code> returns</p>
  <p><code># ::snt Thats what were withYoure not sittin there in a back alley and sayin hey what do you say, five bucks?</code></p>
  <p>which is skipping those tokens and concatenating words, which is a problem.</p>
  <p><code>with open(fn, encoding='CP1252') as f:</code> returns</p>
  <p><code># ::snt ThatA's what weA're withA...YouA're not sittinA' there in a back alley and sayinA' hey what do you say, five bucks?</code></p>
  <p>which is visually inserting &quot;A&quot; for those odd characters.</p>
  <p><code>with io.open(fn, errors='strict')</code> doesn't raise any errors and neither does reading in a byte stream and decoding, so unfortunately at this point I can't even detect the errors much less correct for them.</p>
  <p>Is there a way to read in this large file and detect encoding errors within it.  Even better, is there a way to correct them?</p>