title: Python 3 UnicodeDecodeError - How do I debug UnicodeDecodeError?
body: <p>I have a text file which the publisher (the US Securities Exchange Commission) asserts is encoded in UTF-8 (<a href="https://www.sec.gov/files/aqfs.pdf" rel="nofollow noreferrer">https://www.sec.gov/files/aqfs.pdf</a>, section 4). I'm processing the lines with the following code:</p>
  <pre><code>def tags(filename):
      &quot;&quot;&quot;Yield Tag instances from tag.txt.&quot;&quot;&quot;
      with codecs.open(filename, 'r', encoding='utf-8', errors='strict') as f:
          fields = f.readline().strip().split('\t')
          for line in f.readlines():
              yield process_tag_record(fields, line)
  </code></pre>
  <p>I receive the following error:</p>
  <pre class="lang-none prettyprint-override"><code>Traceback (most recent call last):
    File &quot;/home/randm/Projects/finance/secxbrl.py&quot;, line 151, in &lt;module&gt;
      main()
    File &quot;/home/randm/Projects/finance/secxbrl.py&quot;, line 143, in main
      all_tags = list(tags(&quot;tag.txt&quot;))
    File &quot;/home/randm/Projects/finance/secxbrl.py&quot;, line 109, in tags
      content = f.read()
    File &quot;/home/randm/Libraries/anaconda3/lib/python3.6/codecs.py&quot;, line 698, in read
      return self.reader.read(size)
    File &quot;/home/randm/Libraries/anaconda3/lib/python3.6/codecs.py&quot;, line 501, in read
      newchars, decodedbytes = self.decode(data, self.errors)
  UnicodeDecodeError: 'utf-8' codec can't decode byte 0xad in position 3583587: invalid start byte
  </code></pre>
  <p>Given that I probably can't go back to the SEC and tell them they have files that don't seem to be encoded in UTF-8, how should I debug and catch this error?</p>
  <p><strong>What have I tried</strong></p>
  <p>I did a hexdump of the file and found that the offending text was the text &quot;SUPPLEMENTAL DISCLOSURE OF NON�CASH INVESTING&quot;. If I decode the offending byte as a hex code point (i.e. &quot;U+00AD&quot;), it makes sense in context as it is the soft hyphen. But the following does not seem to work:</p>
  <pre class="lang-none prettyprint-override"><code>Python 3.5.2 (default, Nov 17 2016, 17:05:23)
  [GCC 5.4.0 20160609] on linux
  Type &quot;help&quot;, &quot;copyright&quot;, &quot;credits&quot; or &quot;license&quot; for more information.
  &gt;&gt;&gt; b&quot;\x41&quot;.decode(&quot;utf-8&quot;)
  'A'
  &gt;&gt;&gt; b&quot;\xad&quot;.decode(&quot;utf-8&quot;)
  Traceback (most recent call last):
    File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;
  UnicodeDecodeError: 'utf-8' codec cant decode byte 0xad in position 0: invalid start byte
  &gt;&gt;&gt; b&quot;\xc2ad&quot;.decode(&quot;utf-8&quot;)
  Traceback (most recent call last):
    File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;
  UnicodeDecodeError: 'utf-8' codec cant decode byte 0xc2 in position 0: invalid continuation byte
  </code></pre>
  <p>I've used <code>errors='replace'</code>, which seems to pass. But I'd like to understand what will happen if I try to insert that into a database.</p>
  <p>Hexdump:</p>
  <pre class="lang-none prettyprint-override"><code>0036ae40  31 09 09 09 09 53 55 50  50 4c 45 4d 45 4e 54 41  |1....SUPPLEMENTA|
  0036ae50  4c 20 44 49 53 43 4c 4f  53 55 52 45 20 4f 46 20  |L DISCLOSURE OF |
  0036ae60  4e 4f 4e ad 43 41 53 48  20 49 4e 56 45 53 54 49  |NON.CASH INVESTI|
  0036ae70  4e 47 20 41 4e 44 20 46  49 4e 41 4e 43 49 4e 47  |NG AND FINANCING|
  0036ae80  20 41 43 54 49 56 49 54  49 45 53 3a 09 0a 50 72  | ACTIVITIES:..Pr|
  </code></pre>