title: Detect encoding in wrongly encoded UTF-8 text file
body: <p>I have an encoding issue.</p>
  <p>I have millions of text files that I need to parse for a language data science project. Each text file is encoded as UTF-8, but I just found that some of these source files are not encoded properly.</p>
  <p>For example. I have a Chinese text file, that is encoded as UTF-8, but text in the file looks like this:</p>
  <pre><code>Subject: »Ø¸´: ÎÒÉý¼¶µ½
  </code></pre>
  <p>When I use Python to detect the encoding of this Chinese text file:</p>
  <p>Chardet tells me the file is encoded as UTF-8:</p>
  <pre><code>with open(path,'rb') as f:
      data = &quot;&quot;
      data = f.read()
      encoding=chardet.detect(data)['encoding']
  </code></pre>
  <p>UnicodeDammit also tells me the file is encoded as UTF-8:</p>
  <pre><code>with open(path,'rb') as f:
      data = &quot;&quot;
      data = f.read()
      encoding= UnicodeDammit(data).original_encoding
  </code></pre>
  <p>Meanwhile, I know it's not UTF-8, it should be GB2312 Chinese encoding instead. If I open this file in Notepad++, it's detected also as UTF-8 and all Chinese characters show as gibberish. Only if I manually switch encoding in Notepad++ to GB2312 I get the proper text:</p>
  <pre><code>Subject: 禄脴赂麓: 脦脪脡媒录露碌陆
  </code></pre>
  <p>I have a number of files like this, in all kinds of languages.</p>
  <p>Is there a way I can detect encoding in these badly encoded UTF-8 files?</p>
  <p>Example text file can be downloaded here: <a href="https://gofile.io/d/qMcgkt" rel="nofollow noreferrer">https://gofile.io/d/qMcgkt</a></p>