<p>Apply the <a href="https://github.com/ahupp/python-magic" rel="nofollow noreferrer"><code>python-magic</code> library</a>.</p>
<blockquote>
<p><code>python-magic</code> is a Python interface to the <code>libmagic</code> file type
identification library. <code>libmagic</code> identifies file types by checking
their headers according to a predefined list of file types. This
functionality is exposed to the command line by the Unix command
<code>file</code>.</p>
</blockquote>
<p><em>Commented</em> script (works on Windows 10, Python 3.8.6):</p>
<pre><code># stage #1: read raw data from a url
from urllib.request import urlopen
import gzip
url = &quot;http://export.arxiv.org/e-print/supr-con/9608001&quot;
with urlopen(url) as response:
    rawdata = response.read()

# stage #2: detect raw data type by its signature
print(&quot;file signature&quot;, rawdata[0:2])
import magic
print( magic.from_buffer(rawdata[0:1024]))

# stage #3: decompress raw data and write to a file
data = gzip.decompress(rawdata)
filename = 'test.tex'
file_ = open(filename, 'wb')
file_.write(data)
file_.close()

# stage #4: detect encoding of the data ( == encoding of the written file)
import chardet
print( chardet.detect(data))
</code></pre>
<p><strong>Result</strong>: <code>.\SO\68307124.py</code></p>
<pre><code>file signature b'\x1f\x8b'
gzip compressed data, was &quot;9608001.tex&quot;, last modified: Thu Aug  8 04:57:44 1996, max compression, from Unix
{'encoding': 'ascii', 'confidence': 1.0, 'language': ''}
</code></pre>
How to determine how a (downloaded) byte string is encoded in python?
<p>I am trying to download a file and write it to disk, but somehow I am lost in encoding decoding land.</p>
<pre><code>from urllib.request import urlopen
url = &quot;http://export.arxiv.org/e-print/supr-con/9608001&quot;
with urllib.request.urlopen(url) as response:
    data = response.read()
    filename = 'test.txt'
    file_ = open(filename, 'wb')
    file_.write(data)
    file_.close()
</code></pre>
<p>Here data is a byte string. If I check the file I find a bunch of strange characters. I tried</p>
<pre><code>import chardet
the_encoding = chardet.detect(data)['encoding']
</code></pre>
<p>but this results in None. So I don't really know how the data I downloaded is encoded?</p>
<p>If I just type &quot;http://export.arxiv.org/e-print/supr-con/9608001&quot; into the browser, it downloads a file that I can view with a text editor and it's a perfectly fine .tex file.</p>