<p dir="auto">I'm trying to read numpy arrays from a bz2 file using <code class="notranslate">fromfile</code>, but I don't get the expected results.<br>
More specifically I can't round-trip a simple record array through <code class="notranslate">tofile</code>, bz2 compression, <code class="notranslate">bz2.open</code> and <code class="notranslate">fromfile</code>.<br>
The <code class="notranslate">bz2.open</code> function claims that a file-object is returned: <a href="https://docs.python.org/3/library/bz2.html#bz2.open" rel="nofollow">https://docs.python.org/3/library/bz2.html#bz2.open</a><br>
and <code class="notranslate">fromfile</code> claims it can handle file-objects: <a href="https://numpy.org/doc/stable/reference/generated/numpy.fromfile.html?highlight=fromfile#numpy.fromfile" rel="nofollow">https://numpy.org/doc/stable/reference/generated/numpy.fromfile.html?highlight=fromfile#numpy.fromfile</a></p>
<p dir="auto">but the results from my experiments shows that <code class="notranslate">fromfile</code> is accessing the compressed data, not the uncompressed data.</p>
<p dir="auto">I understand that <code class="notranslate">fromfile</code> is a C function and thus I'm suspecting it isn't using the <code class="notranslate">BZ2File</code> methods for seeking, telling, reading etc...</p>
<h3 dir="auto">Reproducing code example:</h3>

<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import numpy as np
import bz2

_BASIC_INFO_TYPE = np.dtype([(&quot;hblock_number&quot;, &quot;u1&quot;),
                             (&quot;blocklength&quot;, &quot;&lt;u2&quot;),
                             (&quot;total_number_of_hblocks&quot;, &quot;&lt;u2&quot;),
                             (&quot;byte_order&quot;, &quot;u1&quot;),
                             (&quot;satellite&quot;, &quot;S16&quot;),
                             ])

arr = np.array([(1, 282, 11, 0, b'Himawari-8')],
                dtype=_BASIC_INFO_TYPE)


arr.tofile('arr.bin')

# zip the resulting file:
with bz2.open('arr.bin.bz2', mode='wb') as zfd:
    with open('arr.bin', mode='rb') as fd:
        zfd.write(fd.read())


with bz2.open('arr.bin.bz2') as fd:
    arr = np.fromfile(fd, dtype=_BASIC_INFO_TYPE, count=1)
print(arr)
# erraneous (compressed data read directly, would return the same if doing `with open('arr.bin.bz2') ...`)
# [(66, 26714, 12601, 65, b'Y&amp;SY\x85z\xe4}\x00\x00\x03\xfd\x80p\x08')]

with bz2.open('arr.bin.bz2') as fd:
    arr = np.frombuffer(fd.read(_BASIC_INFO_TYPE.itemsize), dtype=_BASIC_INFO_TYPE, count=1)
print(arr)
# correct
# [(1, 282, 11, 0, b'Himawari-8')]"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-k">import</span> <span class="pl-s1">bz2</span>

<span class="pl-s1">_BASIC_INFO_TYPE</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">dtype</span>([(<span class="pl-s">"hblock_number"</span>, <span class="pl-s">"u1"</span>),
                             (<span class="pl-s">"blocklength"</span>, <span class="pl-s">"&lt;u2"</span>),
                             (<span class="pl-s">"total_number_of_hblocks"</span>, <span class="pl-s">"&lt;u2"</span>),
                             (<span class="pl-s">"byte_order"</span>, <span class="pl-s">"u1"</span>),
                             (<span class="pl-s">"satellite"</span>, <span class="pl-s">"S16"</span>),
                             ])

<span class="pl-s1">arr</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">array</span>([(<span class="pl-c1">1</span>, <span class="pl-c1">282</span>, <span class="pl-c1">11</span>, <span class="pl-c1">0</span>, <span class="pl-s">b'Himawari-8'</span>)],
                <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">_BASIC_INFO_TYPE</span>)


<span class="pl-s1">arr</span>.<span class="pl-en">tofile</span>(<span class="pl-s">'arr.bin'</span>)

<span class="pl-c"># zip the resulting file:</span>
<span class="pl-k">with</span> <span class="pl-s1">bz2</span>.<span class="pl-en">open</span>(<span class="pl-s">'arr.bin.bz2'</span>, <span class="pl-s1">mode</span><span class="pl-c1">=</span><span class="pl-s">'wb'</span>) <span class="pl-k">as</span> <span class="pl-s1">zfd</span>:
    <span class="pl-k">with</span> <span class="pl-en">open</span>(<span class="pl-s">'arr.bin'</span>, <span class="pl-s1">mode</span><span class="pl-c1">=</span><span class="pl-s">'rb'</span>) <span class="pl-k">as</span> <span class="pl-s1">fd</span>:
        <span class="pl-s1">zfd</span>.<span class="pl-en">write</span>(<span class="pl-s1">fd</span>.<span class="pl-en">read</span>())


<span class="pl-k">with</span> <span class="pl-s1">bz2</span>.<span class="pl-en">open</span>(<span class="pl-s">'arr.bin.bz2'</span>) <span class="pl-k">as</span> <span class="pl-s1">fd</span>:
    <span class="pl-s1">arr</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">fromfile</span>(<span class="pl-s1">fd</span>, <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">_BASIC_INFO_TYPE</span>, <span class="pl-s1">count</span><span class="pl-c1">=</span><span class="pl-c1">1</span>)
<span class="pl-en">print</span>(<span class="pl-s1">arr</span>)
<span class="pl-c"># erraneous (compressed data read directly, would return the same if doing `with open('arr.bin.bz2') ...`)</span>
<span class="pl-c"># [(66, 26714, 12601, 65, b'Y&amp;SY\x85z\xe4}\x00\x00\x03\xfd\x80p\x08')]</span>

<span class="pl-k">with</span> <span class="pl-s1">bz2</span>.<span class="pl-en">open</span>(<span class="pl-s">'arr.bin.bz2'</span>) <span class="pl-k">as</span> <span class="pl-s1">fd</span>:
    <span class="pl-s1">arr</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">frombuffer</span>(<span class="pl-s1">fd</span>.<span class="pl-en">read</span>(<span class="pl-s1">_BASIC_INFO_TYPE</span>.<span class="pl-s1">itemsize</span>), <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">_BASIC_INFO_TYPE</span>, <span class="pl-s1">count</span><span class="pl-c1">=</span><span class="pl-c1">1</span>)
<span class="pl-en">print</span>(<span class="pl-s1">arr</span>)
<span class="pl-c"># correct</span>
<span class="pl-c"># [(1, 282, 11, 0, b'Himawari-8')]</span></pre></div>
<h3 dir="auto">Error message:</h3>
<p dir="auto">No error message, just erraneous data.</p>
<p dir="auto">If my assumption is correct that numpy's <code class="notranslate">fromfile</code> shortcuts the fileobject's <code class="notranslate">seek</code>, <code class="notranslate">read</code>, etc, then it will prevent the use of other file objects, like the ones returned by fsspec.</p>
<p dir="auto">Note that I suspect the same problem is happening with <code class="notranslate">np.memmap</code>.</p>
<h3 dir="auto">NumPy/Python version information:</h3>

<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="In [1]: import sys, numpy; print(numpy.__version__, sys.version)
1.20.1 3.8.6 | packaged by conda-forge | (default, Jan 25 2021, 23:21:18) 
[GCC 9.3.0]"><pre class="notranslate"><code class="notranslate">In [1]: import sys, numpy; print(numpy.__version__, sys.version)
1.20.1 3.8.6 | packaged by conda-forge | (default, Jan 25 2021, 23:21:18) 
[GCC 9.3.0]
</code></pre></div>
<p dir="auto"><em>Original ticket <a href="http://projects.scipy.org/numpy/ticket/505" rel="nofollow">http://projects.scipy.org/numpy/ticket/505</a> on 2007-04-24 by trac user mesmith, assigned to unknown.</em></p>
<p dir="auto">I would like to use fromfile with a file opened with the gzip module</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="     fp = gzip.open(filename,'r')
     arraybuffer = fromfile(fp, sep=',')"><pre class="notranslate"><code class="notranslate">     fp = gzip.open(filename,'r')
     arraybuffer = fromfile(fp, sep=',')
</code></pre></div>
<p dir="auto">but this gives an error</p>
<p dir="auto">IOError: first argument must be an open file</p>
<p dir="auto">I found a similar problem at</p>
<p dir="auto"><a href="http://projects.scipy.org/astropy/pyfits/ticket/7" rel="nofollow">http://projects.scipy.org/astropy/pyfits/ticket/7</a></p>
<p dir="auto">Has this problem been addressed in numpy? I am using version 1.0rc1.</p>