title: Extract specific bytes from a binary file in Python
body: <p>I have very large binary files with x number of int16 data points for y sensors, along with headers with some basic info. The binary file is written as y values for each sample time up to x samples, then another set of readings and so on. If I want all of the data, I am using <code>numpy.fromfile()</code> which works really nice and fast. However, if I only want a subset of sensor data or only specific sensors, I currently have a hideous double <code>for</code> loop, using <code>file.seek()</code>, <code>file.read()</code>, and <code>struct.unpack()</code> that takes forever.  Is there another way to do this faster in python?  perhaps with <code>mmap()</code> which I do not understand well? or just using the whole <code>fromfile()</code> and then subsampling?</p>
  
  <pre><code>data = numpy.empty(num_pts, sensor_indices)
  for i in range(num_pts):
      for j in range(sensor_indices):
          curr_file.seek(bin_offsets[j])
          data_binary = curr_file.read(2)
          data[j][i] = struct.unpack('h', data_binary)[0]
  </code></pre>
  
  <p>Having followed advice from @rrauenza on <code>mmap</code>, which was great info, I edited the code to be</p>
  
  <pre><code>mm = mmap.mmap(curr_file.fileno(), 0, access=mmap.ACCESS_READ)
  data = numpy.empty(num_pts,sensor_indices)
  for i in range(num_pts):
      for j in range(len(sensor_indices)):
          offset += bin_offsets[j] * 2
          data[j][i] = struct.unpack('h', mm[offset:offset+2])[0]
  </code></pre>
  
  <p>while this IS faster than before, it's still orders of magnitude slower than</p>
  
  <pre><code>shape = (x, y)
  data = np.fromfile(file=self.curr_file, dtype=np.int16).reshape(shape)
  data = data.transpose()
  data = data[sensor_indices, :]
  data = data[:, range(num_pts)]
  </code></pre>
  
  <p>I tested this with a smaller 30 Mb file that is only 16 sensors with 30 seconds of data.  Original code was 160 s, <code>mmap</code> was 105 s, and <code>np.fromfile</code> and subsampling was 0.33 s.    </p>
  
  <p>Remaining question is - Clearly using <code>numpy.fromfile()</code> is better with the small file, but will there be issues with much larger files that may be up 20 Gb with hours or days of data and up to 500 sensors?</p>