title: Python: compress and save/load large data from/into memory
body: <p>I have a huge dictionary with numpy arrays as values which consumes almost all RAM. There is no possibility to pickle or compress it entirely. I've checked <a href="https://stackoverflow.com/questions/11675381/python-compress-large-files">some</a> of <a href="https://gist.github.com/davidcrawford/6774493" rel="nofollow noreferrer">solutions</a> to read/write in chunks using <code>zlib</code>, but they work with files, <code>StringIO</code>, etc, when I want to read/write from/into RAM.</p>
  
  <p><a href="https://stackoverflow.com/a/33563348/1692060">Here is the closest example to what I want</a>, but it has only writing part. How can I read the object after saving this way, because chunks were written together and compressed chunks of course have different length?</p>
  
  <pre><code>import zlib
  
  
  class ZlibWrapper():
      # chunksize is used to save memory, otherwise huge object will be copied
      def __init__(self, filename, chunksize=268435456): # 256 MB
          self.filename = filename
          self.chunksize = chunksize
  
  
      def save(self, data): 
          """Saves a compressed object to disk
          """
          mdata = memoryview(data)
          with open(self.filename, 'wb') as f:
            for i in range(0, len(mdata), self.chunksize):
               mychunk = zlib.compress(bytes(mdata[i:i+self.chunksize]))
               f.write(mychunk)
  
      def load(self):
  
          # ???
  
          return data
  </code></pre>
  
  <p>Uncompressed objects unfortunately would be too huge to be sent over network, and zipping them externally would create additional complications.</p>
  
  <p>Pickle unfortunately starts to consume RAM and system hangs.</p>
  
  <p>Following the discussion with Charles Duffy, here is my attempt of serialization (does not work at the moment - does not even compress the strings):</p>
  
  <pre><code>import zlib
  
  import json
  
  import numpy as np
  
  
  
  mydict = {"a":np.array([1,2,3]),"b":np.array([4,5,6]),"c":np.array([0,0,0])}
  
  
  #------------
  
  
  # write to compressed stream ---------------------
  
  def string_stream_serialization(dic):
      for key, val in dic.items():        
          #key_encoded = key.encode("utf-8")  # is not json serializable
          yield json.dumps([key,val.tolist()])
  
  
  output = ""
  compressor = zlib.compressobj()
  decompressor = zlib.decompressobj()
  
  
  stream = string_stream_serialization(mydict)
  
  with open("outfile.compressed", "wb") as f:
      for s in stream:
          if not s:
              f.write(compressor.flush())
              break
          f.write(compressor.compress(s.encode('utf-8'))) # .encode('utf-8') converts to bytes
  
  
  
  
  # read from compressed stream: --------------------
  
  def read_in_chunks(file_object, chunk_size=1024): # I set another chunk size intentionally
      """Lazy function (generator) to read a file piece by piece.
      Default chunk size: 1k."""
      while True:
          data = file_object.read(chunk_size)
          if not data:
              break
          yield data
  
  
  reconstructed = {}
  
  with open("outfile.compressed", "rb") as f:
      for s in read_in_chunks(f):
          data = decompressor.decompress(decompressor.unconsumed_tail + s)
          while data:
              arr = json.loads(data.decode("utf-8"))            
              reconstructed[arr[0]] = np.array(arr[1])
              data = decompressor.decompress(decompressor.unconsumed_tail)
  
  
  print(reconstructed)
  </code></pre>