title: What's the best way to serialize a large scipy sparse matrix?
body: <p>I have a large scipy sparse matrix, which is taking up >90% of my total system memory. I would like to save it to disk, as it takes hours to build the matrix...</p>
  
  <p>I tried cPickle, but that leads to a major memory explosion...</p>
  
  <pre><code>import numpy as np
  from scipy.sparse import lil_matrix
  import cPickle
  
  dim = 10**8
  
  M = lil_matrix((dim, dim), dtype=np.float)
  
  with open(filename, 'wb') as f:
      cpickle.dump(M, f)  # leads to a major memory explosion, presumably there is lots of copying
  </code></pre>
  
  <p>while HDF5 didn't like the datatype: TypeError: Object dtype dtype('O') has no native HDF5 equivalent</p>
  
  <p>So what should I do?</p>