title: Python: How to save 2d numpy array efficiently to disk?
body: <p>I have a huge 2d numpy array that's suppose to work as a co-occurrence matrix. I've tried to use scipy.sparse as my data structure, but <code>dok_matrix</code> indexing is incredibly slow (4x times slower).</p>
  
  <pre><code># Impossible
  import numpy
  N = 1000000 (1 milion)
  coo = np.zeros((N, N), dtype=np.uint32)
  </code></pre>
  
  <p>I want to persist this array. </p>
  
  <p>After searching for ways to save it I tried to use <code>PyTables</code> or <code>hd5py</code>, but I couldn't find a way to save it without running out of memory.</p>
  
  <pre><code>with open(name, 'w') as _file:
     np.save(_file, coo)
  </code></pre>
  
  <p>For instance, using <code>PyTables</code>:</p>
  
  <pre><code>    import tables
      _file = tables.openFile(
                  name,
                  mode='w',
                  title='Co-occurrence matrix')
      atom = tables.Atom.from_dtype(coo.dtype)
      _filters = tables.Filters(complib='blosc', complevel=5)
      ds = _file.createEArray(
              _file.root,
              'coo_matrix',
              atom,
              shape=(0, coo.shape[-1]),
              expectedrows=coo.shape[-1],
              filters=_filters)
      # ds[:] = coo =&gt; not an option
      for _index, _data in enumerate(coo):
          ds.append(coo[_index][np.newaxis,:])
      _file.close()
  </code></pre>
  
  <p>And using <code>hd5py</code>:</p>
  
  <pre><code>import h5py
  h5f = h5py.File(name, 'w')
  h5f.create_dataset('dataset_1', data=coo)
  </code></pre>
  
  <p>Both methods keep increasing memory usage until I have to kill the process. So, is there any way to do it incrementally? If it's not possible to do it can you recommend another way for persisting this matrix?</p>
  
  <p>EDIT</p>
  
  <p>I'm creating this co-occurrence matrix like this:</p>
  
  <pre><code>    coo = np.zeros((N, N), dtype=np.uint32)
      for doc_id, doc in enumerate(self.w.get_docs()):
          for w1, w2 in combinations(doc, 2):
                  if w1 != w2:
                      coo[w1, w2] += 1
  </code></pre>
  
  <p>I want to save coo (2d numpy array) to retrieve it from disk later and find co-occurrence values, like: coo[w1, w2]</p>