title: Is there a way to vstack() a very large array
body: <p>I have a huge data set (<a href="http://opendata.deepsig.io/datasets/2018.01/2018.01.OSC.0001_1024x2M.h5.tar.gz" rel="nofollow noreferrer">deepsig data set 2018</a>) for radio signals that is organized as following:</p>
  
  <p>the dataset is a .hdf5 file that has 3 keys (X, Y, Z)</p>
  
  <ul>
  <li>X : shape(2 555 904 x 1024 x 2) floats <code>over 2 Millions sample each has 1024 complex points (2 floats for real and imaginary parts)</code></li>
  <li>Y : shape(2 555 904 x 24) integer to point on the modulation type</li>
  <li>Z : shape(2 555 904 x 1) integer the Signal Noise Level (SNR)</li>
  </ul>
  
  <p>I organized both the modulation types and the SNR as a list of tuples <code>((mod,snr),(mod,snr),(mod,snr), ...)</code></p>
  
  <p>My problem is that when I extract the X data and try to organize them vertically i get a memory problem (cannot allocate enough memory, although I have 32GB RAM). here is my code:</p>
  
  <pre><code>import h5py
  import numpy as np
  
  filename = 'RML_2018.hdf5'
  f = h5py.File(filename, 'r')
  
  X = f['X'][:]
  X = np.vstack(X)
  print (X.shape)
  </code></pre>
  
  <p>an here is the error:</p>
  
  <pre><code>Traceback (most recent call last):
    File "/home/nechi/PycharmProjects/CNN_MR_v2/cnn.py", line 11, in &lt;module&gt;
      X = np.vstack(X)
    File "&lt;__array_function__ internals&gt;", line 6, in vstack
    File "/home/nechi/PycharmProjects/CNN_MR_v2/venv/lib/python3.6/site-packages/numpy/core/shape_base.py", line 282, in vstack
      return _nx.concatenate(arrs, 0)
    File "&lt;__array_function__ internals&gt;", line 6, in concatenate
  numpy.core._exceptions.MemoryError: Unable to allocate array with shape (2617245696, 2) and data type float32
  </code></pre>
  
  <p>So, my questions are :</p>
  
  <ol>
  <li>Is there a possible way to work around this memory issue?</li>
  <li>If I want to reduce the data (for example take 512 sample instead of 1024 samples), Do you have an idea how to do it?</li>
  </ol>