title: Extending a series of nonuniform netcdf data in a numpy array
body: <p>I am new to python, apologies if this has been asked already.</p>
  
  <p>Using python and numpy, I am trying to gather data across many netcdf files into a single array by iteratively calling <code>append()</code>.</p>
  
  <p>Naively, I am trying to do something like this:</p>
  
  <pre><code>from numpy import *
  from pupynere import netcdf_file
  
  x = array([])
  y = [...some list of files...]
  
  for file in y:
      ncfile = netcdf_file(file,'r')
      xFragment = ncfile.variables["varname"][:]
      ncfile.close()
      x = append(x, xFragment)
  </code></pre>
  
  <p>I know that under normal circumstances this is a bad idea, since it reallocates new memory on each <code>append()</code> call. But two things discourage preallocation of x:</p>
  
  <p>1) The files are not necessarily the same size along axis 0 (but should be the same size along subsequent axes), so I would need to read the array sizes from each file beforehand to precalculate the final size of x. </p>
  
  <p>However...</p>
  
  <p>2) From what I can tell, pupynere (and other netcdf modules) load the entire file into memory upon opening the file, rather than just a reference (such as many netcdf modules in other enviroments). So to preallocate, I'd have to open the files twice.</p>
  
  <p>There are many (>100) large (>1GB) files, so overallocating and reshaping is not practical, from what I can tell.</p>
  
  <p>My first question is whether I am missing some intelligent way to preallocate.</p>
  
  <p>My second question is more serious. The above snippet works for a single-dimension array. But if I try to load in a matrix, then initialisation becomes a problem. I can append a one-dimensional array to an empty array:</p>
  
  <pre><code>append( array([]), array([1, 2, 3]) )
  </code></pre>
  
  <p>but I cannot append an empty array to a matrix:</p>
  
  <pre><code>append( array([]), array([ [1, 2], [3, 4] ]), axis=0)
  </code></pre>
  
  <p>Something like x.extend(xFragment) would work, I believe, but I don't think numpy arrays have this functionality. I could also avoid the initialisation problem by treating the first file as a special case, but I'd prefer to avoid that if there's a better way to do it.</p>
  
  <p>If anyone can offer help or a suggestion, or can identify a problem with my approach, then I'd be grateful. Thanks</p>