title: How do I pass large numpy arrays between python subprocesses without saving to disk?
body: <p>Is there a good way to pass a large chunk of data between two python subprocesses without using the disk? Here's a cartoon example of what I'm hoping to accomplish:</p>
  
  <pre><code>import sys, subprocess, numpy
  
  cmdString = """
  import sys, numpy
  
  done = False
  while not done:
      cmd = raw_input()
      if cmd == 'done':
          done = True
      elif cmd == 'data':
          ##Fake data. In real life, get data from hardware.
          data = numpy.zeros(1000000, dtype=numpy.uint8)
          data.dump('data.pkl')
          sys.stdout.write('data.pkl' + '\\n')
          sys.stdout.flush()"""
  
  proc = subprocess.Popen( #python vs. pythonw on Windows?
      [sys.executable, '-c %s'%cmdString],
      stdin=subprocess.PIPE,
      stdout=subprocess.PIPE,
      stderr=subprocess.PIPE)
  
  for i in range(3):
      proc.stdin.write('data\n')
      print proc.stdout.readline().rstrip()
      a = numpy.load('data.pkl')
      print a.shape
  
  proc.stdin.write('done\n')
  </code></pre>
  
  <p>This creates a subprocess which generates a numpy array and saves the array to disk. The parent process then loads the array from disk. It works!</p>
  
  <p>The problem is, our hardware can generate data 10x faster than the disk can read/write. Is there a way to transfer data from one python process to another purely in-memory, maybe even without making a copy of the data? Can I do something like passing-by-reference?</p>
  
  <p>My first attempt at transferring data purely in-memory is pretty lousy:</p>
  
  <pre><code>import sys, subprocess, numpy
  
  cmdString = """
  import sys, numpy
  
  done = False
  while not done:
      cmd = raw_input()
      if cmd == 'done':
          done = True
      elif cmd == 'data':
          ##Fake data. In real life, get data from hardware.
          data = numpy.zeros(1000000, dtype=numpy.uint8)
          ##Note that this is NFG if there's a '10' in the array:
          sys.stdout.write(data.tostring() + '\\n')
          sys.stdout.flush()"""
  
  proc = subprocess.Popen( #python vs. pythonw on Windows?
      [sys.executable, '-c %s'%cmdString],
      stdin=subprocess.PIPE,
      stdout=subprocess.PIPE,
      stderr=subprocess.PIPE)
  
  for i in range(3):
      proc.stdin.write('data\n')
      a = numpy.fromstring(proc.stdout.readline().rstrip(), dtype=numpy.uint8)
      print a.shape
  
  proc.stdin.write('done\n')
  </code></pre>
  
  <p>This is extremely slow (much slower than saving to disk) and very, very fragile. There's got to be a better way!</p>
  
  <p>I'm not married to the 'subprocess' module, as long as the data-taking process doesn't block the parent application. I briefly tried 'multiprocessing', but without success so far.</p>
  
  <p>Background: We have a piece of hardware that generates up to ~2 GB/s of data in a series of ctypes buffers. The python code to handle these buffers has its hands full just dealing with the flood of information. I want to coordinate this flow of information with several other pieces of hardware running simultaneously in a 'master' program, without the subprocesses blocking each other. My current approach is to boil the data down a little bit in the subprocess before saving to disk, but it'd be nice to pass the full monty to the 'master' process.</p>