title: How to spawn parallel child processes on a multi-processor system?
body: <p>I have a Python script that I want to use as a controller to another Python script. I have a server with 64 processors, so want to spawn up to 64 child processes of this second Python script. The child script is called:</p>
  
  <pre><code>$ python create_graphs.py --name=NAME
  </code></pre>
  
  <p>where NAME is something like XYZ, ABC, NYU etc.</p>
  
  <p>In my parent controller script I retrieve the name variable from a list:</p>
  
  <pre><code>my_list = [ 'XYZ', 'ABC', 'NYU' ]
  </code></pre>
  
  <p>So my question is, what is the best way to spawn off these processes as children? I want to limit the number of children to 64 at a time, so need to track the status (if the child process has finished or not) so I can efficiently keep the whole generation running.</p>
  
  <p>I looked into using the subprocess package, but rejected it because it only spawns one child at a time. I finally found the multiprocessor package, but I admit to being overwhelmed by the whole threads vs. subprocesses documentation.</p>
  
  <p>Right now, my script uses <code>subprocess.call</code> to only spawn one child at a time and looks like this:</p>
  
  <pre><code>#!/path/to/python
  import subprocess, multiprocessing, Queue
  from multiprocessing import Process
  
  my_list = [ 'XYZ', 'ABC', 'NYU' ]
  
  if __name__ == '__main__':
      processors = multiprocessing.cpu_count()
  
      for i in range(len(my_list)):
          if( i &lt; processors ):
               cmd = ["python", "/path/to/create_graphs.py", "--name="+ my_list[i]]
               child = subprocess.call( cmd, shell=False )
  </code></pre>
  
  <p>I really want it to spawn up 64 children at a time. In other stackoverflow questions I saw people using Queue, but it seems like that creates a performance hit?</p>