title: Control the number of subprocesses using to call external commands in python
body: <p>I understand using <a href="http://docs.python.org/library/subprocess.html">subprocess</a> is the preferred way of calling external command. </p>
  
  <p>But what if I want to run several commands in parall, but limit the number of processes being spawned? What bothers me is that I can't block the subprocesses. For example, if I call </p>
  
  <pre><code>subprocess.Popen(cmd, stderr=outputfile, stdout=outputfile)
  </code></pre>
  
  <p>Then the process will continue, without waiting for <code>cmd</code> to finish. Therefore, I can't wrap it up in a worker of <code>multiprocessing</code> library. </p>
  
  <p>For example, if I do: </p>
  
  <pre><code>def worker(cmd): 
      subprocess.Popen(cmd, stderr=outputfile, stdout=outputfile);
  
  pool = Pool( processes = 10 );
  results =[pool.apply_async(worker, [cmd]) for cmd in cmd_list];
  ans = [res.get() for res in results];
  </code></pre>
  
  <p>then each worker will finish and return after spawning a subprocess. So I can't really limit the number of processes generated by <code>subprocess</code> by using <code>Pool</code>. </p>
  
  <p>What's the proper way of limiting the number of subprocesses? </p>