title: Why is subprocess.Popen not waiting until the child process terminates?
body: <p>I'm having a problem with Python's subprocess.Popen method. </p>
  
  <p>Here's a test script which demonstrates the problem. It's being run on a Linux box.</p>
  
  <pre><code>#!/usr/bin/env python
  import subprocess
  import time
  
  def run(cmd):
    p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
    return p
  
  ### START MAIN
  # copy some rows from a source table to a destination table
  # note that the destination table is empty when this script is run
  cmd = 'mysql -u ve --skip-column-names --batch --execute="insert into destination (select * from source limit 100000)" test'
  run(cmd)
  
  # check to see how many rows exist in the destination table
  cmd = 'mysql -u ve --skip-column-names --batch --execute="select count(*) from destination" test'
  process = run(cmd)
  count = (int(process.communicate()[0][:-1]))
  
  # if subprocess.Popen() waited for the child to terminate than count should be
  # greater than 0
  if count &gt; 0:
    print "success: " + str(count)
  else:
    print "failure: " + str(count)
    time.sleep(5)
  
    # find out how many rows exists in the destination table after sleeping
    process = run(cmd)
    count = (int(process.communicate()[0][:-1]))
    print "after sleeping the count is " + str(count)
  </code></pre>
  
  <p>Usually the output from this script is:</p>
  
  <pre><code>success: 100000
  </code></pre>
  
  <p>but sometimes it's</p>
  
  <pre><code>failure: 0
  after sleeping the count is 100000
  </code></pre>
  
  <p>Note that in the failure case, the select immediately after the insert shows 0 rows but after sleeping for 5 seconds a second select correctly shows a row count of 100000. My conclusion is that one of the following is true:</p>
  
  <ol>
  <li>subprocess.Popen is not waiting for the child thread to terminate - This seems to contradict the documentation</li>
  <li>the mysql insert is not atomic - my understanding of mysql seems to indicate insert is atomic</li>
  <li>the select is not seeing the correct row count right away - according to a friend who knows mysql better than I do this should not happen either</li>
  </ol>
  
  <p>What am I missing?</p>
  
  <p>FYI, I'm aware that this is a hacky way of interacting with mysql from Python and MySQLdb would likely not have this problem but I'm curious as to why this method does not work.</p>