title: Run subprocess inside Python thread reading the output in realtime
body: <p>Consider the following Python code:</p>
  
  <pre><code>import io
  import time
  import subprocess
  import sys
  
  from thread import start_new_thread
  
  def ping_function(ip):
  
      filename = 'file.log'
      command = ["ping", ip]
  
      with io.open(filename, 'wb') as writer, io.open(filename, 'rb', 1) as reader:
          process = subprocess.Popen(command, stdout=writer)
          while process.poll() is None:
              line = reader.read()
              # Do something with line
              sys.stdout.write(line)
              time.sleep(0.5)
          # Read the remaining
          sys.stdout.write(reader.read())
  
  ping_function("google.com")
  </code></pre>
  
  <p>The goal is to run a shell command (in this case <strong>ping</strong>, but it is not relevant here) and to process the output in real time, which is also saved on a log file.</p>
  
  <p>In other word, <strong>ping</strong> is running in background and it produces output on the terminal every second. My code will read this output (every 0.5 seconds), parse it and take some action in (<em>almost</em>) real time.</p>
  
  <p>Realtime here means that I don't want to wait the end of the process to read the output. In this case actually <strong>ping</strong> never completes so an approach like the one I have just described is mandatory.</p>
  
  <p>I have tested the code above and it actually works OK :)</p>
  
  <p>Now I'd like to tun this in a separate thread, so I have replaced the last line with the following:</p>
  
  <pre><code>from thread import start_new_thread
  start_new_thread(ping_function, ("google.com", ))
  </code></pre>
  
  <p>For some reason this does not work anymore, and the reader always return empty strings.
  In particular, the string returned by <strong>reader.read()</strong> is always empty.</p>
  
  <p>Using a Queue or another global variable is not going to help, because I am having problems even to retrieve the data in the first place (i.e. to obtain the output of the shell command)</p>
  
  <p>My questions are:</p>
  
  <ul>
  <li><p>How can I explain this behavior?</p></li>
  <li><p>Is it a good idea to run a process inside a separate thread or I should use a different approach? <a href="http://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-using-them" rel="nofollow">This article</a> suggests that it is not...</p></li>
  <li><p>How can I fix the code?</p></li>
  </ul>
  
  <p>Thanks!</p>