Take the 2-minute tour ×

I'm using the subprocess module to start a subprocess and connect to it's output stream (stdout). I want to be able to execute non-blocking reads on its stdout. Is there a way to make .readline non-blocking or to check if there is data on the stream before I invoke .readline? I'd like this to be portable or at least work under Windows and Linux.

here is how I do it for now (It's blocking on the .readline if no data is avaible):

p = subprocess.Popen('myprogram.exe', stdout = subprocess.PIPE)
str = p.stdout.readline()
share|improve this question
The answer you checked doesn't work on windows like you asked... or am I mistaken? –  dlamotte Dec 10 '09 at 22:13
Indeed, as it says un the answer it does not work on Windows. I'll unnacept it, but I won't accept any other answer because I did not test them. –  Mathieu Pagé Dec 11 '09 at 14:13
add comment

18 Answers

fcntl, select, asyncproc won't help in this case.

Reliable way to read a stream without blocking on both Windows and Linux is to use Queue.get_nowait():

import sys
from subprocess import PIPE, Popen
from threading  import Thread

    from Queue import Queue, Empty
except ImportError:
    from queue import Queue, Empty  # python 3.x

ON_POSIX = 'posix' in sys.builtin_module_names

def enqueue_output(out, queue):
    for line in iter(out.readline, b''):

q = Queue()
t.daemon = True # thread dies with the program

# ... do other things here

# read line without blocking
except Empty:
    print('no output yet')
else: # got line
    # ... do something with line
share|improve this answer
This actually works. Thanks! –  Macke Feb 1 '12 at 13:26
Yes this works for me, I removed a lot though. It includes good practices but not always necessary. Python 3.x 2.X compat and close_fds may be omitted, it will still work. But just be aware of what everything does and don't copy it blindly, even if it just works! (Actually the simplest solution is to use a thread and do a readline as Seb did, Qeues are just an easy way to get the data, there are others, threads are the answer!) –  Aki Feb 22 '12 at 13:19
Inside the thread, the call to out.readline blocks the thread, and main thread, and I have to wait until readline returns before everything else continues. Any easy way around that? (I'm reading multiple lines from my process, which is also another .py file that's doing DB and things) –  Justin Apr 9 '12 at 19:00
@Justin: 'out.readline' doesn't block the main thread it is executed in another thread. –  J.F. Sebastian Apr 15 '12 at 18:45
what if I fail to shut down the subprocess, eg. due to exceptions? the stdout-reader thread won't die and python will hang, even if the main thread exited, isn't it? how could one work around this? python 2.x doesn't support killing the threads, what's worse, doesn't support interrupting them. :( (obviously one should handle the exceptions to assure the subprocess is shut down, but just in case it won't, what can you do?) –  naxa Sep 9 '13 at 6:51
show 4 more comments


import fcntl
import os
import sys

# make stdin a non-blocking file
fd = sys.stdin.fileno()
fl = fcntl.fcntl(fd, fcntl.F_GETFL)
fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)

# user input handling thread
while mainThreadIsRunning:
      try: input = sys.stdin.readline()
      except: continue


share|improve this answer
According to the docs, fcntl() can receive either a file descriptor, or an object that has .fileno() method. –  Denilson Sá Apr 27 '10 at 19:10
Jesse's answer is not correct. According to Guido, readline doesn't work correctly with non-blocking mode, and it won't before Python 3000. bugs.python.org/issue1175#msg56041 If you want to use fcntl to set the file to non-blocking mode, you have to use the lower-level os.read() and separate out the lines yourself. Mixing fcntl with high-level calls that perform line buffering is asking for trouble. –  anonnn Oct 26 '10 at 16:49
The use of readline seems incorrect in Python 2. See anonnn's answer stackoverflow.com/questions/375427/… –  Catalin Iacob Oct 27 '10 at 14:19
Please, don't use busy loops. Use poll() with a timeout to wait for the data. –  Ivo Danihelka Feb 13 '11 at 21:10
add comment

Try the asyncproc module. For example:

import os
from asyncproc import Process
myProc = Process("myprogram.app")

while True:
    # check to see if process has ended
    poll = myProc.wait(os.WNOHANG)
    if poll != None:
    # print any new output
    out = myProc.read()
    if out != "":
        print out


share|improve this answer
Absolutely brilliant. Much easier than the raw subprocess module. Works perfectly for me on Ubuntu. –  Cerin Dec 2 '10 at 12:30
asyncproc doesn't work on windows, and windows doesn't support os.WNOHANG :-( –  Bryan Oakley Jan 10 '11 at 22:01
asyncproc is GPL, which further limits its use :-( –  Bryan Oakley Feb 16 '11 at 22:28
Thanks. One small thing: It seems that replacing tabs with 8 spaces in asyncproc.py is the way to go :) –  benjaoming Nov 11 '12 at 14:49
add comment


from twisted.internet import protocol, reactor

class MyProcessProtocol(protocol.ProcessProtocol):

    def outReceived(self, data):
        print data

proc = MyProcessProtocol()

The Twisted documentation has some good information on this.


share|improve this answer
no good. select should not work on windows with file descriptors, according to docs –  naxa Sep 9 '13 at 7:10
@naxa I don't think the select() he's referring to is the same one you are. I'm assuming this because Twisted works on windows... –  notbad.jpeg Sep 29 '13 at 21:25
I've added similar solution based on asyncio from stdlib. –  J.F. Sebastian Dec 20 '13 at 5:51
add comment


Here's the threaded version of a timeout function:



f = open('myprogram_output.txt','w')
p = subprocess.Popen('myprogram.exe', stdout=f)

share|improve this answer
seems like recpie's thread would not exit after timeout and killing it depends on being able to kill the subprocess (sg. otherwise unrelated in this regard) it reads (a thing you should be able to but just in case you can't..). –  naxa Sep 9 '13 at 7:08
add comment

Use select & read(1).

import subprocess     #no new requirements
def readAllSoFar(proc, retVal=''): 
  return retVal
while not p.poll():
  print (readAllSoFar(p))

For readline()-like:

lines = ['']
while not p.poll():
    print a
  print a
share|improve this answer
add comment

Disclaimer: this works only for tornado


easy_install tornado_subprocess

now you can do something like this:

import tornado_subprocess
import tornado.ioloop

    def print_res( status, stdout, stderr ) :
    print status, stdout, stderr
    if status == 0:
        print "OK:"
        print stdout
        print "ERROR:"
        print stderr


you can also use it with a RequestHandler

class MyHandler(tornado.web.RequestHandler):
    def on_done(self, status, stdout, stderr):
        self.write( stdout )

    def get(self):
share|improve this answer
Thanks for the nice feature! Just to clarify, why can't we simply use threading.Thread for creating new non-blocking processes? I used it in on_message of Tornado websocket instance, and it did the job fine. –  VisioN Nov 26 '12 at 22:46
threading is mostly discouraged in tornado. they are fine for small, short running functions. You can read about it here: stackoverflow.com/questions/7846323/tornado-web-and-threads github.com/facebook/tornado/wiki/Threading-and-concurrency –  Vukasin Toroman Dec 1 '12 at 22:25
@VukasinToroman you really saved me here with this. thank you so much for the tornado_subprocess module :) –  James Gentes May 23 '13 at 19:14
@JamesGentes t'was a pleasure –  Vukasin Toroman May 27 '13 at 10:17
does this work on windows? (note that select, with file descriptors, does not) –  naxa Sep 9 '13 at 7:11
show 1 more comment


import fcntl

def non_block_read(output):
    fd = output.fileno()
    fl = fcntl.fcntl(fd, fcntl.F_GETFL)
    fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)
        return output.read()
        return ""

# Use example
from subprocess import *

# sb.stdout.read() # <-- This will block
share|improve this answer
fcntl doesn't work on windows, according to the docs. –  naxa Sep 9 '13 at 7:13
add comment


from subprocess import Popen, PIPE
from threading import Thread
def process_output(myprocess): #output-consuming thread
    nextline = None
    buf = ''
    while True:
        #--- extract line using read(1)
        out = myprocess.stdout.read(1)
        if out != '':
            buf += out
            if out == '\n':
                nextline = buf
                buf = ''
        if not nextline: continue
        line = nextline
        nextline = None

        #--- do whatever you want with line here
        print 'Line is:', line

p1 = Thread(target=process_output, args=(dcmpid,)) #output-consuming thread
p1.daemon = True


Why existing solutions did not work:

share|improve this answer
1. q.get_nowait() from my answer must not block, ever, that is the point of using it. 2. The thread that executes readline (enqueue_output() function) exits on EOF e.g., including the case when the output-producing process is killed. If you believe it is not so; please, provide a complete minimal code example that shows otherwise (maybe as a new question). –  J.F. Sebastian Apr 3 '13 at 9:32
@sebastian I spent an hour or more trying to come up with a minimal example. In the end I must agree that your answer handles all the cases. I guess it didn't work earlier for me because when I was trying to kill the output-producing process, it was already killed and gave a hard-to-debug error. The hour was well spent, because while coming up with a minimal example, I could come up with a simpler solution. –  Vikram Pudi Apr 5 '13 at 10:11
Could you post the simpler solution, too? :) (if it's different from Sebastian's) –  naxa Sep 9 '13 at 7:33
add comment



share|improve this answer
I think this answer is unhelpful for two reasons: (a) The select module will not work on pipes under Windows (as the provided link clearly states), which defeats the OP's intentions to have a portable solution. (b) Asynchronous threads do not allow for a synchronous dialogue between the parent and the child process. What if the parent process wants to dispatch the next action according to the next line read from the child?! –  ThomasH Jul 14 '09 at 22:32
2.6 documentation link -- docs.python.org/library/select.html –  gatoatigrado Jun 3 '10 at 18:57
select is also not useful in that Python's reads will block even after the select, because it does not have standard C semantics and will not return partial data. –  Helmut Grohne Jan 27 '11 at 14:51
A separate thresd for reading from child's output solved my problem which was similar to this. If you need syncronous interaction I guess you can't use this solution (unless you know what output to expect). I would have accepted this answer –  Emiliano Feb 18 '11 at 7:20
add comment


Tested and correctly worked on Python 2.7 linux & windows.

# Runner with stdout/stderr catcher
from sys import argv
from subprocess import Popen, PIPE
import os, io
from threading import Thread
import Queue
def __main__():
        import time, sys
        print "Application runned!"
        print "Slept 2 second"
        print "Slept 1 additional second",
        sys.stderr.write("Stderr output after 5 seconds")
        print "Eol on stdin"
        sys.stderr.write("Eol on stderr\n")
        print "Wow, we have end of work!",
                       bufsize=0, # line-buffered
                       stdin=PIPE, stdout=PIPE, stderr=PIPE )
        except WindowsError, W:
            if W.winerror==193:
                           shell=True, # Try to run via shell
                           bufsize=0, # line-buffered
                           stdin=PIPE, stdout=PIPE, stderr=PIPE )
        inp = Queue.Queue()
        def Pump(stream, category):
            queue = Queue.Queue()
            def rdr():
                while True:
                    buf = stream.read1(8192)
                    if len(buf)>0:
                        queue.put( buf )
                        queue.put( None )
            def clct():
                active = True
                while active:
                    r = queue.get()
                        while True:
                            r1 = queue.get(timeout=0.005)
                            if r1 is None:
                                active = False
                                r += r1
                    except Queue.Empty:
                    inp.put( (category, r) )
            for tgt in [rdr, clct]:
                th = Thread(target=tgt)
        Pump(sout, 'stdout')
        Pump(serr, 'stderr')

        while p.poll() is None:
            # App still working
                chan,line = inp.get(timeout = 1.0)
                if chan=='stdout':
                elif chan=='stderr':
            except Queue.Empty:
        print "Finish"

if __name__ == '__main__':
share|improve this answer
add comment



#!/usr/bin/env python3.4
import asyncio

class SubprocessProtocol(asyncio.SubprocessProtocol):
    def pipe_data_received(self, fd, data):

    def process_exited(self):
        loop.stop() # end loop.run_forever()

loop = asyncio.get_event_loop()

On Windows, the ProactorEventLoop should be used to support subprocesses. See "Running subprocesses" in the docs.

share|improve this answer
add comment

I have port http://code.google.com/p/subprocdev/ to python 2.5, and it works in Windows

share|improve this answer
this seems to use Mark Hammond's pywin32 –  naxa Sep 9 '13 at 7:23
add comment



share|improve this answer
wow, this is a tester module. –  naxa Sep 9 '13 at 7:40
add comment

I did a little checking and as far as I can tell the most convenient solution is still this old recipe for an asynchronous Popen subclass, which appears to be also available on PyPI.

share|improve this answer
No, it actually isn't on PyPI. The one on PyPI is a module with the same name inspired by @J.F.Sebastian's answer. –  Nicola Musatti 7 hours ago
add comment

EDIT: This implementation still blocks. Use J.F.Sebastian's answer instead.



import io
from subprocess import PIPE, Popen



# Create an io.BufferedReader on the file descriptor for stdout
  while p.poll() == None:
      while '\n' in bufferedStdout.peek(bufferedStdout.buffer_size):
          line = buffer.readline()
          # do stuff with the line

  # Handle any remaining output after the process has ended
  while buffer.peek():
    line = buffer.readline()
    # do stuff with the line
share|improve this answer
have you tried for line in iter(p.stdout.readline, ""): # do stuff with the line? It is threadless (single thread) and blocks when your code blocks. –  J.F. Sebastian Nov 10 '13 at 3:20
@j-f-sebastian Yeah, I eventually reverted to your answer. My implementation still occasionally blocked. I'll edit my answer to warn others not to go down this route. –  romc Nov 27 '13 at 16:22
add comment

This allows you to pass a timeout to read() http://www.pixelbeat.org/libs/subProcess.py

share|improve this answer
no good, uses select, unavailable in windows to file descriptors. –  naxa Sep 9 '13 at 7:34
add comment



share|improve this answer
add comment

Your Answer


