<p>I am trying to consume an event stream provided by the  <a href="https://github.com/googlecloudplatform/kubernetes">Kubernetes</a>
api using the <code>requests</code> module.  I have run into what looks like a
buffering problem: the <code>requests</code> module seems to lag by one event.</p>

<p>I have code that looks something like this:</p>

<pre><code>r = requests.get('http://localhost:8080/api/v1beta1/watch/services',
                 stream=True)

for line in r.iter_lines():
    print 'LINE:', line
</code></pre>

<p>As Kubernetes emits event notifications, this code will only display
the last event emitted when a new event comes in, which makes it
almost completely useless for code that needs to respond to service
add/delete events.</p>

<p>I have solved this by spawning <code>curl</code> in a subprocess instead of using
the <code>requests</code> library:</p>

<pre><code>p = subprocess.Popen(['curl', '-sfN',
                      'http://localhost:8080/api/watch/services'],
                     stdout=subprocess.PIPE,
                     bufsize=1)

for line in iter(p.stdout.readline, b''):
    print 'LINE:', line
</code></pre>

<p>This works, but at the expense of some flexibility.  Is there a way to
avoid this buffering problem with the <code>requests</code> library?</p>

<p>This behavior is due to a buggy implementation of the <code>iter_lines</code>
method in the <code>requests</code> library.</p>

<p><code>iter_lines</code> iterates over the response content in <code>chunk_size</code> blocks
of data using the <code>iter_content</code> iterator.  If there are less than
<code>chunk_size</code> bytes of data available for reading from the remote
server (which will typically be the case when reading the last line of
output), the read operation will block until <code>chunk_size</code> bytes of
data are available.</p>

<p>I have written my own <code>iter_lines</code> routine that operates correctly:</p>

<pre><code>import os


def iter_lines(fd, chunk_size=1024):
    '''Iterates over the content of a file-like object line-by-line.'''

    pending = None

    while True:
        chunk = os.read(fd.fileno(), chunk_size)
        if not chunk:
            break

        if pending is not None:
            chunk = pending + chunk
            pending = None

        lines = chunk.splitlines()

        if lines and lines[-1]:
            pending = lines.pop()

        for line in lines:
            yield line

    if pending:
        yield(pending)
</code></pre>

<p>This works because <code>os.read</code> will return less than <code>chunk_size</code> bytes
of data rather than waiting for a buffer to fill.</p>
