title: How to stream data in python without loading it all into memory at once?
body: <p>I am trying to write and read to a stream without loading everything into memory at once. Here's what I would imagine working:</p>
  <pre><code>import io
  
  stream = io.BytesIO()
  
  def process_stream(stream):
    while True:
      chunk = stream.read(5).decode('utf-8')
      if not chunk:
        return
      yield chunk
  
  # this would be a separate thread, but here we just do it in serial:
  for i in range(3):
    stream.write(b'asdf')
  
  for chunk in process_stream(stream):
    print('I read', chunk)
  </code></pre>
  <p>But this actually doesn't print out anything.
  I can get it working, but only with the following two changes, either of which requires that all the bytes are held in memory at once:</p>
  <ul>
  <li>initializing <code>stream = io.BytesIO(b'asdf' * 3)</code> instead of incrementally writing</li>
  <li>using <code>stream.getvalue()</code> instead of incrementally reading</li>
  </ul>
  <p>I'm quite baffled that incremental writing can only be read by batch reading, and that incremental reading only works for batch writing. How can a get a constant-memory (assuming <code>process_stream</code> outpaces writing) solution working?</p>