title: using generators and cStringIO in python to stream strings
body: <p>I'm trying to read a very large string stream using cStringIO in a python dictionary:</p>
  
  <pre><code>def stream_read(self, path):
      try:
          # create a string stream from the contents at 'path'
          # note: the string at self._my_dict[path] is 7MB in size
          stream = StringIO.StringIO(self._my_dict[path])
          while True:
              # buffer size is 128kB, or 128 * 1024
              buf = stream.read(self.buffer_size)
              if buf != '':
                  yield buf
              else:
                  raise StopIteration
      except KeyError:
          raise IOError("Could not get content")
  </code></pre>
  
  <p>And in my test suite, I'm testing this function by first testing stream_write, asserting that the data exists at that path, and then calling stream_read:</p>
  
  <pre><code>def test_stream(self):
      filename = self.gen_random_string()
      # test 7MB
      content = self.gen_random_string(7 * 1024 * 1024)
      # test stream write
      io = StringIO.StringIO(content)
      self._storage.stream_write(filename, io)
      io.close()
      self.assertTrue(self._storage.exists(filename))
      # test read / write
      data = ''
      for buf in self._storage.stream_read(filename):
          data += buf
      self.assertEqual(content, data)
  </code></pre>
  
  <p>Yet in my test suite, I'm catching an AssertionError:</p>
  
  <pre><code>======================================================================
  FAIL: test_stream (test_swift_storage.TestSwiftStorage)
  ----------------------------------------------------------------------
  Traceback (most recent call last):
    File "/home/bacongobbler/.../test/test_local_storage.py", line 44, in test_stream
      self.assertEqual(content, data)
  AssertionError: '[squelched]' != '&lt;cStringIO.StringI object at 0x3148e70&gt;'
  ----------------------------------------------------------------------
  Ran 28 tests in 20.495s
  
  FAILED (failures=1)
  </code></pre>
  
  <p>It looks related to <a href="https://stackoverflow.com/questions/20429971/issues-working-with-python-generators-and-openstack-swift-client">an issue I posted last week</a>, but I'm still not quite sure I understand why <code>stream</code> is getting set to the Generator as a string in this case.</p>
  
  <p>If anyone wants to take a closer look at the source code, it's all up at <a href="https://github.com/bacongobbler/docker-registry/blob/106-swift-storage/test/utils/mock_swift_storage.py" rel="nofollow noreferrer">https://github.com/bacongobbler/docker-registry/blob/106-swift-storage/test/utils/mock_swift_storage.py</a></p>