title: Split a generator into chunks without pre-walking it
body: <p>(This question is related to <a href="https://stackoverflow.com/q/8290397/647991">this one</a> and <a href="https://stackoverflow.com/q/312443/647991">this one</a>, but those are pre-walking the generator, which is exactly what I want to avoid)</p>
  
  <p>I would like to split a generator in chunks. The requirements are:</p>
  
  <ul>
  <li>do not pad the chunks: if the number of remaining elements is less than the chunk size, the last chunk must be smaller.</li>
  <li>do not walk the generator beforehand: computing the elements is expensive, and it must only be done by the consuming function, not by the chunker</li>
  <li>which means, of course: do not accumulate in memory (no lists)</li>
  </ul>
  
  <p>I have tried the following code:</p>
  
  <pre><code>def head(iterable, max=10):
      for cnt, el in enumerate(iterable):
          yield el
          if cnt &gt;= max:
              break
  
  def chunks(iterable, size=10):
      i = iter(iterable)
      while True:
          yield head(i, size)
  
  # Sample generator: the real data is much more complex, and expensive to compute
  els = xrange(7)
  
  for n, chunk in enumerate(chunks(els, 3)):
      for el in chunk:
          print 'Chunk %3d, value %d' % (n, el)
  </code></pre>
  
  <p>And this somehow works:</p>
  
  <pre><code>Chunk   0, value 0
  Chunk   0, value 1
  Chunk   0, value 2
  Chunk   1, value 3
  Chunk   1, value 4
  Chunk   1, value 5
  Chunk   2, value 6
  ^CTraceback (most recent call last):
    File "xxxx.py", line 15, in &lt;module&gt;
      for el in chunk:
    File "xxxx.py", line 2, in head
      for cnt, el in enumerate(iterable):
  KeyboardInterrupt
  </code></pre>
  
  <p>Buuuut ... it never stops (I have to press <code>^C</code>) because of the <code>while True</code>. I would like to stop that loop whenever the generator has been consumed, but I do not know how to detect that situation. I have tried raising an Exception:</p>
  
  <pre><code>class NoMoreData(Exception):
      pass
  
  def head(iterable, max=10):
      for cnt, el in enumerate(iterable):
          yield el
          if cnt &gt;= max:
              break
      if cnt == 0 : raise NoMoreData()
  
  def chunks(iterable, size=10):
      i = iter(iterable)
      while True:
          try:
              yield head(i, size)
          except NoMoreData:
              break
  
  # Sample generator: the real data is much more complex, and expensive to compute    
  els = xrange(7)
  
  for n, chunk in enumerate(chunks(els, 2)):
      for el in chunk:
          print 'Chunk %3d, value %d' % (n, el)
  </code></pre>
  
  <p>But then the exception is only raised in the context of the consumer, which is not what I want (I want to keep the consumer code clean)</p>
  
  <pre><code>Chunk   0, value 0
  Chunk   0, value 1
  Chunk   0, value 2
  Chunk   1, value 3
  Chunk   1, value 4
  Chunk   1, value 5
  Chunk   2, value 6
  Traceback (most recent call last):
    File "xxxx.py", line 22, in &lt;module&gt;
      for el in chunk:
    File "xxxx.py", line 9, in head
      if cnt == 0 : raise NoMoreData
  __main__.NoMoreData()
  </code></pre>
  
  <p>How can I detect that the generator is exhausted in the <code>chunks</code> function, without walking it?</p>