title: Design pattern for aggregating lazy lists
body: <p>I'm writing a program as follows:</p>
  
  <ul>
  <li>Find all files with the correct extension in a given directory</li>
  <li>Foreach, find all occurrences of a given string in those files</li>
  <li>Print each line</li>
  </ul>
  
  <p>I'd like to write this in a functional way, as a series of generator functions (things that call <code>yield return</code> and only return one item at a time lazily-loaded), so my code would read like this:</p>
  
  <pre><code>IEnumerable&lt;string&gt; allFiles = GetAllFiles();
  IEnumerable&lt;string&gt; matchingFiles = GetMatches( "*.txt", allFiles );
  IEnumerable&lt;string&gt; contents = GetFileContents( matchingFiles );
  IEnumerable&lt;string&gt; matchingLines = GetMatchingLines( contents );
  
  foreach( var lineText in matchingLines )
    Console.WriteLine( "Found: " + lineText );
  </code></pre>
  
  <p>This is all fine, but what I'd also like to do is print some statistics at the end. Something like this:</p>
  
  <pre><code>Found 233 matches in 150 matching files. Scanned 3,297 total files in 5.72s
  </code></pre>
  
  <p>The problem is, writing the code in a 'pure functional' style like above, each item is lazily loaded.<br>
  You only know how many files match in total until the final foreach loop completes, and because only one item is ever <code>yield</code>ed at a time, the code doesn't have any place to keep track of how many things it's found previously. If you invoke LINQ's <code>matchingLines.Count()</code> method, it will re-enumerate the collection!</p>
  
  <p>I can think of many ways to solve this problem, but all of them seem to be somewhat ugly. It strikes me as something that people are bound to have done before, and I'm sure there'll be a nice design pattern which shows a best practice way of doing this.</p>
  
  <p>Any ideas? Cheers</p>