title: Pattern in C# for loading data from large files with low memory impact
body: <p>I have a scenario, where I load data from large files, e.g. json and I would like to keep the memory usage as low as possible. The other problem is that I would like to encapsulate the data loading from the processing part, so I can also implement an importer for xml for example.</p>
  
  <p>If the data contains only a simple list it is quite easy, I can define an interface like this:</p>
  
  <pre><code>interface IImporter 
  {
     IEnumerable&lt;Data&gt; LoadFrom(Stream stream);
  }
  </code></pre>
  
  <p>And then I can implement it using the yield keyword:</p>
  
  <pre><code>class JsonImporter : IImporter
  {
      public IEnumerable&lt;Data&gt; LoadFrom(Stream stream)
      {
          JsonReader reader = ...;
          ...
          while (...)
          {
              yield return new Data(reader.Read(...));
          }
      }
  }
  </code></pre>
  
  <p>The advantage is that the memory impact is very low, if the JsonReader works efficiently only one Data object is in memory (depending on the processing). The negative side is that you cannot iterate over the enumerable twice depending on the situation.</p>
  
  <p>But what happens if there is more complex data, e.g. first a header with some simple properties in json and then two array with totally different structure? In this case an IEnumerable does not work anymore. Do you have some patterns here or is there a framework for this. I think also about buffering and how the consumer can notify the producer that processing has been finished, so the stream (or the sql connection, ...) can be closed and disposed. Looking for some ideas.</p>