title: Serializing a very large List of items into Azure blob storage using C#
body: <p>I have a large list of objects that I need to store and retrieve later. The list will always be used as a unit and list items are not retrieved individually. The list contains about 7000 items totaling about 1GB, but could easily escalate to ten times that or more.</p>
  
  <p>We have been using <code>BinaryFormatter.Serialize()</code> to do the serialization (<code>System.Runtime.Serialization.Formatters.Binary.BinaryFormatter</code>). Then, this string was uploaded as a blob to Azure blob storage. We found it to be generally fast and efficient, but it became inadequate as we are testing it with a greater file size, throwing an <code>OutOfMemoryException</code>. From what I understand, although I'm using a stream, my problem is that the <code>BinaryFormatter.Serialize()</code> method must first serialize everything to memory before I can upload the blob, causing my exception.</p>
  
  <p>The binary serializer looks as follows:</p>
  
  <pre><code>public void Upload(object value, string blobName, bool replaceExisting)
  {
      CloudBlockBlob blockBlob = BlobContainer.GetBlockBlobReference(blobName);
      var formatter = new BinaryFormatter()
      {
          AssemblyFormat = FormatterAssemblyStyle.Simple,
          FilterLevel = TypeFilterLevel.Low,
          TypeFormat = FormatterTypeStyle.TypesAlways
      };
  
      using (var stream = blockBlob.OpenWrite())
      {
          formatter.Serialize(stream, value);
      }
  }
  </code></pre>
  
  <p>The OutOfMemoryException occurs on the <code>formatter.Serialize(stream, value)</code> line.</p>
  
  <p>I therefore tried to using a different protocol, Protocol Buffers. I tried using both the implementations in the Nuget packages protobuf-net and Google.Protobuf, but the serialization was horribly slow (roughly 30mins) and, from what I have read, Protobuf is not optimized for serializing data larger than 1MB. So, I went back to the drawing board, and came across Cap'n Proto, which promised to solve my speed issues by using memory mapping. I am trying to use @marc-gravell 's C# bindings but I am having some difficulty implementing a serializer, as the project does not have thorough documentation yet. Moreover, I'm not 100% sure that Cap'n Proto is the correct choice of protocol - but I am struggling to find any alternative suggestions online.</p>
  
  <p><strong>How can I serialize a very large collection of items to blob storage, without hitting memory issues, and in a reasonably fast way?</strong></p>