title: Convert a VERY LARGE binary file into a Base64String incrementally
body: <p>I need help converting a VERY LARGE binary file (ZIP file) to a Base64String and back again. The files are too large to be loaded into memory all at once (they throw OutOfMemoryExceptions) otherwise this would be a simple task. I do not want to process the contents of the ZIP file individually, I want to process the entire ZIP file.</p>
  
  <p>The problem:</p>
  
  <p>I can convert the entire ZIP file (test sizes vary from 1 MB to 800 MB at present) to Base64String, but when I convert it back, it is corrupted. The new ZIP file is the correct size, it is recognized as a ZIP file by Windows and WinRAR/7-Zip, etc., and I can even look inside the ZIP file and see the contents with the correct sizes/properties, but when I attempt to extract from the ZIP file, I get: "Error: 0x80004005" which is a general error code. </p>
  
  <p>I am not sure where or why the corruption is happening. I have done some investigating, and I have noticed the following: </p>
  
  <p>If you have a large text file, you can convert it to Base64String incrementally without issue. If calling <code>Convert.ToBase64String</code> on the entire file yielded: <strong>"abcdefghijklmnopqrstuvwx"</strong>, then calling it on the file in two pieces would yield: <strong>"abcdefghijkl"</strong> and <strong>"mnopqrstuvwx"</strong>.</p>
  
  <p>Unfortunately, if the file is a binary then the result is different. While the entire file might yield: <strong>"abcdefghijklmnopqrstuvwx"</strong>, trying to process this in two pieces would yield something like: <strong>"oiweh87yakgb"</strong> and <strong>"kyckshfguywp"</strong>.</p>
  
  <p>Is there a way to incrementally base 64 encode a binary file while avoiding this corruption? </p>
  
  <p>My code:</p>
  
  <pre><code>        private void ConvertLargeFile()
          {
             FileStream inputStream  = new FileStream("C:\\Users\\test\\Desktop\\my.zip", FileMode.Open, FileAccess.Read);
             byte[] buffer = new byte[MultipleOfThree];
             int bytesRead = inputStream.Read(buffer, 0, buffer.Length);
             while(bytesRead &gt; 0)
             {
                byte[] secondaryBuffer = new byte[buffer.Length];
                int secondaryBufferBytesRead = bytesRead;
                Array.Copy(buffer, secondaryBuffer, buffer.Length);
                bool isFinalChunk = false;
                Array.Clear(buffer, 0, buffer.Length);
                bytesRead = inputStream.Read(buffer, 0, buffer.Length);
                if(bytesRead == 0)
                {
                   isFinalChunk = true;
                   buffer = new byte[secondaryBufferBytesRead];
                   Array.Copy(secondaryBuffer, buffer, buffer.length);
                }
  
                String base64String = Convert.ToBase64String(isFinalChunk ? buffer : secondaryBuffer);
                File.AppendAllText("C:\\Users\\test\\Desktop\\Base64Zip", base64String); 
              }
              inputStream.Dispose();
          }
  </code></pre>
  
  <p>The decoding is more of the same. I use the size of the <code>base64String</code> variable above (which varies depending on the original buffer size that I test with), as the buffer size for decoding. Then, instead of <code>Convert.ToBase64String()</code>, I call <code>Convert.FromBase64String()</code> and write to a different file name/path.  </p>
  
  <p>EDIT:</p>
  
  <p>In my haste to reduce the code (I refactored it into a new project, separate from other processing to eliminate code that isn't central to the issue) I introduced a bug. The base 64 conversion should be performed on the <code>secondaryBuffer</code> for all iterations save the last (Identified by <code>isFinalChunk</code>), when <code>buffer</code> should be used. I have corrected the code above. </p>
  
  <p>EDIT #2:</p>
  
  <p>Thank you all for your comments/feedback. After correcting the bug (see the above edit), I re-tested my code, and it is actually working now. I intend to test and implement @rene's solution as it appears to be the best, but I thought that I should let everyone know of my discovery as well.</p>