title: Is it possible to access the compressed data before decompression in HttpClient?
body: <p>I'm working on the <a href="https://googlecloudplatform.github.io/google-cloud-dotnet/docs/Google.Cloud.Storage.V1/index.html" rel="noreferrer">Google Cloud Storage .NET client
  library</a>. There are three features (between .NET, my client
  library, and the Storage service) that are combining in an
  unpleasant way:</p>
  
  <ul>
  <li><p>When downloading files (objects in Google Cloud Storage
  terminology), the server includes a hash of the stored data. My
  client code then validates that hash against the data it's
  downloaded.</p></li>
  <li><p>A separate feature of Google Cloud Storage is that the user can
  set the Content-Encoding of the object, and that's included as a
  header when downloading, when the request contains a matching
  Accept-Encoding. (For the moment, let's ignore the behavior when the
  request doesn't include that...)</p></li>
  <li><p><code>HttpClientHandler</code> can decompress gzip (or deflate) content
  automatically and transparently.</p></li>
  </ul>
  
  <p>When all three of these are combined, we get into trouble. Here's a
  short but complete program demonstrating that, but without using my
  client library (and hitting a publicly accessible file):</p>
  
  <pre><code>using System;
  using System.Linq;
  using System.Net;
  using System.Net.Http;
  using System.Security.Cryptography;
  using System.Text;
  using System.Threading.Tasks;
  
  class Program
  {
      static async Task Main()
      {
          string url = "https://www.googleapis.com/download/storage/v1/b/"
              + "storage-library-test-bucket/o/gzipped-text.txt?alt=media";
          var handler = new HttpClientHandler
          {
              AutomaticDecompression = DecompressionMethods.GZip
          };
          var client = new HttpClient(handler);
  
          var response = await client.GetAsync(url);
          byte[] content = await response.Content.ReadAsByteArrayAsync();
          string text = Encoding.UTF8.GetString(content);
          Console.WriteLine($"Content: {text}");
  
          var hashHeader = response.Headers.GetValues("X-Goog-Hash").FirstOrDefault();
          Console.WriteLine($"Hash header: {hashHeader}");
  
          using (var md5 = MD5.Create())
          {
              var md5Hash = md5.ComputeHash(content);
              var md5HashBase64 = Convert.ToBase64String(md5Hash);
              Console.WriteLine($"MD5 of content: {md5HashBase64}");
          }
      }
  }
  </code></pre>
  
  <p>.NET Core project file:</p>
  
  <pre><code>&lt;Project Sdk="Microsoft.NET.Sdk"&gt;
    &lt;PropertyGroup&gt;
      &lt;OutputType&gt;Exe&lt;/OutputType&gt;
      &lt;TargetFramework&gt;netcoreapp2.0&lt;/TargetFramework&gt;
      &lt;LangVersion&gt;7.1&lt;/LangVersion&gt;
    &lt;/PropertyGroup&gt;
  &lt;/Project&gt;
  </code></pre>
  
  <p>Output:</p>
  
  <pre><code>Content: hello world
  Hash header: crc32c=T1s5RQ==,md5=xhF4M6pNFRDQnvaRRNVnkA==
  MD5 of content: XrY7u+Ae7tCTyyK7j1rNww==
  </code></pre>
  
  <p>As you can see, the MD5 of the content isn't the same as the MD5
  part of the <code>X-Goog-Hash</code> header. (In my client library I'm using the crc32c
  hash, but that shows the same behavior.)</p>
  
  <p>This isn't a bug in <code>HttpClientHandler</code> - it's expected, but a pain
  when I want to validate the hash. Basically, I need to at the
  content before <em>and</em> after decompression. And I can't find any way
  of doing that.</p>
  
  <p>To clarify my requirements somewhat, I know how to prevent the decompression in <code>HttpClient</code> and instead decompress afterwards when reading from the stream - but I need to be able to do this without changing any the code that uses the resulting <code>HttpResponseMessage</code> from the <code>HttpClient</code>. (There's a lot of code that deals with responses, and I want to only make the change in one central place.)</p>
  
  <p>I have a plan, which I've prototyped and which works as far as I've
  found so far, but is a bit ugly. It involves creating a three-layer
  handler:</p>
  
  <ul>
  <li><code>HttpClientHandler</code> with automatic decompression disabled.</li>
  <li>A new handler which replaces the content stream with a new <code>Stream</code> subclass 
  which delegates to the original content stream, but hashes the data as it's read.</li>
  <li>A decompression-only handler, based on the Microsoft <a href="https://github.com/dotnet/corefx/blob/master/src/System.Net.Http/src/System/Net/Http/Managed/DecompressionHandler.cs" rel="noreferrer"><code>DecompressionHandler</code></a> code.</li>
  </ul>
  
  <p>While this works, it has disadvantages of:</p>
  
  <ul>
  <li>Open source licensing: checking exactly what I need to do in order
  to create a new file in my repo based on the MIT-licensed
  Microsoft code</li>
  <li>Effectively forking the MS code, which means I should probably
  make a regular check to see if any bugs have been found in it</li>
  <li>The Microsoft code uses internal members of the assembly, so it
  doesn't port as cleanly as it might.</li>
  </ul>
  
  <p>If Microsoft made <code>DecompressionHandler</code> public, that would help a
  lot - but that's likely to be in a longer timeframe than I need.</p>
  
  <p>What I'm looking for is an alternative approach if possible -
  something I've missed that lets me get at the content before
  decompression. I don't want to reinvent <code>HttpClient</code> - the response
  is often chunked for example, and I don't want to have to get into
  that side of things. It's a pretty specific interception point that
  I'm looking for.</p>