title: How to use ReadAsync() on a network stream in combination with processing?
body: <p>I am trying to download from a server a large file, about 500 Mb, but instead of saving that file to the filesystem, I am trying to process it "on the fly", retrieving some chunks of data, analysing them and, when there is enough information, saving them to the database. Here is what I am trying to do:</p>
  
  <pre><code>byte[] buffer = new byte[64 * 1024];
  using (HttpResponseMessage response = await httpClient.GetAsync(Server + file, HttpCompletionOption.ResponseHeadersRead))
  using (Stream streamToReadFrom = await response.Content.ReadAsStreamAsync())
  {
      int wereRead;
      do
      {
          wereRead = await streamToReadFrom.ReadAsync(buffer, 0, buffer.Length);
  
          // Do the processing and saving
      } while (wereRead == buffer.Length);
  </code></pre>
  
  <p>I tried to use the buffer of 64k as the chunks of data I need to process are about that size. My reasoning was that since I am 'awaiting' on ReadAsync, the method call will not return until the buffer is full but that is not the case. The method was returning with only 7k to 14k bytes read. I tried to use a much smaller buffer, but anyway the speed of my processing is much higher than the speed of the download, so with a 4k buffer I might have a full buffer on the first iteration but only, say, 3k on the second iteration. </p>
  
  <p>Is there an approach that would be recommended in my situation? Basically, I want ReadAsync to only return once the buffer is full, or once the end of the stream is reached. </p>