title: Reading large text files with streams in C#
body: <p>I've got the lovely task of working out how to handle large files being loaded into our application's script editor (it's like <a href="http://en.wikipedia.org/wiki/Visual_Basic_for_Applications" rel="noreferrer">VBA</a> for our internal product for quick macros). Most files are about 300-400&nbsp;KB which is fine loading. But when they go beyond 100&nbsp;MB the process has a hard time (as you'd expect).</p>
  
  <p>What happens is that the file is read and shoved into a RichTextBox which is then navigated - don't worry too much about this part.</p>
  
  <p>The developer who wrote the initial code is simply using a StreamReader and doing</p>
  
  <pre><code>[Reader].ReadToEnd()
  </code></pre>
  
  <p>which could take quite a while to complete.</p>
  
  <p>My task is to break this bit of code up, read it in chunks into a buffer and show a progressbar with an option to cancel it.</p>
  
  <p>Some assumptions:</p>
  
  <ul>
  <li>Most files will be 30-40&nbsp;MB</li>
  <li>The contents of the file is text (not binary), some are Unix format, some are DOS.</li>
  <li>Once the contents is retrieved we work out what terminator is used.</li>
  <li>No-one's concerned once it's loaded the time it takes to render in the richtextbox. It's just the initial load of the text.</li>
  </ul>
  
  <p>Now for the questions:</p>
  
  <ul>
  <li>Can I simply use StreamReader, then check the Length property (so ProgressMax) and issue a Read for a set buffer size and iterate through in a while loop <em>WHILST</em> inside a background worker, so it doesn't block the main UI thread? Then return the stringbuilder to the main thread once it's completed.</li>
  <li>The contents will be going to a StringBuilder. can I initialise the StringBuilder with the size of the stream if the length is available?</li>
  </ul>
  
  <p>Are these (in your professional opinions) good ideas? I've had a few issues in the past with reading content from Streams, because it will always miss the last few bytes or something, but I'll ask another question if this is the case.</p>