title: Multi threaded file processing with .NET
body: <p>There is a folder that contains 1000s of small text files. I aim to parse and process all of them while more files are being populated into the folder. My intention is to multithread this operation as the single threaded prototype took six minutes to process 1000 files.</p>
  
  <p>I like to have reader and writer thread(s) as the following. While the reader thread(s) are reading the files, I'd like to have writer thread(s) to process them. Once the reader is started reading a file, I d like to mark it as being processed, such as by renaming it. Once it's read, rename it to completed.</p>
  
  <p>How do I approach such a multithreaded application?</p>
  
  <p>Is it better to use a distributed hash table or a queue?</p>
  
  <p>Which data structure do I use that would avoid locks?</p>
  
  <p>Is there a better approach to this scheme?</p>