title: Parallel.Invoke - Dynamically creating more 'threads'
body: <p>I am educating myself on Parallel.Invoke, and parallel processing in general, for use in current project. I need a push in the right direction to understand how you can dynamically\intelligently allocate more parallel 'threads' as required.</p>
  
  <p>As an example. Say you are parsing large log files. This involves reading from file, some sort of parsing of the returned lines and finally writing to a database.</p>
  
  <p>So to me this is a typical problem that can benefit from parallel processing.</p>
  
  <p>As a simple first pass the following code implements this.</p>
  
  <pre><code>Parallel.Invoke(
    ()=&gt; readFileLinesToBuffer(),
    ()=&gt; parseFileLinesFromBuffer(),
    ()=&gt; updateResultsToDatabase()    
  );
  </code></pre>
  
  <p>Behind the scenes </p>
  
  <ol>
  <li>readFileLinesToBuffer() reads each line and stores to a buffer.</li>
  <li>parseFileLinesFromBuffer comes along and consumes lines from buffer and then let's say it put them on another buffer so that updateResultsToDatabase() can come along and consume this buffer.</li>
  </ol>
  
  <p>So the code shown assumes that each of the three steps uses the same amount of time\resources but lets say the parseFileLinesFromBuffer() is a long running process so instead of running just one of these methods you want to run two in parallel.</p>
  
  <p>How can you have the code intelligently decide to do this based on any bottlenecks it might perceive?</p>
  
  <p>Conceptually I can see how some approach of monitoring the buffer sizes might work, spawning a new 'thread' to consume the buffer at an increased rate for example...but I figure this type of issue has been considered in putting together the TPL library.</p>
  
  <p>Some sample code would be great but I really just need a clue as to what concepts I should investigate next. It looks like maybe the System.Threading.Tasks.TaskScheduler holds the key?</p>