title: How to prioritize tasks generated by await when using async / await
body: <p>I have a large number of pieces of data to process. The current code can be simplified as following:</p>
  <pre><code>public void ProcessData(string data)
  {
      string resultOfA = doCpuBoundWorkA(data);
  
      string resultOfS1 = sendToServiceS1(resultOfA);
  
      string resultOfB = doCpuBoundWorkB(resultOfS1);
  
      string resultOfS2 = sendToServiceS2(resultOfB);
  
      string resultOfC = doCpuBoundWorkC(resultOfS2);
  }
  </code></pre>
  <p>The ProcessData is invoked using <code>Parallel.ForEach</code>. This implementation is not optimal from at least two perspectives. First of all the calls to the services are blocking so we are blocking threads while waiting for the call to return. Secondly <code>Parallel.ForEach</code> creates Tasks that are scheduled for execution on the thread pool. The Thread Pool creates additional threads every 500ms (if I'm not wrong on that) and because 'ProcessData' takes longer than 500ms to complete, over time, we end up with hundreds of threads that spend most of the time waiting for the services to comeback.</p>
  <p>My naive idea for 'improvement' was this:</p>
  <pre><code>public async Task ProcessData(string data)
  {
      string resultOfA = doCpuBoundWorkA(data);
  
      string resultOfS1 = await sendToServiceS1Async(resultOfA);
  
      string resultOfB = doCpuBoundWorkB(resultOfS1);
  
      string resultOfS2 = await sendToServiceS2Async(resultOfB);
  
      string resultOfC = doCpuBoundWorkC(resultOfS2);
  }
  </code></pre>
  <p>I'm new to async/await so I could be totally wrong in my understanding of what it is actually happening.</p>
  <p>With the async/await keywords the compiler breaks the code of the ProcessData it into multiple Tasks.</p>
  <ul>
  <li>Task-A: From the beginning of the ProcessData method up-to the point where call to The ServiceA &quot;hits the wire&quot;.</li>
  <li>Task-B: From the moment when we pick up the results of call to the ServiceA up-to the point where call to The ServiceB &quot;hits the wire&quot;.</li>
  <li>Task-C: From the moment when we pick up the results of call to the ServiceB up-to the end of the ProcessData method.</li>
  </ul>
  <p>As a result instead of having a single &quot;processing unit of work&quot; we have three, where each piece is scheduled for execution based on its positions in the scheduler's queue.</p>
  <p>The problem is that by the time Task-B (for the first piece of work) is put on the scheduler's queue I may have hundreds of Task-A, put there by <code>Parallel.ForEach</code>, and by the time Task-C (for the first piece of work) is placed on the scheduler's queue the situation will be even worse.</p>
  <p>I would like the data pass through as fast as possible, so I need to be able to prioritize Task-C over Task-B over Task-A. What will be the best way to achieve this?</p>
  <p><code>INotifyCompletion</code>, <code>SynchronizationContext</code> come to mind, but it seems to be the &quot;dark corners&quot; of the async/await. ParallelExtensionsExtras has <code>ReprioritizableTaskScheduler</code> and <code>QueuedTaskScheduler</code> with priority queues, but how do I tell async/await to use the desired scheduler?</p>
  <p>John Skeet talks about this problem in his blog: <a href="https://codeblog.jonskeet.uk/2010/11/02/configuring-waiting/" rel="nofollow noreferrer">https://codeblog.jonskeet.uk/2010/11/02/configuring-waiting/</a></p>