title: TPL Complete vs Completion
body: <p>I need to import customer related data from legacy DB and perform several transformations during the process. This means a single entry needs to perform additional "events" (synchronize products, create invoices, etc.). </p>
  
  <p>My initial solution was a simple parallel approach. 
  It works okay, but <em>sometimes</em> it has issues. If the currently processed customers need to wait for the same type of events, their processing queues might got stuck and eventually time out, causing every underlying events to fail too (they depend on the one which failed). It doesn't happen all the time, yet it's annoying.</p>
  
  <p>So I got another idea, work in batches. I mean not only limiting the number of customers being processed at the same time, but also the number of the events which are broadcasted to the queues. While searching around for ideas, I found <a href="https://stackoverflow.com/questions/11564506/nesting-await-in-parallel-foreach">this</a> answer, which points to the <a href="https://docs.microsoft.com/en-us/dotnet/standard/parallel-programming/dataflow-task-parallel-library" rel="nofollow noreferrer">TPL DataFlow</a>.</p>
  
  <p>I made a skeleton to get familiar with it. I set up a simple pipeline, but I'm a bit confused about the usage of <code>Complete()</code> and awaiting <code>Completion()</code>. </p>
  
  <p>The steps are the following</p>
  
  <ol>
  <li>Make a list of numbers (the ids of the customers to be imported) - this is outside the import logic, it just there to be able to trigger the rest of the logic</li>
  <li>Create a <code>BatchBlock</code> (to be able to limit the number of customers to be processed at the same time)</li>
  <li>Create a single <code>MyClass1</code> item based on the id (<code>TransformBlock&lt;int, MyClass1&gt;</code>)</li>
  <li>Perform some logic and generate a collection of <code>MyClass2</code> (<code>TransformManyBlock&lt;MyClass1, MyClass2&gt;</code>) - as example, sleep for 1 second</li>
  <li>Perform some logic on every item of the collection (<code>ActionBlock&lt;MyClass2&gt;</code>) - as example, sleep for 1 second</li>
  </ol>
  
  <p>Here's the full code:</p>
  
  <pre><code>public static class Program
  {
      private static void Main(string[] args)
      {
          var batchBlock = new BatchBlock&lt;int&gt;(2);
          for (var i = 1; i &lt; 10; i++)
          {
              batchBlock.Post(i);
          }
  
  
          batchBlock.Complete();
          while (batchBlock.TryReceive(null, out var ids))
          {
              var transformBlock = new TransformBlock&lt;int, MyClass1&gt;(delegate (int id)
              {
                  Console.WriteLine($"TransformBlock(id: {id})");
                  return new MyClass1(id, "Star Wars");
              });
              var transformManyBlock = new TransformManyBlock&lt;MyClass1, MyClass2&gt;(delegate (MyClass1 myClass1)
              {
                  Console.WriteLine($"TransformManyBlock(myClass1: {myClass1.Id}|{myClass1.Value})");
                  Thread.Sleep(1000);
                  return GetMyClass22Values(myClass1);
              });
  
              var actionBlock = new ActionBlock&lt;MyClass2&gt;(delegate (MyClass2 myClass2)
              {
                  Console.WriteLine($"ActionBlock(myClass2: {myClass2.Id}|{myClass2.Value})");
                  Thread.Sleep(1000);
              });
              transformBlock.LinkTo(transformManyBlock);
              transformManyBlock.LinkTo(actionBlock);
              foreach (var id in ids)
              {
                  transformBlock.Post(id);
              }
  
              // this is the point when I'm not 100% sure
  
              //transformBlock.Complete();
              //transformManyBlock.Complete();
              //transformManyBlock.Completion.Wait();
              actionBlock.Complete();
              actionBlock.Completion.Wait();
          }
  
          Console.WriteLine();
          Console.WriteLine("Press any key to continue...");
          Console.ReadKey();
      }
  
      private static IEnumerable&lt;MyClass2&gt; GetMyClass22Values(MyClass1 myClass1)
      {
          return new List&lt;MyClass2&gt;
                 {
                     new MyClass2(1, myClass1.Id+ " did this"),
                     new MyClass2(2, myClass1.Id+ " did that"),
                     new MyClass2(3, myClass1.Id+ " did this again")
                 };
      }
  }
  
  public class MyClass1
  {
      public MyClass1(int id, string value)
      {
          Id = id;
          Value = value;
      }
  
      public int Id { get; set; }
  
      public string Value { get; set; }
  }
  
  public class MyClass2
  {
      public MyClass1(int id, string value)
      {
          Id = id;
          Value = value;
      }
  
      public int Id { get; set; }
  
      public string Value { get; set; }
  }
  </code></pre>
  
  <p>So the point I struggle with is the end, where I'd need to call <code>Complete()</code> or wait for <code>Completion</code>. I can't seem to find the right combination. I'd like to see an output as follows:</p>
  
  <pre><code>TransformBlock(id: 1)
  TransformBlock(id: 2)
  TransformManyBlock(myClass1: 1|Star Wars)
  TransformManyBlock(myClass1: 2|Star Wars)
  ActionBlock(myClass2: 1|1 did this)
  ActionBlock(myClass2: 2|1 did that)
  ActionBlock(myClass2: 3|1 did this again)
  ActionBlock(myClass2: 1|2 did this)
  ActionBlock(myClass2: 2|2 did that)
  ActionBlock(myClass2: 3|2 did this again)
  TransformBlock(id: 3)
  TransformBlock(id: 4)
  TransformManyBlock(myClass1: 3|Star Wars)
  TransformManyBlock(myClass1: 4|Star Wars)
  ActionBlock(myClass2: 1|3 did this)
  ActionBlock(myClass2: 2|3 did that)
  ActionBlock(myClass2: 3|3 did this again)
  ActionBlock(myClass2: 1|4 did this)
  ActionBlock(myClass2: 2|4 did that)
  ActionBlock(myClass2: 3|4 did this again)
  
  [the rest of the items]
  
  
  Press any key to exit...   
  </code></pre>
  
  <p>Anyone can point me to the right direction?</p>