title: Optimizing the Akka.NET message dispatcher
body: <p>I'm currently trying to find the bottlenecks in our message dispatcher for Akka.NET (port of java/scala actor model framework)
  For those interested, it can be found here: <a href="https://github.com/akkadotnet/akka.net" rel="noreferrer">https://github.com/akkadotnet/akka.net</a></p>
  
  <p>We seem to scale great up to 8 cores, everything seems fine so far.
  However, when running on larger machines, everything falls apart eventually.
  We have tested this on a 16 core machine, and it scales nicely up to a certain point, then suddenly, message throughput is halved.</p>
  
  <p>This image is while profiling on my laptop
  <img src="https://i.stack.imgur.com/DxboR.png" alt="Akka.NET profiling">
  See: <a href="https://i.stack.imgur.com/DxboR.png" rel="noreferrer">http://i.stack.imgur.com/DxboR.png</a> for full image</p>
  
  <p>. 
  Is the bottleneck the Task.Factory.StartNew, or is it the ConcurrentQueue.Enqueue ?
  I'm not sure I read those numbers right.</p>
  
  <p>Here is a breif decription of how our message dispatcher and mailbox works:</p>
  
  <p>Once a message is posted to a mailbox, the mailbox checks if it is currently processing messages.
  if it is processing messages, it simply let the currently running Task consume the new message.</p>
  
  <p>So essentially, we post a message to a ConcurrentQueue, and the current mailbox run will find it.</p>
  
  <p>if the mailbox is currently idle while the message is posted, then we schedule the mailbox using <code>Task.Factory.StartNew(mailboxAction)</code>.</p>
  
  <p>To ensure that only one task is running at any given time for a specific mailbox, we use Interlocked checks to see if the mailbox is bussy or idle.
  The interlock checks works, this is tested extensively so we know we don't start up multiple tasks for the same mailbox.</p>
  
  <p>Any ideas what could cause throughput to break completely on the 16 core machine?
  The same effect does not happen on smaller machines, those stay stable at max throughput when they can not scale any more.</p>
  
  <p>One thing that I have verified on the 16 core machine, is that the mailbox seems to process too fast, depleting all the messages in the actors messagequeue, which will lead a new scheduling of the mailbox once a new message arrives.
  That is, the consumer is faster than the producer.</p>