title: Tuning Netty on 32 Core / 10Gbit Hosts
body: <p>Netty Server streams to a Netty client (point to point, 1 to 1):</p>
  <h2>Good</h2>
  <ul>
  <li>case: Server and Client are both <code>12 cores</code>, <code>1Gbit NIC</code> =&gt; going at the steady rate of 300K 200 byte messages per second</li>
  </ul>
  <h2>Not So Good</h2>
  <ul>
  <li>case: Server and Client are both <code>32 cores</code>, <code>10Gbit NIC</code> =&gt; (same code) starting at 130K/s degrading down to hundreds per second within minutes</li>
  </ul>
  <h2>Observations</h2>
  <ul>
  <li><p>Netperf shows that the &quot;bad&quot; environment is actually quite excellent ( can stream 600MB/s steady for a half an hour ).</p>
  </li>
  <li><p>It does not seem to be a client issue, since if I swap the client to a <em>known</em> good client (wrote it in C) that sets a max OS's <code>SO_RCVBUF</code> and does nothing but reads byte[]s and ignores them =&gt; the behavior is still the same.</p>
  </li>
  <li><p>Performance degradation starts before a high write watermark ( 200MB, but tried others ) is reached</p>
  </li>
  <li><p>Heap feels up quickly, and of course once reaches the max, GC kicks in locking the world, but that happens way after the &quot;bad&quot; symptoms surface. On a &quot;good&quot; environment heap stays steady somewhere at 1Gb, where it logically, given the configs, should be.</p>
  </li>
  <li><p>One thing that I noticed: <em>most of the 32 cores are utilized</em> while Netty Server streams, which I tried to limit by setting all the Boss/NioWorker threads to <code>1</code> (although there is a single channel anyway, but just in case):</p>
  </li>
  </ul>
  <pre class="lang-scala prettyprint-override"><code>val bootstrap = new ServerBootstrap(
    new NioServerSocketChannelFactory (
      Executors.newFixedThreadPool( 1 ),
      Executors.newFixedThreadPool( 1 ), 1 ) )
  
  // 1 thread max, memory limitation: 1GB by channel, 2GB global, 100ms of timeout for an inactive thread
  val pipelineExecutor = new OrderedMemoryAwareThreadPoolExecutor(
                  1, 1 *1024 *1024 *1024, 2 *1024 *1024 *1024, 100, TimeUnit.MILLISECONDS,
    Executors.defaultThreadFactory() )
  
  bootstrap.setPipelineFactory(
    new ChannelPipelineFactory {
      def getPipeline = {
        val pipeline = Channels.pipeline( serverHandlers.toArray : _* )
        pipeline.addFirst( &quot;pipelineExecutor&quot;, new ExecutionHandler( pipelineExecutor ) )
        pipeline
      }
  } )
  </code></pre>
  <p>But that does not limit the number of cores used =&gt; still most of the cores are utilized. I understand that Netty tries to round robin worker tasks, but have a suspicion that 32 cores &quot;at once&quot; may be just too much for the NIC to handle.</p>
  <h2>Question(s)</h2>
  <ol>
  <li>Suggestions on the degrading performance?</li>
  <li>How do I limit the number of cores used by Netty (without of course going the OIO route)?</li>
  </ol>
  <p><em>side notes: would've loved to discuss it on Netty's mailing list, but it is closed. tried Netty's IRC, but it is dead</em></p>