title: Experiencing "Connection reset by peer" while using grpc on Mesosphere Marathon
body: <p>I'm getting tons of these errors deploying a grpc (1.5.0) java app on marathon.</p>
  
  <pre><code>Aug 01, 2017 4:49:12 PM io.grpc.netty.NettyServerTransport notifyTerminated
  SEVERE: Transport failed
  java.io.IOException: Connection reset by peer
      at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
      at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
      at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
      at sun.nio.ch.IOUtil.read(IOUtil.java:192)
      at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
      at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)
      at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1100)
      at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:372)
      at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123)
      at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
      at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
      at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
      at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
      at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
      at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
      at java.lang.Thread.run(Thread.java:745)
  </code></pre>
  
  <p>EDIT:</p>
  
  <p>I'm also noticing</p>
  
  <pre><code>[pool-1-thread-4] --- 2017-08-02 12:31:09,465 WARN  e.a.s.grcp.ConnectionStatusServer - requests.size 1
  [nioEventLoopGroup-2-1] --- 2017-08-02 12:31:09,465 DEBUG i.n.h.c.http2.Http2ConnectionHandler - [id: 0x0a5278e3, L:0.0.0.0/0.0.0.0:31169 ! R:/10.124.56.175:47384] Sending GOAWAY failed: lastStreamId '7', errorCode '2', debugData 'Connection reset by peer'. Forcing shutdown of the connection.java.io.IOException: Broken pipe
   ---&gt;   at sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
   ---&gt;   at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
   ---&gt;   at sun.nio.ch.IOUtil.write(IOUtil.java:148)
   ---&gt;   at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:504)
   ---&gt;   at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:432)
   ---&gt;   at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:856)
   ---&gt;   at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:362)
   ---&gt;   at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:823)
   ---&gt;   at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1296)
   ---&gt;   at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
   ---&gt;   at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768)
   ---&gt;   at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749)
   ---&gt;   at io.netty.handler.codec.http2.Http2ConnectionHandler.onError(Http2ConnectionHandler.java:573)
   ---&gt;   at io.grpc.netty.AbstractNettyHandler.exceptionCaught(AbstractNettyHandler.java:84)
   ---&gt;   at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285)
   ---&gt;   at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:264)
   ---&gt;   at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:256)
   ---&gt;   at io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1301)
   ---&gt;   at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285)
   ---&gt;   at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:264)
   ---&gt;   at io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:914)
   ---&gt;   at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:104)
   ---&gt;   at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:145)
   ---&gt;   at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
   ---&gt;   at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
   ---&gt;   at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
   ---&gt;   at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
   ---&gt;   at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
   ---&gt;   at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
   ---&gt;   at java.lang.Thread.run(Thread.java:745)
   ---&gt; 
  </code></pre>