{"action":"created","issue":{"url":"https://api.github.com/repos/netty/netty/issues/6133","repository_url":"https://api.github.com/repos/netty/netty","labels_url":"https://api.github.com/repos/netty/netty/issues/6133/labels{/name}","comments_url":"https://api.github.com/repos/netty/netty/issues/6133/comments","events_url":"https://api.github.com/repos/netty/netty/issues/6133/events","html_url":"https://github.com/netty/netty/issues/6133","id":195650547,"number":6133,"title":"ClassCastException while accessing PooledHeapByteBuf array","user":{"login":"ppersad","id":5296246,"avatar_url":"https://avatars.githubusercontent.com/u/5296246?v=3","gravatar_id":"","url":"https://api.github.com/users/ppersad","html_url":"https://github.com/ppersad","followers_url":"https://api.github.com/users/ppersad/followers","following_url":"https://api.github.com/users/ppersad/following{/other_user}","gists_url":"https://api.github.com/users/ppersad/gists{/gist_id}","starred_url":"https://api.github.com/users/ppersad/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ppersad/subscriptions","organizations_url":"https://api.github.com/users/ppersad/orgs","repos_url":"https://api.github.com/users/ppersad/repos","events_url":"https://api.github.com/users/ppersad/events{/privacy}","received_events_url":"https://api.github.com/users/ppersad/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-12-14T21:28:14Z","updated_at":"2016-12-16T22:24:55Z","closed_at":null,"body":"### Context\r\n\r\nAfter our server has been running for a few weeks, we notice that some TCP connections are broken in one direction (we can receive messages, but not send them).  From looking at the stack traces, it seems like the \"memory\" byte[] field of the PooledHeapByteBuf has been replaced by something else.  After this exception appears on a particular worker thread, that thread then remains broken and we cannot send any messages to channels associated with that thread.  The exceptions we get are (note that application specific portions of this stack trace have been removed or re-named in order to comply with IT security requirements):\r\n\r\n```\r\n15:57:33,328  WARN            TcpSender [ tcpWorkerThreadPoolExecutor-11] Failed to send message to channelId: [69551bb9] io.netty.handler.codec.EncoderException: java.lang.ArrayStoreException\r\n        at io.netty.handler.codec.MessageToByteEncoder.write(MessageToByteEncoder.java:125) ~[netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:748) ~[netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:740) ~[netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:826) ~[netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:733) ~[netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at com.example.jmx.MessageTracker.write(MessageTracker.java:19) ~[****-*.*.*.jar:?]\r\n        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:748) ~[netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:740) ~[netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:826) ~[netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:733) ~[netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.handler.codec.MessageToByteEncoder.write(MessageToByteEncoder.java:113) ~[netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:748) ~[netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:811) ~[netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:824) ~[netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:804) ~[netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:841) ~[netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1032) ~[netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:296) ~[netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at com.example.transport.TcpSender.sendMessage(TcpSender.java:158) [****-*.*.*.jar:?]\r\n        <Application code which generates message to send>\r\nCaused by: java.lang.ArrayStoreException\r\n        at java.lang.System.arraycopy(Native Method) ~[?:1.8.0_66]\r\n        at io.netty.buffer.PooledHeapByteBuf.setBytes(PooledHeapByteBuf.java:238) ~[netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1037) ~[netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1044) ~[netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.handler.codec.compression.JdkZlibEncoder.encode(JdkZlibEncoder.java:219) ~[netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.handler.codec.compression.JdkZlibEncoder.encode(JdkZlibEncoder.java:33) ~[netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.handler.codec.MessageToByteEncoder.write(MessageToByteEncoder.java:107) ~[netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        ... 64 more\r\n\r\n15:57:47,932  WARN            TcpSender [ tcpWorkerThreadPoolExecutor-11] Failed to send message to channelId: [69551bb9] io.netty.handler.codec.EncoderException: java.lang.ClassCastException: java.util.HashMap$Node cannot be cast to [B\r\n        at io.netty.handler.codec.MessageToByteEncoder.write(MessageToByteEncoder.java:125) ~[netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:748) [netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:740) [netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:826) [netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:733) [netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at com.example.jmx.MessageTracker.write(MessageTracker.java:19) ~[****-*.*.*.jar:?]\r\n        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:748) [netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:740) [netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:826) [netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:733) [netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.handler.codec.MessageToByteEncoder.write(MessageToByteEncoder.java:113) [netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:748) [netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:740) [netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:38) [netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:1101) [netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1148) [netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1090) [netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:418) [netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:440) [netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873) [netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_66]\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_66]\r\n        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_66]\r\nCaused by: java.lang.ClassCastException: java.util.HashMap$Node cannot be cast to [B\r\n        at io.netty.buffer.PooledHeapByteBuf.array(PooledHeapByteBuf.java:319) ~[netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.handler.codec.compression.JdkZlibEncoder.deflate(JdkZlibEncoder.java:315) ~[netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.handler.codec.compression.JdkZlibEncoder.encode(JdkZlibEncoder.java:229) ~[netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.handler.codec.compression.JdkZlibEncoder.encode(JdkZlibEncoder.java:33) ~[netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        at io.netty.handler.codec.MessageToByteEncoder.write(MessageToByteEncoder.java:107) ~[netty-all-4.1.5.Final.jar:4.1.5.Final]\r\n        ... 23 more\r\n```\r\n\r\nThe cause of the first exception is pretty clear from looking at the second exception.  Somehow the byte[] in the \"memory\" field has been replaced by a HashMap$Node.  On other broken threads I've seen ClassCastExceptions for other strange things like a AbstractQueuedSynchronizer$Node or for a anonymous Runnable instance from half way across the system (that never gets anywhere near the netty code).  I'm somewhat at a loss for how it is possible for this to happen.  My best guess at this point is that an unsafe memory copy somewhere in netty is incorrectly calculating addresses (or the GC is moving things between address calculation and performing the copy).  However this is relatively unsatisfying as an explanation given that this is a Heap buffer not a Direct buffer.\r\n\r\n\r\n### Steps to reproduce\r\n\r\nUnfortunately this issue only seems to be reproducible on a system which has been running for a couple weeks under load (at least that's the only time we've been able to encounter it).\r\n\r\n### Netty version\r\n\r\n4.1.5\r\n\r\n### JVM version (e.g. `java -version`)\r\n\r\njava version \"1.8.0_66\"\r\nJava(TM) SE Runtime Environment (build 1.8.0_66-b17)\r\nJava HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)\r\n\r\n### OS version (e.g. `uname -a`)\r\n\r\nCentOS release 6.7 (Final)\r\n$ uname -a\r\nLinux 2.6.32-573.18.1.el6.x86_64 GNU/Linux"},"comment":{"url":"https://api.github.com/repos/netty/netty/issues/comments/267710856","html_url":"https://github.com/netty/netty/issues/6133#issuecomment-267710856","issue_url":"https://api.github.com/repos/netty/netty/issues/6133","id":267710856,"user":{"login":"ppersad","id":5296246,"avatar_url":"https://avatars.githubusercontent.com/u/5296246?v=3","gravatar_id":"","url":"https://api.github.com/users/ppersad","html_url":"https://github.com/ppersad","followers_url":"https://api.github.com/users/ppersad/followers","following_url":"https://api.github.com/users/ppersad/following{/other_user}","gists_url":"https://api.github.com/users/ppersad/gists{/gist_id}","starred_url":"https://api.github.com/users/ppersad/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ppersad/subscriptions","organizations_url":"https://api.github.com/users/ppersad/orgs","repos_url":"https://api.github.com/users/ppersad/repos","events_url":"https://api.github.com/users/ppersad/events{/privacy}","received_events_url":"https://api.github.com/users/ppersad/received_events","type":"User","site_admin":false},"created_at":"2016-12-16T22:24:55Z","updated_at":"2016-12-16T22:24:55Z","body":"@Scottmitch If we are able to reproduce the issue in our perf testing environment, then I can try with a forked version of netty that has the changes you mentioned.  Unfortunately our prod environment is too tightly controlled to allow that kind of change there."}}
{"actor":{"display_login":"ppersad"}}