{"action":"created","issue":{"url":"https://api.github.com/repos/netty/netty/issues/5999","repository_url":"https://api.github.com/repos/netty/netty","labels_url":"https://api.github.com/repos/netty/netty/issues/5999/labels{/name}","comments_url":"https://api.github.com/repos/netty/netty/issues/5999/comments","events_url":"https://api.github.com/repos/netty/netty/issues/5999/events","html_url":"https://github.com/netty/netty/issues/5999","id":188341179,"number":5999,"title":"IllegalReferenceCountException with DefaultHttp2ConnectionEncoder#writeData() and PooledSlicedByteBuf","user":{"login":"rkapsi","id":191635,"avatar_url":"https://avatars.githubusercontent.com/u/191635?v=3","gravatar_id":"","url":"https://api.github.com/users/rkapsi","html_url":"https://github.com/rkapsi","followers_url":"https://api.github.com/users/rkapsi/followers","following_url":"https://api.github.com/users/rkapsi/following{/other_user}","gists_url":"https://api.github.com/users/rkapsi/gists{/gist_id}","starred_url":"https://api.github.com/users/rkapsi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rkapsi/subscriptions","organizations_url":"https://api.github.com/users/rkapsi/orgs","repos_url":"https://api.github.com/users/rkapsi/repos","events_url":"https://api.github.com/users/rkapsi/events{/privacy}","received_events_url":"https://api.github.com/users/rkapsi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"Scottmitch","id":7562868,"avatar_url":"https://avatars.githubusercontent.com/u/7562868?v=3","gravatar_id":"","url":"https://api.github.com/users/Scottmitch","html_url":"https://github.com/Scottmitch","followers_url":"https://api.github.com/users/Scottmitch/followers","following_url":"https://api.github.com/users/Scottmitch/following{/other_user}","gists_url":"https://api.github.com/users/Scottmitch/gists{/gist_id}","starred_url":"https://api.github.com/users/Scottmitch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Scottmitch/subscriptions","organizations_url":"https://api.github.com/users/Scottmitch/orgs","repos_url":"https://api.github.com/users/Scottmitch/repos","events_url":"https://api.github.com/users/Scottmitch/events{/privacy}","received_events_url":"https://api.github.com/users/Scottmitch/received_events","type":"User","site_admin":false},"assignees":[{"login":"Scottmitch","id":7562868,"avatar_url":"https://avatars.githubusercontent.com/u/7562868?v=3","gravatar_id":"","url":"https://api.github.com/users/Scottmitch","html_url":"https://github.com/Scottmitch","followers_url":"https://api.github.com/users/Scottmitch/followers","following_url":"https://api.github.com/users/Scottmitch/following{/other_user}","gists_url":"https://api.github.com/users/Scottmitch/gists{/gist_id}","starred_url":"https://api.github.com/users/Scottmitch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Scottmitch/subscriptions","organizations_url":"https://api.github.com/users/Scottmitch/orgs","repos_url":"https://api.github.com/users/Scottmitch/repos","events_url":"https://api.github.com/users/Scottmitch/events{/privacy}","received_events_url":"https://api.github.com/users/Scottmitch/received_events","type":"User","site_admin":false}],"milestone":null,"comments":16,"created_at":"2016-11-09T20:12:22Z","updated_at":"2016-11-12T19:26:49Z","closed_at":null,"body":"I have a custom Channel class that treats each Http2Stream like a connection and it acts like a proxy to translate between H2 and HTTP/1.1. It looks something like this (very similar to Netty's own HttpToHttp2ConnectionHandler):\r\n\r\n```java\r\n// HttpToHttp2ConnectionHandler\r\npublic class HttpToHttp2Handler extends ChannelDuplexHandler {\r\n  \r\n  private static final boolean HACK = true;\r\n  \r\n  private final Http2ConnectionEncoder encoder;\r\n  \r\n  private final Http2Stream stream;\r\n  \r\n  private final ChannelHandlerContext h2callback;\r\n  \r\n  private final Channel h2channel;\r\n  \r\n  public HttpToHttp2Handler(Http2ConnectionEncoder encoder, Http2Stream stream) {\r\n    this.encoder = encoder;\r\n    this.stream = stream;\r\n    \r\n    Http2RemoteFlowController flowController = encoder.flowController();\r\n    this.h2callback = flowController.channelHandlerContext();\r\n    this.h2channel = h2callback.channel();\r\n  }\r\n  \r\n  @Override\r\n  public void flush(ChannelHandlerContext ctx) throws Exception {\r\n    h2channel.flush();\r\n  }\r\n  \r\n  @Override\r\n  public void read(ChannelHandlerContext ctx) throws Exception {\r\n    h2channel.read();\r\n  }\r\n\r\n  @Override\r\n  public void write(ChannelHandlerContext ctx, Object msg, ChannelPromise promise) throws Exception {\r\n    \r\n    SimpleChannelPromiseAggregator promiseAggregator =\r\n        new SimpleChannelPromiseAggregator(promise, ctx.channel(), ctx.executor());\r\n    \r\n    boolean release = true;\r\n    boolean endStream = false;\r\n    \r\n    try {\r\n      \r\n      int streamId = stream.id();\r\n      \r\n      if (msg instanceof HttpMessage) {\r\n        HttpMessage message = (HttpMessage)msg;\r\n        \r\n        Http2Headers http2Headers = HttpConversionUtil.toHttp2Headers(message, true);\r\n        \r\n        if (message instanceof FullHttpMessage) {\r\n          endStream = !((FullHttpMessage)message).content().isReadable();\r\n        }\r\n        \r\n        encoder.writeHeaders(h2callback, streamId, http2Headers, 0, endStream, promiseAggregator.newPromise());\r\n      }\r\n      \r\n      if (!endStream && msg instanceof HttpContent) {\r\n        \r\n        Http2Headers trailers = EmptyHttp2Headers.INSTANCE;\r\n        if (msg instanceof LastHttpContent) {\r\n          \r\n          LastHttpContent lastContent = (LastHttpContent) msg;\r\n          trailers = HttpConversionUtil.toHttp2Headers(lastContent.trailingHeaders(), true);\r\n          \r\n          if (trailers.isEmpty()) {\r\n            endStream = true;\r\n          }\r\n        }\r\n\r\n        ByteBuf content = ((HttpContent) msg).content();\r\n        \r\n        if (HACK) {\r\n          ByteBuf copy = h2callback.alloc().buffer();\r\n          copy.writeBytes(content);\r\n          content = copy;\r\n          \r\n          // content = Unpooled.copiedBuffer(content); // <- works too\r\n          \r\n        } else {\r\n          // We're transferring ownership of the ByteBuf to the encoder\r\n          // and we're not longer responsible for releasing it.\r\n          release = false;\r\n        }\r\n        \r\n        encoder.writeData(h2callback, streamId, content, 0, endStream, promiseAggregator.newPromise());\r\n        \r\n        if (!trailers.isEmpty()) {\r\n          encoder.writeHeaders(h2callback, streamId, trailers, 0, true, promiseAggregator.newPromise());\r\n        }\r\n      }\r\n    } catch (Throwable t) {\r\n      promiseAggregator.setFailure(t);\r\n      \r\n    } finally {\r\n      if (release) {\r\n        ReferenceCountUtil.release(msg);\r\n      }\r\n      \r\n      promiseAggregator.doneAllocatingPromises();\r\n    }\r\n  }\r\n}\r\n```\r\n\r\nPlease notice the `if (HACK) {}` condition. Everything works if I make a copy of the HttpContent's underlying ByteBuf which is a `PooledSlicedByteBuf`. It starts failing with an IllegalReferenceCountException if I try to passthrough the ByteBuf directly and it continues failing if I explicitly `retain()` it to rule out erroneous release calls on my end. All it does is to raise additional leak detector errors.\r\n\r\nThe exact exception I'm getting is this:\r\n\r\n```java\r\nio.netty.util.IllegalReferenceCountException: refCnt: 0\r\n\tat io.netty.buffer.AbstractByteBuf.ensureAccessible(AbstractByteBuf.java:1407) ~[netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1353) ~[netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.buffer.PooledUnsafeDirectByteBuf.nioBuffer(PooledUnsafeDirectByteBuf.java:324) ~[netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.buffer.AbstractUnpooledSlicedByteBuf.nioBuffer(AbstractUnpooledSlicedByteBuf.java:454) ~[netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.buffer.CompositeByteBuf.nioBuffers(CompositeByteBuf.java:1498) ~[netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.buffer.AbstractUnpooledSlicedByteBuf.nioBuffers(AbstractUnpooledSlicedByteBuf.java:460) ~[netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.buffer.AbstractByteBuf.nioBuffers(AbstractByteBuf.java:1203) ~[netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:685) ~[netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:522) ~[netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.handler.ssl.SslHandler.flush(SslHandler.java:490) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:787) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:779) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:760) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:117) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:787) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:779) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:760) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.handler.codec.http2.Http2ConnectionHandler.flush(Http2ConnectionHandler.java:163) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.handler.codec.http2.Http2ConnectionHandler.channelReadComplete(Http2ConnectionHandler.java:464) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:409) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:391) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelReadComplete(AbstractChannelHandlerContext.java:384) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.ChannelInboundHandlerAdapter.channelReadComplete(ChannelInboundHandlerAdapter.java:97) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:409) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:391) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelReadComplete(AbstractChannelHandlerContext.java:384) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.handler.timeout.IdleStateHandler.channelReadComplete(IdleStateHandler.java:275) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:409) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:391) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelReadComplete(AbstractChannelHandlerContext.java:384) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.handler.ssl.SslHandler.channelReadComplete(SslHandler.java:928) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:409) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:391) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelReadComplete(AbstractChannelHandlerContext.java:384) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.ChannelInboundHandlerAdapter.channelReadComplete(ChannelInboundHandlerAdapter.java:97) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:409) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:391) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelReadComplete(AbstractChannelHandlerContext.java:384) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.ChannelInboundHandlerAdapter.channelReadComplete(ChannelInboundHandlerAdapter.java:97) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:409) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:391) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelReadComplete(AbstractChannelHandlerContext.java:384) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelReadComplete(DefaultChannelPipeline.java:1339) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:409) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:391) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelReadComplete(DefaultChannelPipeline.java:932) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:651) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:574) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:488) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:450) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873) [netty-all-4.1.7.Final-SNAPSHOT.jar:4.1.7.Final-SNAPSHOT]\r\n\tat java.lang.Thread.run(Thread.java:745) [?:1.8.0_102]\r\n```\r\n\r\nI haven't been able to reproduce it locally/with clean traffic but it happens pretty quickly in our production environment.\r\n\r\nI'm suspecting it's something in `FlowControlledData#write(...)` or maybe `CoalescingBufferQueue` that doesn't like PooledSlicedByteBufs.\r\n"},"comment":{"url":"https://api.github.com/repos/netty/netty/issues/comments/260142770","html_url":"https://github.com/netty/netty/issues/5999#issuecomment-260142770","issue_url":"https://api.github.com/repos/netty/netty/issues/5999","id":260142770,"user":{"login":"rkapsi","id":191635,"avatar_url":"https://avatars.githubusercontent.com/u/191635?v=3","gravatar_id":"","url":"https://api.github.com/users/rkapsi","html_url":"https://github.com/rkapsi","followers_url":"https://api.github.com/users/rkapsi/followers","following_url":"https://api.github.com/users/rkapsi/following{/other_user}","gists_url":"https://api.github.com/users/rkapsi/gists{/gist_id}","starred_url":"https://api.github.com/users/rkapsi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rkapsi/subscriptions","organizations_url":"https://api.github.com/users/rkapsi/orgs","repos_url":"https://api.github.com/users/rkapsi/repos","events_url":"https://api.github.com/users/rkapsi/events{/privacy}","received_events_url":"https://api.github.com/users/rkapsi/received_events","type":"User","site_admin":false},"created_at":"2016-11-12T19:26:49Z","updated_at":"2016-11-12T19:26:49Z","body":"I made an another experiment and everything works fine if I switch to `-Dio.netty.allocator.type=unpooled`. This is pointing towards PooledUnsafeDirectByteBuf (I also tried `-Dio.netty.noPreferDirect=true` which didn't make a difference in terms of outcome).\r\n\r\n"}}
{"actor":{"display_login":"rkapsi"}}