{"action":"created","issue":{"url":"https://api.github.com/repos/basho/riak-java-client/issues/638","repository_url":"https://api.github.com/repos/basho/riak-java-client","labels_url":"https://api.github.com/repos/basho/riak-java-client/issues/638/labels{/name}","comments_url":"https://api.github.com/repos/basho/riak-java-client/issues/638/comments","events_url":"https://api.github.com/repos/basho/riak-java-client/issues/638/events","html_url":"https://github.com/basho/riak-java-client/issues/638","id":165750974,"number":638,"title":"Overload leads to BlockingOperationException [JIRA: CLIENTS-925]","user":{"login":"hankipanky","id":1611513,"avatar_url":"https://avatars.githubusercontent.com/u/1611513?v=3","gravatar_id":"","url":"https://api.github.com/users/hankipanky","html_url":"https://github.com/hankipanky","followers_url":"https://api.github.com/users/hankipanky/followers","following_url":"https://api.github.com/users/hankipanky/following{/other_user}","gists_url":"https://api.github.com/users/hankipanky/gists{/gist_id}","starred_url":"https://api.github.com/users/hankipanky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hankipanky/subscriptions","organizations_url":"https://api.github.com/users/hankipanky/orgs","repos_url":"https://api.github.com/users/hankipanky/repos","events_url":"https://api.github.com/users/hankipanky/events{/privacy}","received_events_url":"https://api.github.com/users/hankipanky/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/basho/riak-java-client/labels/Bug","name":"Bug","color":"FF0000"},{"url":"https://api.github.com/repos/basho/riak-java-client/labels/JIRA:%20Closed","name":"JIRA: Closed","color":"e11d21"}],"state":"closed","locked":false,"assignee":{"login":"alexmoore","id":1438,"avatar_url":"https://avatars.githubusercontent.com/u/1438?v=3","gravatar_id":"","url":"https://api.github.com/users/alexmoore","html_url":"https://github.com/alexmoore","followers_url":"https://api.github.com/users/alexmoore/followers","following_url":"https://api.github.com/users/alexmoore/following{/other_user}","gists_url":"https://api.github.com/users/alexmoore/gists{/gist_id}","starred_url":"https://api.github.com/users/alexmoore/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexmoore/subscriptions","organizations_url":"https://api.github.com/users/alexmoore/orgs","repos_url":"https://api.github.com/users/alexmoore/repos","events_url":"https://api.github.com/users/alexmoore/events{/privacy}","received_events_url":"https://api.github.com/users/alexmoore/received_events","type":"User","site_admin":false},"assignees":[{"login":"alexmoore","id":1438,"avatar_url":"https://avatars.githubusercontent.com/u/1438?v=3","gravatar_id":"","url":"https://api.github.com/users/alexmoore","html_url":"https://github.com/alexmoore","followers_url":"https://api.github.com/users/alexmoore/followers","following_url":"https://api.github.com/users/alexmoore/following{/other_user}","gists_url":"https://api.github.com/users/alexmoore/gists{/gist_id}","starred_url":"https://api.github.com/users/alexmoore/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexmoore/subscriptions","organizations_url":"https://api.github.com/users/alexmoore/orgs","repos_url":"https://api.github.com/users/alexmoore/repos","events_url":"https://api.github.com/users/alexmoore/events{/privacy}","received_events_url":"https://api.github.com/users/alexmoore/received_events","type":"User","site_admin":false}],"milestone":null,"comments":6,"created_at":"2016-07-15T09:55:58Z","updated_at":"2016-08-30T15:25:24Z","closed_at":"2016-08-29T19:46:48Z","body":"Riak Java Client: 2.0.6\r\nJava version: 1.8.0_74 \r\nRiak version: 2.1.2\r\nOperating system: Linux, Debian 7\r\n\r\nWhen updating objects in Riak under load, the riak-java-client occasionally throws a io.netty.util.concurrent.BlockingOperationException. Since https://github.com/basho/riak-java-client/issues/523 is fixed, this happens only in - what I assume is - an overload-scenario.\r\n\r\nThe exception:\r\n\r\n```\r\n2016-07-13 14:41:12.789  localhost [nioEventLoopGroup-2-2] ERROR\r\ncom.basho.riak.client.core.RiakNode - Operation onException() channel:\r\nid:237445453 localhost:8087 {}\r\nio.netty.util.concurrent.BlockingOperationException:\r\nDefaultChannelPromise at 77ccd827(incomplete)\r\n        at io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:390)\r\n        at io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:157)\r\n        at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:251)\r\n        at io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:129)\r\n        at io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:28)\r\n        at com.basho.riak.client.core.RiakNode.doGetConnection(RiakNode.java:697)\r\n        at com.basho.riak.client.core.RiakNode.getConnection(RiakNode.java:656)\r\n        at com.basho.riak.client.core.RiakNode.execute(RiakNode.java:587)\r\n        at com.basho.riak.client.core.DefaultNodeManager.executeOnNode(DefaultNodeManager.java:91)\r\n        at com.basho.riak.client.core.RiakCluster.execute(RiakCluster.java:322)\r\n        at com.basho.riak.client.core.RiakCluster.execute(RiakCluster.java:240)\r\n        at com.basho.riak.client.api.commands.kv.StoreValue.executeAsync(StoreValue.java:117)\r\n        at com.basho.riak.client.api.commands.kv.UpdateValue$1.handle(UpdateValue.java:182)\r\n        at com.basho.riak.client.api.commands.ListenableFuture.notifyListeners(ListenableFuture.java:78)\r\n        at com.basho.riak.client.api.commands.CoreFutureAdapter.handle(CoreFutureAdapter.java:120)\r\n        at com.basho.riak.client.core.FutureOperation.fireListeners(FutureOperation.java:176)\r\n        at com.basho.riak.client.core.FutureOperation.setComplete(FutureOperation.java:224)\r\n        at com.basho.riak.client.core.RiakNode.onSuccess(RiakNode.java:878)\r\n        at com.basho.riak.client.core.netty.RiakResponseHandler.channelRead(RiakResponseHandler.java:30)\r\n        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)\r\n        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)\r\n        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:276)\r\n        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:263)\r\n        at io.netty.handler.codec.ByteToMessageCodec.channelRead(ByteToMessageCodec.java:103)\r\n        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)\r\n        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)\r\n        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)\r\n        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)\r\n        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)\r\n        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)\r\n        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)\r\n        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)\r\n        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)\r\n        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)\r\n        at java.lang.Thread.run(Thread.java:745)\r\n```\r\n\r\nThen shortly thereafter:\r\n\r\n```\r\n2016-07-13 14:41:12.820  localhost [nioEventLoopGroup-2-2] ERROR com.basho.riak.client.core.RiakNode - Write failed on RiakNode localhost:8087 id: 237445453; cause: {} \r\njava.nio.channels.ClosedChannelException: null\r\n2016-07-13 14:41:12.843  localhost [nioEventLoopGroup-2-2] ERROR com.basho.riak.client.core.RiakNode - Write failed on RiakNode localhost:8087 id: 237445453; cause: {} \r\njava.nio.channels.ClosedChannelException: null\r\n```\r\n\r\nThe result is a hanging thread, never returning from the call to `RiakClient.execute()`. It would be better, if in such a case, the client threw a RiakException (as javadoc says).\r\n\r\nRiak's error.log shows nothing about this.\r\n\r\nIn the meantime, how do I protect against overloading Riak? Setting `RiakNode.Builder().withMaxConnections()` to something like 60 or below (on my single-node dev system here) seems to work: Attempts to open more connections lead to a `com.basho.riak.client.core.NoNodesAvailableException`. What is considered overload?"},"comment":{"url":"https://api.github.com/repos/basho/riak-java-client/issues/comments/243477727","html_url":"https://github.com/basho/riak-java-client/issues/638#issuecomment-243477727","issue_url":"https://api.github.com/repos/basho/riak-java-client/issues/638","id":243477727,"user":{"login":"hankipanky","id":1611513,"avatar_url":"https://avatars.githubusercontent.com/u/1611513?v=3","gravatar_id":"","url":"https://api.github.com/users/hankipanky","html_url":"https://github.com/hankipanky","followers_url":"https://api.github.com/users/hankipanky/followers","following_url":"https://api.github.com/users/hankipanky/following{/other_user}","gists_url":"https://api.github.com/users/hankipanky/gists{/gist_id}","starred_url":"https://api.github.com/users/hankipanky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hankipanky/subscriptions","organizations_url":"https://api.github.com/users/hankipanky/orgs","repos_url":"https://api.github.com/users/hankipanky/repos","events_url":"https://api.github.com/users/hankipanky/events{/privacy}","received_events_url":"https://api.github.com/users/hankipanky/received_events","type":"User","site_admin":false},"created_at":"2016-08-30T15:25:24Z","updated_at":"2016-08-30T15:25:24Z","body":"Thanks so much. I was following the progress and looked at the changeset - looks great. I'll report back after testing. you guys rock!"}}
{"actor":{"display_login":"hankipanky"}}