{"action":"created","issue":{"url":"https://api.github.com/repos/hazelcast/hazelcast/issues/9215","repository_url":"https://api.github.com/repos/hazelcast/hazelcast","labels_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/9215/labels{/name}","comments_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/9215/comments","events_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/9215/events","html_url":"https://github.com/hazelcast/hazelcast/issues/9215","id":187174014,"number":9215,"title":"Hundreds of threads stuck in state BLOCKED","user":{"login":"royaldark","id":1669397,"avatar_url":"https://avatars.githubusercontent.com/u/1669397?v=3","gravatar_id":"","url":"https://api.github.com/users/royaldark","html_url":"https://github.com/royaldark","followers_url":"https://api.github.com/users/royaldark/followers","following_url":"https://api.github.com/users/royaldark/following{/other_user}","gists_url":"https://api.github.com/users/royaldark/gists{/gist_id}","starred_url":"https://api.github.com/users/royaldark/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/royaldark/subscriptions","organizations_url":"https://api.github.com/users/royaldark/orgs","repos_url":"https://api.github.com/users/royaldark/repos","events_url":"https://api.github.com/users/royaldark/events{/privacy}","received_events_url":"https://api.github.com/users/royaldark/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-11-03T19:58:10Z","updated_at":"2016-11-07T04:12:04Z","closed_at":null,"body":"We've been experiencing issues in our cluster where individual servers become unresponsive after an indeterminate amount of time. We use Hazelcast heavily, and running `jstack` on an unresponsive process shows hundreds of threads in `BLOCKED` state waiting for various Hazelcast operations to complete. Other, unaffected, machines in the cluster do not show this behavior.\r\n\r\nExample stack traces:\r\n```\r\nThread 5377: (state = BLOCKED)\r\n - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)\r\n - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)\r\n - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=78, line=2078 (Compiled frame)\r\n - java.util.concurrent.LinkedBlockingQueue.poll(long, java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)\r\n - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066 (Compiled frame)\r\n - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Compiled frame)\r\n - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)\r\n - java.lang.Thread.run() @bci=11, line=745 (Compiled frame)\r\n - com.hazelcast.util.executor.HazelcastManagedThread.executeRun() @bci=1, line=76 (Interpreted frame)\r\n - com.hazelcast.util.executor.HazelcastManagedThread.run() @bci=5, line=92 (Interpreted frame)\r\n```\r\n\r\n```\r\nThread 5285: (state = BLOCKED)\r\n - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)\r\n - java.util.concurrent.locks.LockSupport.park() @bci=5, line=304 (Compiled frame)\r\n - com.hazelcast.spi.impl.AbstractInvocationFuture.get() @bci=24, line=156 (Compiled frame)\r\n - com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(com.hazelcast.nio.serialization.Data, com.hazelcast.map.impl.operation.MapOperation) @bci=59, line=376 (Compiled frame)\r\n - com.hazelcast.map.impl.proxy.MapProxySupport.getInternal(com.hazelcast.nio.serialization.Data) @bci=47, line=307 (Compiled frame)\r\n - com.hazelcast.map.impl.proxy.MapProxyImpl.get(java.lang.Object) @bci=20, line=94 (Compiled frame)\r\n```\r\n\r\n```\r\nThread 4978: (state = BLOCKED)\r\n - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)\r\n - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)\r\n - java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=160, line=460 (Compiled frame)\r\n - java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=362 (Compiled frame)\r\n - java.util.concurrent.SynchronousQueue.poll(long, java.util.concurrent.TimeUnit) @bci=11, line=941 (Compiled frame)\r\n - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066 (Compiled frame)\r\n - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Compiled frame)\r\n - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)\r\n - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)\r\n - com.hazelcast.util.executor.HazelcastManagedThread.executeRun() @bci=1, line=76 (Interpreted frame)\r\n - com.hazelcast.util.executor.HazelcastManagedThread.run() @bci=5, line=92 (Interpreted frame)\r\n```\r\n\r\nWe are running Hazelcast 3.7.2 on Java 1.8u45."},"comment":{"url":"https://api.github.com/repos/hazelcast/hazelcast/issues/comments/258743641","html_url":"https://github.com/hazelcast/hazelcast/issues/9215#issuecomment-258743641","issue_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/9215","id":258743641,"user":{"login":"pveentjer","id":105243,"avatar_url":"https://avatars.githubusercontent.com/u/105243?v=3","gravatar_id":"","url":"https://api.github.com/users/pveentjer","html_url":"https://github.com/pveentjer","followers_url":"https://api.github.com/users/pveentjer/followers","following_url":"https://api.github.com/users/pveentjer/following{/other_user}","gists_url":"https://api.github.com/users/pveentjer/gists{/gist_id}","starred_url":"https://api.github.com/users/pveentjer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pveentjer/subscriptions","organizations_url":"https://api.github.com/users/pveentjer/orgs","repos_url":"https://api.github.com/users/pveentjer/repos","events_url":"https://api.github.com/users/pveentjer/events{/privacy}","received_events_url":"https://api.github.com/users/pveentjer/received_events","type":"User","site_admin":false},"created_at":"2016-11-07T04:12:04Z","updated_at":"2016-11-07T04:12:04Z","body":"Can you enable diagnostics. This is meant to run in production, so it has a very low performance impact (we even have it enabled while doing benchmarking).\r\n\r\n```\r\n-Dhazelcast.diagnostics.enabled=true\r\n-Dhazelcast.diagnostics.metric.level=info\r\n-Dhazelcast.diagnostics.invocation.sample.period.seconds=30\r\n-Dhazelcast.diagnostics.pending.invocations.period.seconds=30\r\n-Dhazelcast.diagnostics.slowoperations.period.seconds=30\r\n```\r\n\r\nOn the client side the following parameters need to be added\r\n```\r\n-Dhazelcast.diagnostics.enabled=true\r\n-Dhazelcast.diagnostics.metric.level=info\r\n```\r\n\r\nThis will create a diagnostics file which contains a lot of valuable information. \r\n\r\nCan you also send the whole threaddump of all threads?"}}
{"actor":{"display_login":"pveentjer"}}