{"action":"created","issue":{"url":"https://api.github.com/repos/hazelcast/hazelcast/issues/13725","repository_url":"https://api.github.com/repos/hazelcast/hazelcast","labels_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/13725/labels{/name}","comments_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/13725/comments","events_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/13725/events","html_url":"https://github.com/hazelcast/hazelcast/issues/13725","id":358625739,"node_id":"MDU6SXNzdWUzNTg2MjU3Mzk=","number":13725,"title":"RejectedExecutionException on shutdownCluster","user":{"login":"Danny-Hazelcast","id":5988678,"node_id":"MDQ6VXNlcjU5ODg2Nzg=","avatar_url":"https://avatars1.githubusercontent.com/u/5988678?v=4","gravatar_id":"","url":"https://api.github.com/users/Danny-Hazelcast","html_url":"https://github.com/Danny-Hazelcast","followers_url":"https://api.github.com/users/Danny-Hazelcast/followers","following_url":"https://api.github.com/users/Danny-Hazelcast/following{/other_user}","gists_url":"https://api.github.com/users/Danny-Hazelcast/gists{/gist_id}","starred_url":"https://api.github.com/users/Danny-Hazelcast/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Danny-Hazelcast/subscriptions","organizations_url":"https://api.github.com/users/Danny-Hazelcast/orgs","repos_url":"https://api.github.com/users/Danny-Hazelcast/repos","events_url":"https://api.github.com/users/Danny-Hazelcast/events{/privacy}","received_events_url":"https://api.github.com/users/Danny-Hazelcast/received_events","type":"User","site_admin":false},"labels":[{"id":137752575,"node_id":"MDU6TGFiZWwxMzc3NTI1NzU=","url":"https://api.github.com/repos/hazelcast/hazelcast/labels/Team:%20Core","name":"Team: Core","color":"fef2c0","default":false},{"id":1740690,"node_id":"MDU6TGFiZWwxNzQwNjkw","url":"https://api.github.com/repos/hazelcast/hazelcast/labels/Type:%20Defect","name":"Type: Defect","color":"eb6420","default":false}],"state":"open","locked":false,"assignee":{"login":"ahmetmircik","id":2960360,"node_id":"MDQ6VXNlcjI5NjAzNjA=","avatar_url":"https://avatars2.githubusercontent.com/u/2960360?v=4","gravatar_id":"","url":"https://api.github.com/users/ahmetmircik","html_url":"https://github.com/ahmetmircik","followers_url":"https://api.github.com/users/ahmetmircik/followers","following_url":"https://api.github.com/users/ahmetmircik/following{/other_user}","gists_url":"https://api.github.com/users/ahmetmircik/gists{/gist_id}","starred_url":"https://api.github.com/users/ahmetmircik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ahmetmircik/subscriptions","organizations_url":"https://api.github.com/users/ahmetmircik/orgs","repos_url":"https://api.github.com/users/ahmetmircik/repos","events_url":"https://api.github.com/users/ahmetmircik/events{/privacy}","received_events_url":"https://api.github.com/users/ahmetmircik/received_events","type":"User","site_admin":false},"assignees":[{"login":"ahmetmircik","id":2960360,"node_id":"MDQ6VXNlcjI5NjAzNjA=","avatar_url":"https://avatars2.githubusercontent.com/u/2960360?v=4","gravatar_id":"","url":"https://api.github.com/users/ahmetmircik","html_url":"https://github.com/ahmetmircik","followers_url":"https://api.github.com/users/ahmetmircik/followers","following_url":"https://api.github.com/users/ahmetmircik/following{/other_user}","gists_url":"https://api.github.com/users/ahmetmircik/gists{/gist_id}","starred_url":"https://api.github.com/users/ahmetmircik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ahmetmircik/subscriptions","organizations_url":"https://api.github.com/users/ahmetmircik/orgs","repos_url":"https://api.github.com/users/ahmetmircik/repos","events_url":"https://api.github.com/users/ahmetmircik/events{/privacy}","received_events_url":"https://api.github.com/users/ahmetmircik/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/hazelcast/hazelcast/milestones/106","html_url":"https://github.com/hazelcast/hazelcast/milestone/106","labels_url":"https://api.github.com/repos/hazelcast/hazelcast/milestones/106/labels","id":2867560,"node_id":"MDk6TWlsZXN0b25lMjg2NzU2MA==","number":106,"title":"3.11","description":"","creator":{"login":"jerrinot","id":158619,"node_id":"MDQ6VXNlcjE1ODYxOQ==","avatar_url":"https://avatars1.githubusercontent.com/u/158619?v=4","gravatar_id":"","url":"https://api.github.com/users/jerrinot","html_url":"https://github.com/jerrinot","followers_url":"https://api.github.com/users/jerrinot/followers","following_url":"https://api.github.com/users/jerrinot/following{/other_user}","gists_url":"https://api.github.com/users/jerrinot/gists{/gist_id}","starred_url":"https://api.github.com/users/jerrinot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jerrinot/subscriptions","organizations_url":"https://api.github.com/users/jerrinot/orgs","repos_url":"https://api.github.com/users/jerrinot/repos","events_url":"https://api.github.com/users/jerrinot/events{/privacy}","received_events_url":"https://api.github.com/users/jerrinot/received_events","type":"User","site_admin":false},"open_issues":81,"closed_issues":604,"state":"open","created_at":"2017-10-26T14:31:27Z","updated_at":"2018-09-21T13:35:18Z","due_on":"2018-10-12T07:00:00Z","closed_at":null},"comments":6,"created_at":"2018-09-10T13:35:01Z","updated_at":"2018-09-21T13:48:22Z","closed_at":null,"author_association":"MEMBER","body":"i try to shutdown my cluster but hazelcast cluster gives RejectedExecutionException,\r\nand will not do a graceful shutdown when requested.\r\n\r\n```\r\njava.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4a4416bc rejected from ThreadPoolExecutor{name='hz:async'}\r\n\tat java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2070)\r\n\tat java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:828)\r\n\tat java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1382)\r\n\tat java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)\r\n\tat com.hazelcast.instance.Node.callGracefulShutdownAwareServices(Node.java:472)\r\n\tat com.hazelcast.instance.Node.shutdown(Node.java:429)\r\n\tat com.hazelcast.instance.LifecycleServiceImpl.shutdown(LifecycleServiceImpl.java:98)\r\n\tat com.hazelcast.instance.LifecycleServiceImpl.shutdown(LifecycleServiceImpl.java:81)\r\n\tat com.hazelcast.internal.cluster.impl.ClusterServiceImpl.shutdownNodes(ClusterServiceImpl.java:973)\r\n\tat com.hazelcast.internal.cluster.impl.ClusterServiceImpl.shutdownCluster(ClusterServiceImpl.java:943)\r\n\tat com.hazelcast.internal.cluster.impl.ClusterServiceImpl.shutdown(ClusterServiceImpl.java:923)\r\n\tat hzcmd.ops.ClusterShutDown.timeStep(ClusterShutDown.java:8)\r\n\r\n```\r\n\r\n\r\nfor example in another test i see a RejectedExecutionException from Durable Executor submit,\r\nwhich looks ok, as it some operation that is rejected as the cluster is shutting down.\r\n\r\n```\r\njava.util.concurrent.RejectedExecutionException\r\n\tat com.hazelcast.durableexecutor.impl.DurableExecutorServiceProxy.throwNotActiveException(DurableExecutorServiceProxy.java:244)\r\n\tat com.hazelcast.spi.AbstractDistributedObject.lifecycleCheck(AbstractDistributedObject.java:100)\r\n\tat com.hazelcast.spi.AbstractDistributedObject.getNodeEngine(AbstractDistributedObject.java:94)\r\n\tat com.hazelcast.spi.AbstractDistributedObject.invokeOnPartition(AbstractDistributedObject.java:72)\r\n\tat com.hazelcast.durableexecutor.impl.DurableExecutorServiceProxy.submitToPartition(DurableExecutorServiceProxy.java:263)\r\n\tat com.hazelcast.durableexecutor.impl.DurableExecutorServiceProxy.submit(DurableExecutorServiceProxy.java:154)\r\n```"},"comment":{"url":"https://api.github.com/repos/hazelcast/hazelcast/issues/comments/423536739","html_url":"https://github.com/hazelcast/hazelcast/issues/13725#issuecomment-423536739","issue_url":"https://api.github.com/repos/hazelcast/hazelcast/issues/13725","id":423536739,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMzUzNjczOQ==","user":{"login":"ahmetmircik","id":2960360,"node_id":"MDQ6VXNlcjI5NjAzNjA=","avatar_url":"https://avatars2.githubusercontent.com/u/2960360?v=4","gravatar_id":"","url":"https://api.github.com/users/ahmetmircik","html_url":"https://github.com/ahmetmircik","followers_url":"https://api.github.com/users/ahmetmircik/followers","following_url":"https://api.github.com/users/ahmetmircik/following{/other_user}","gists_url":"https://api.github.com/users/ahmetmircik/gists{/gist_id}","starred_url":"https://api.github.com/users/ahmetmircik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ahmetmircik/subscriptions","organizations_url":"https://api.github.com/users/ahmetmircik/orgs","repos_url":"https://api.github.com/users/ahmetmircik/repos","events_url":"https://api.github.com/users/ahmetmircik/events{/privacy}","received_events_url":"https://api.github.com/users/ahmetmircik/received_events","type":"User","site_admin":false},"created_at":"2018-09-21T13:48:22Z","updated_at":"2018-09-21T13:48:22Z","author_association":"MEMBER","body":"Summarizing 2 outcomes from my investigations:\r\n\r\n__First outcome:__\r\n\r\n`RejectedExecutionException` was thrown by `ASYNC_EXECUTOR`.\r\n\r\n`ASYNC_EXECUTOR` is a shared executor used by many proxies/services. Near-cached-map-proxies or reliable-topic-proxies are 2 examples among many usages.\r\n\r\nCreation of `ASYNC_EXECUTOR` is initiated by\r\n[OperationServiceImpl#initInvocationContext](https://github.com/hazelcast/hazelcast/blob/55027987bbbe4e9d93540538f3ffec2f6476b0cb/hazelcast/src/main/java/com/hazelcast/spi/impl/operationservice/impl/OperationServiceImpl.java#L446) like this: \r\n\r\n```\r\nManagedExecutorService asyncExecutor = nodeEngine.getExecutionService().register(\r\n                ExecutionService.ASYNC_EXECUTOR, RuntimeAvailableProcessors.get(),\r\n                ASYNC_QUEUE_CAPACITY, ExecutorType.CONCRETE);\r\n```\r\n\r\nThis executor can have at most `RuntimeAvailableProcessors.get()` thread count and it can only contain `100_000 (ASYNC_QUEUE_CAPACITY)` entries at most in its queue to process.\r\n\r\nIt seems easy to reject shutdown tasks under load. Since shutdown task also using this globally shared executor. And these shutdown tasks may not found either an empty place in queue or a new worker thread, then `ThreadPoolExecutor` throws this exception. \r\n\r\n__Second outcome:__\r\n\r\nASYNC_EXECUTOR creation is fragile.\r\n\r\n[NodeEngineImpl#start](https://github.com/hazelcast/hazelcast/blob/55027987bbbe4e9d93540538f3ffec2f6476b0cb/hazelcast/src/main/java/com/hazelcast/spi/impl/NodeEngineImpl.java#L208)\r\n\r\n```\r\npublic void start() {\r\n        RuntimeMetricSet.register(metricsRegistry);\r\n        GarbageCollectionMetricSet.register(metricsRegistry);\r\n        OperatingSystemMetricSet.register(metricsRegistry);\r\n        ThreadMetricSet.register(metricsRegistry);\r\n        ClassLoadingMetricSet.register(metricsRegistry);\r\n        FileMetricSet.register(metricsRegistry);\r\n        if (node.getProperties().getBoolean(METRICS_DISTRIBUTED_DATASTRUCTURES)) {\r\n            new StatisticsAwareMetricsSet(serviceManager, this).register(metricsRegistry);\r\n        }\r\n        metricsRegistry.scanAndRegister(node.getNodeExtension().getMemoryStats(), \"memory\");\r\n        metricsRegistry.collectMetrics(operationService, proxyService, eventService, operationParker);\r\n\r\n        serviceManager.start();\r\n        proxyService.init();\r\n        operationService.start();<-- ASYNC_EXECUTOR is created at this stage now.\r\n        quorumService.start();\r\n        diagnostics.start();\r\n\r\n        node.getNodeExtension().registerPlugins(diagnostics);\r\n    }\r\n```\r\n\r\nIn current code base, it is created by `operationService.start()` as a `CONCRETE` executor but it is possible that any time in the future any other class starting before `operationService` can create `ASYNC_EXECUTOR` either as `CONCRETE` or as `CACHED` type and with different settings. Maybe we can think of creating an executor specific to operation service. \r\n\r\n__Conclusion:__ \r\n\r\n`RejectedExecutionException` can be seen as an expected exception in a loaded cluster or we can create a specific executor only for operation service, so internal working parts of the system affects operation service less.\r\n\r\n@mdogan @mmedenjak Can you please share your thoughts on these outcomes? \r\n\r\n@Danny-Hazelcast Can you re-run this test by disabling map-stats? Map stats are on by default and using `ASYNC_EXECUTOR` for async puts as in this test.\r\n\r\n"}}
{"actor":{"display_login":"ahmetmircik"}}