{"action":"created","issue":{"url":"https://api.github.com/repos/netty/netty/issues/5486","repository_url":"https://api.github.com/repos/netty/netty","labels_url":"https://api.github.com/repos/netty/netty/issues/5486/labels{/name}","comments_url":"https://api.github.com/repos/netty/netty/issues/5486/comments","events_url":"https://api.github.com/repos/netty/netty/issues/5486/events","html_url":"https://github.com/netty/netty/issues/5486","id":163457711,"number":5486,"title":"netty 4.0.37 causes hangs in Apache Cassandra","user":{"login":"JeremiahDJordan","id":1037592,"avatar_url":"https://avatars.githubusercontent.com/u/1037592?v=3","gravatar_id":"","url":"https://api.github.com/users/JeremiahDJordan","html_url":"https://github.com/JeremiahDJordan","followers_url":"https://api.github.com/users/JeremiahDJordan/followers","following_url":"https://api.github.com/users/JeremiahDJordan/following{/other_user}","gists_url":"https://api.github.com/users/JeremiahDJordan/gists{/gist_id}","starred_url":"https://api.github.com/users/JeremiahDJordan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JeremiahDJordan/subscriptions","organizations_url":"https://api.github.com/users/JeremiahDJordan/orgs","repos_url":"https://api.github.com/users/JeremiahDJordan/repos","events_url":"https://api.github.com/users/JeremiahDJordan/events{/privacy}","received_events_url":"https://api.github.com/users/JeremiahDJordan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"normanmaurer","id":439362,"avatar_url":"https://avatars.githubusercontent.com/u/439362?v=3","gravatar_id":"","url":"https://api.github.com/users/normanmaurer","html_url":"https://github.com/normanmaurer","followers_url":"https://api.github.com/users/normanmaurer/followers","following_url":"https://api.github.com/users/normanmaurer/following{/other_user}","gists_url":"https://api.github.com/users/normanmaurer/gists{/gist_id}","starred_url":"https://api.github.com/users/normanmaurer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/normanmaurer/subscriptions","organizations_url":"https://api.github.com/users/normanmaurer/orgs","repos_url":"https://api.github.com/users/normanmaurer/repos","events_url":"https://api.github.com/users/normanmaurer/events{/privacy}","received_events_url":"https://api.github.com/users/normanmaurer/received_events","type":"User","site_admin":false},"assignees":[{"login":"normanmaurer","id":439362,"avatar_url":"https://avatars.githubusercontent.com/u/439362?v=3","gravatar_id":"","url":"https://api.github.com/users/normanmaurer","html_url":"https://github.com/normanmaurer","followers_url":"https://api.github.com/users/normanmaurer/followers","following_url":"https://api.github.com/users/normanmaurer/following{/other_user}","gists_url":"https://api.github.com/users/normanmaurer/gists{/gist_id}","starred_url":"https://api.github.com/users/normanmaurer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/normanmaurer/subscriptions","organizations_url":"https://api.github.com/users/normanmaurer/orgs","repos_url":"https://api.github.com/users/normanmaurer/repos","events_url":"https://api.github.com/users/normanmaurer/events{/privacy}","received_events_url":"https://api.github.com/users/normanmaurer/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/netty/netty/milestones/146","html_url":"https://github.com/netty/netty/milestones/4.0.39.Final","labels_url":"https://api.github.com/repos/netty/netty/milestones/146/labels","id":1859618,"number":146,"title":"4.0.39.Final","description":"","creator":{"login":"normanmaurer","id":439362,"avatar_url":"https://avatars.githubusercontent.com/u/439362?v=3","gravatar_id":"","url":"https://api.github.com/users/normanmaurer","html_url":"https://github.com/normanmaurer","followers_url":"https://api.github.com/users/normanmaurer/followers","following_url":"https://api.github.com/users/normanmaurer/following{/other_user}","gists_url":"https://api.github.com/users/normanmaurer/gists{/gist_id}","starred_url":"https://api.github.com/users/normanmaurer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/normanmaurer/subscriptions","organizations_url":"https://api.github.com/users/normanmaurer/orgs","repos_url":"https://api.github.com/users/normanmaurer/repos","events_url":"https://api.github.com/users/normanmaurer/events{/privacy}","received_events_url":"https://api.github.com/users/normanmaurer/received_events","type":"User","site_admin":false},"open_issues":50,"closed_issues":3,"state":"open","created_at":"2016-07-01T11:24:45Z","updated_at":"2016-07-02T06:25:19Z","due_on":null,"closed_at":null},"comments":27,"created_at":"2016-07-01T19:12:23Z","updated_at":"2016-07-02T12:23:04Z","closed_at":null,"body":"Ever since we updated Apache Cassandra to using netty 4.0.37 from netty 4.0.36 we have been seeing random hangs on our OSS test suite where a response from the server to the client seemingly is getting lost or something (the client just hangs) until the test times out.  With the OSS test suite it only happens sometimes, it isn't super easy to reproduce (right now I can only reproduce it using a 1.5 hour test suite we have for DataStax Enterprise) but I bisected it to f984870ccca133d6056e8b0df0b2352f8f90b0fe.  I then reverted f984870ccca133d6056e8b0df0b2352f8f90b0fe from 4.0.37 and I can run my tests with 4.0.37+f984870ccca133d6056e8b0df0b2352f8f90b0fe reverted and everything seems good."},"comment":{"url":"https://api.github.com/repos/netty/netty/issues/comments/230099231","html_url":"https://github.com/netty/netty/issues/5486#issuecomment-230099231","issue_url":"https://api.github.com/repos/netty/netty/issues/5486","id":230099231,"user":{"login":"normanmaurer","id":439362,"avatar_url":"https://avatars.githubusercontent.com/u/439362?v=3","gravatar_id":"","url":"https://api.github.com/users/normanmaurer","html_url":"https://github.com/normanmaurer","followers_url":"https://api.github.com/users/normanmaurer/followers","following_url":"https://api.github.com/users/normanmaurer/following{/other_user}","gists_url":"https://api.github.com/users/normanmaurer/gists{/gist_id}","starred_url":"https://api.github.com/users/normanmaurer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/normanmaurer/subscriptions","organizations_url":"https://api.github.com/users/normanmaurer/orgs","repos_url":"https://api.github.com/users/normanmaurer/repos","events_url":"https://api.github.com/users/normanmaurer/events{/privacy}","received_events_url":"https://api.github.com/users/normanmaurer/received_events","type":"User","site_admin":false},"created_at":"2016-07-02T12:23:04Z","updated_at":"2016-07-02T12:23:04Z","body":"@snazy @JeremiahDJordan @tjake ok I took the time to debug the issue and thanks to @snazy it was quite easy to understand whats going on :) I would have taken me a lot longer without the unit test.\r\n\r\nTo make it short, its a cassandra bug IMHO and you was just \"lucky\" that it never broke before. \r\n\r\nSo here is the problem.... In Cassandra you are using your own `EventExecutor` implementation which break the `EventExecutor` contract.  The contract is that `execute(...)` calls must be executed in a serial fashion which is not the case here as you just return `false` from the `inEventLoop()` method and then delegate the `execute` call to an underlying Executor, which gives no such guarantees. Because of this a race can happen where an \"event\" may be handed over to an `AbstractChannelHandlerContext` before `setAdded()` is called, which will then have the effect that the handler is skipped.\r\n\r\nThis was previous no problem as we synchronized which had other problems, thus we removed it. \r\n\r\nSo to make it short your `EventExecutor` implementation must guarantee ordering. That said you can have an `EventExecutorGroup` that holds multiple `EventExecutor` implementations and each of them only needs to guarantee ordering by itself, not for the whole group. \r\n\r\nHope this makes sense.... Let me know if you have more questions.\r\n\r\n@snazy when doing this patch it never fails for me with your branch:\r\n\r\n```\r\niff --git a/src/java/org/apache/cassandra/service/NativeTransportService.java b/src/java/org/apache/cassandra/service/NativeTransportService.java\r\nindex 48839f1..e46b895 100644\r\n--- a/src/java/org/apache/cassandra/service/NativeTransportService.java\r\n+++ b/src/java/org/apache/cassandra/service/NativeTransportService.java\r\n@@ -81,7 +81,7 @@ public class NativeTransportService\r\n\r\n         org.apache.cassandra.transport.Server.Builder builder = new org.apache.cassandra.transport.Server.Builder()\r\n                                                                 .withEventExecutor(eventExecutorGroup)\r\n-                                                                .withEventLoopGroup(workerGroup)\r\n+                                                                //.withEventLoopGroup(workerGroup)\r\n                                                                 .withHost(nativeAddr);\r\n\r\n         if (!DatabaseDescriptor.getClientEncryptionOptions().enabled)\r\ndiff --git a/test/unit/org/apache/cassandra/cql3/CQLTester.java b/test/unit/org/apache/cassandra/cql3/CQLTester.java\r\nindex 30ddf2c..f1fedce 100644\r\n--- a/test/unit/org/apache/cassandra/cql3/CQLTester.java\r\n+++ b/test/unit/org/apache/cassandra/cql3/CQLTester.java\r\n@@ -344,7 +344,7 @@ public abstract class CQLTester\r\n         server = new Server.Builder().\r\n                             withHost(nativeAddr).\r\n                             withPort(nativePort).\r\n-                            withEventExecutor(new RequestThreadPoolExecutor()).\r\n+                            //withEventExecutor(new RequestThreadPoolExecutor()).\r\n                             build();\r\n         server.start();\r\n```"}}
{"actor":{"display_login":"normanmaurer"}}