{"action":"opened","issue":{"url":"https://api.github.com/repos/crate/crate/issues/3990","repository_url":"https://api.github.com/repos/crate/crate","labels_url":"https://api.github.com/repos/crate/crate/issues/3990/labels{/name}","comments_url":"https://api.github.com/repos/crate/crate/issues/3990/comments","events_url":"https://api.github.com/repos/crate/crate/issues/3990/events","html_url":"https://github.com/crate/crate/issues/3990","id":172071405,"number":3990,"title":"MASTER and DATA node has an ESTABLISHED connection but Recv-Q is not zero for a long time","user":{"login":"jaykelin","id":523906,"avatar_url":"https://avatars.githubusercontent.com/u/523906?v=3","gravatar_id":"","url":"https://api.github.com/users/jaykelin","html_url":"https://github.com/jaykelin","followers_url":"https://api.github.com/users/jaykelin/followers","following_url":"https://api.github.com/users/jaykelin/following{/other_user}","gists_url":"https://api.github.com/users/jaykelin/gists{/gist_id}","starred_url":"https://api.github.com/users/jaykelin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jaykelin/subscriptions","organizations_url":"https://api.github.com/users/jaykelin/orgs","repos_url":"https://api.github.com/users/jaykelin/repos","events_url":"https://api.github.com/users/jaykelin/events{/privacy}","received_events_url":"https://api.github.com/users/jaykelin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-08-19T07:31:29Z","updated_at":"2016-08-19T07:31:29Z","closed_at":null,"body":"crate 0.55.2 cluster\r\n10.1.113.172 - master\r\n10.1.113.175、10.1.113.176\r\n`[root@YiZhuang_10_1_113_172 logs]# netstat -nap  | grep 4300 | grep 10.1.113.175`\r\n\r\n> tcp        0      0 ::ffff:10.1.113.172:6488    ::ffff:10.1.113.175:4300    ESTABLISHED 63545/java          \r\n> tcp        0      0 ::ffff:10.1.113.172:6485    ::ffff:10.1.113.175:4300    ESTABLISHED 63545/java          \r\n> tcp        0      0 ::ffff:10.1.113.172:6492    ::ffff:10.1.113.175:4300    ESTABLISHED 63545/java          \r\n> tcp   121347      0 ::ffff:10.1.113.172:4300    ::ffff:10.1.113.175:2063    ESTABLISHED 63545/java          \r\n> tcp        0      0 ::ffff:10.1.113.172:4300    ::ffff:10.1.113.175:2066    ESTABLISHED 63545/java          \r\n> tcp        0      0 ::ffff:10.1.113.172:6482    ::ffff:10.1.113.175:4300    ESTABLISHED 63545/java          \r\n> tcp        0      0 ::ffff:10.1.113.172:6489    ::ffff:10.1.113.175:4300    ESTABLISHED 63545/java          \r\n> tcp        0      0 ::ffff:10.1.113.172:4300    ::ffff:10.1.113.175:2059    ESTABLISHED 63545/java          \r\n> tcp        0      0 ::ffff:10.1.113.172:4300    ::ffff:10.1.113.175:2064    ESTABLISHED 63545/java          \r\n> tcp        0      0 ::ffff:10.1.113.172:6490    ::ffff:10.1.113.175:4300    ESTABLISHED 63545/java          \r\n> tcp        0      0 ::ffff:10.1.113.172:4300    ::ffff:10.1.113.175:2057    ESTABLISHED 63545/java          \r\n> tcp        0      0 ::ffff:10.1.113.172:6493    ::ffff:10.1.113.175:4300    ESTABLISHED 63545/java          \r\n> tcp   2092697      0 ::ffff:10.1.113.172:4300    ::ffff:10.1.113.175:2056    ESTABLISHED 63545/java          \r\n> tcp    78319      0 ::ffff:10.1.113.172:4300    ::ffff:10.1.113.175:2065    ESTABLISHED 63545/java          \r\n> tcp   148138      0 ::ffff:10.1.113.172:4300    ::ffff:10.1.113.175:2061    ESTABLISHED 63545/java          \r\n> tcp        0      0 ::ffff:10.1.113.172:6484    ::ffff:10.1.113.175:4300    ESTABLISHED 63545/java          \r\n> tcp        0      0 ::ffff:10.1.113.172:4300    ::ffff:10.1.113.175:2068    ESTABLISHED 63545/java          \r\n> tcp        0      0 ::ffff:10.1.113.172:4300    ::ffff:10.1.113.175:2060    ESTABLISHED 63545/java          \r\n> tcp        0      0 ::ffff:10.1.113.172:4300    ::ffff:10.1.113.175:2058    ESTABLISHED 63545/java          \r\n> tcp        0      0 ::ffff:10.1.113.172:6491    ::ffff:10.1.113.175:4300    ESTABLISHED 63545/java          \r\n> tcp   112078      0 ::ffff:10.1.113.172:4300    ::ffff:10.1.113.175:2062    ESTABLISHED 63545/java          \r\n> tcp        0      0 ::ffff:10.1.113.172:6494    ::ffff:10.1.113.175:4300    ESTABLISHED 63545/java          \r\n> tcp        0      0 ::ffff:10.1.113.172:6486    ::ffff:10.1.113.175:4300    ESTABLISHED 63545/java          \r\n> tcp        0      0 ::ffff:10.1.113.172:4300    ::ffff:10.1.113.175:2067    ESTABLISHED 63545/java          \r\n> tcp        0      0 ::ffff:10.1.113.172:6487    ::ffff:10.1.113.175:4300    ESTABLISHED 63545/java          \r\n> tcp        0      0 ::ffff:10.1.113.172:6483    ::ffff:10.1.113.175:4300    ESTABLISHED 63545/java\r\n\r\nabove 5 socket with Recv-Q not zero for a long time， and not change. I‘m confused, how that could by happened.\r\n\r\nand then i check the logs of DATA node[10.1.113.175], there are many WARN info, like this \r\n\r\n> [WARN ][indices.cluster          ] [crate175] [[.partitioned.tableName.045j4c1h6omj0dpd64sg][6]] marking and sending shard failed due to [failed recovery]  \r\n> ......... nested: ElasticsearchTimeoutException[no activity after [30m]];\r\n>         at org.elasticsearch.indices.recovery.RecoveriesCollection$RecoveryMonitor.doRun(RecoveriesCollection.java:235)\r\n>         at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n>         at java.lang.Thread.run(Thread.java:745)\r\n> Caused by: ElasticsearchTimeoutException[no activity after [30m]]\r\n\r\nat the same time， JDBC connect to cluster is not working :\r\n\r\n> Error: java.io.IOException: java.sql.SQLException: Connect to '10.1.113.172:4300' failed\r\n> ....\r\n> Caused by: java.sql.SQLException: Connect to '10.1.113.172:4300' failed\r\n> at io.crate.client.jdbc.CrateConnection.connect(CrateConnection.java:60)\r\n> at io.crate.client.jdbc.CrateDriver.connect(CrateDriver.java:76)\r\n> at java.sql.DriverManager.getConnection(DriverManager.java:571)\r\n> ....\r\n> Caused by: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{10.1.113.172}{10.1.113.172:4300}]]\r\n> at io.crate.shade.org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:290)\r\n> at io.crate.shade.org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:207)\r\n> at io.crate.client.InternalCrateClient.execute(InternalCrateClient.java:85)\r\n> at io.crate.client.InternalCrateClient.execute(InternalCrateClient.java:68)\r\n> at io.crate.client.InternalCrateClient.sql(InternalCrateClient.java:57)\r\n> at io.crate.client.CrateClient.sql(CrateClient.java:145)\r\n> at io.crate.client.jdbc.CrateStatement.execute(CrateStatement.java:65)\r\n> at io.crate.client.jdbc.CrateStatementBase.executeQuery(CrateStatementBase.java:37)\r\n> at io.crate.client.jdbc.CrateDatabaseMetaData.getDatabaseProductVersion(CrateDatabaseMetaData.java:79)\r\n> at io.crate.client.jdbc.CrateConnection.connect(CrateConnection.java:57)"}}
{"actor":{"display_login":"jaykelin"}}