{"action":"closed","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/21596","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/21596/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/21596/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/21596/events","html_url":"https://github.com/elastic/elasticsearch/issues/21596","id":189712826,"number":21596,"title":"elasticsearch throw cluster:monitor/nodes/stats[n error ","user":{"login":"GeoffreyFx","id":4462273,"avatar_url":"https://avatars.githubusercontent.com/u/4462273?v=3","gravatar_id":"","url":"https://api.github.com/users/GeoffreyFx","html_url":"https://github.com/GeoffreyFx","followers_url":"https://api.github.com/users/GeoffreyFx/followers","following_url":"https://api.github.com/users/GeoffreyFx/following{/other_user}","gists_url":"https://api.github.com/users/GeoffreyFx/gists{/gist_id}","starred_url":"https://api.github.com/users/GeoffreyFx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GeoffreyFx/subscriptions","organizations_url":"https://api.github.com/users/GeoffreyFx/orgs","repos_url":"https://api.github.com/users/GeoffreyFx/repos","events_url":"https://api.github.com/users/GeoffreyFx/events{/privacy}","received_events_url":"https://api.github.com/users/GeoffreyFx/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-11-16T14:02:53Z","updated_at":"2016-11-19T14:58:15Z","closed_at":"2016-11-19T14:58:15Z","body":"Hi All, \r\n\r\nThis is a serious issue about elasticsearch.  \r\n\r\nThe elasticsearch log throws 'cluster:monitor/nodes/stats[n]]' error, and one of the cluster nodes response time exceeds 15s.  This error badly impact the stable of the whole system causing the search and index operations not workable. \r\n\r\nThe ES version we use is 1.7.3. Our cluster topology is 5 primary shard , 2 replica shard, and two ES nodes are installed on each machine, total 10 es nodes on 5 machines.  And every day there is a new index created, the old index are removed. The data amount are about 12 G. \r\n\r\nThe traffic of search & index rate are both very high. \r\n\r\nAnyone has this similar issue? Very appreciate your help！！\r\n\r\n~~~~~~~~~~~~~~~~log as below~~~~~~~~~~~~~~~~~\r\n[2016-11-16 21:31:31,847][DEBUG][action.admin.cluster.node.stats] [es147] failed to execute on node [cp2Xr5sGTSK7Qw2Ycc-r0Q]\r\norg.elasticsearch.transport.ReceiveTimeoutTransportException: [es34][inet[/10.0.0.34:9300]][cluster:monitor/nodes/stats[n]] request_id [65940515] timed out after [15001ms]\r\n        at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:529)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n        at java.lang.Thread.run(Thread.java:745)\r\n[2016-11-16 21:31:52,026][WARN ][transport                ] [es147] Received response for a request that has timed out, sent [65180ms] ago, timed out [50180ms] ago, action [cluster:monitor/nodes/stats[n]], node [[es34][cp2Xr5sGTSK7Qw2Ycc-r0Q][node34][inet[/10.0.0.34:9300]]], id [65928009]\r\n[2016-11-16 21:31:56,504][WARN ][transport                ] [es147] Received response for a request that has timed out, sent [39658ms] ago, timed out [24657ms] ago, action [cluster:monitor/nodes/stats[n]], node [[es34][cp2Xr5sGTSK7Qw2Ycc-r0Q][node34][inet[/10.0.0.34:9300]]], id [65940515]   "}}
{"actor":{"display_login":"clintongormley"}}