{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/22189","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/22189/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/22189/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/22189/events","html_url":"https://github.com/elastic/elasticsearch/issues/22189","id":195786904,"number":22189,"title":"elastic 5.1.1 master node always pinged time out, failed to form a cluster","user":{"login":"sinsonglew","id":12896760,"avatar_url":"https://avatars.githubusercontent.com/u/12896760?v=3","gravatar_id":"","url":"https://api.github.com/users/sinsonglew","html_url":"https://github.com/sinsonglew","followers_url":"https://api.github.com/users/sinsonglew/followers","following_url":"https://api.github.com/users/sinsonglew/following{/other_user}","gists_url":"https://api.github.com/users/sinsonglew/gists{/gist_id}","starred_url":"https://api.github.com/users/sinsonglew/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sinsonglew/subscriptions","organizations_url":"https://api.github.com/users/sinsonglew/orgs","repos_url":"https://api.github.com/users/sinsonglew/repos","events_url":"https://api.github.com/users/sinsonglew/events{/privacy}","received_events_url":"https://api.github.com/users/sinsonglew/received_events","type":"User","site_admin":false},"labels":[{"id":152506210,"url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Cluster","name":":Cluster","color":"009800","default":false},{"id":146854632,"url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Network","name":":Network","color":"009800","default":false},{"id":111624690,"url":"https://api.github.com/repos/elastic/elasticsearch/labels/feedback_needed","name":"feedback_needed","color":"d4c5f9","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2016-12-15T11:53:38Z","updated_at":"2016-12-19T14:15:58Z","closed_at":null,"body":"**Elasticsearch version**: 5.1.1\r\n\r\n**Plugins installed**: [] only default\r\n\r\n**JVM version**:  Java(TM) SE Runtime Environment (build 1.8.0_111-b14)\r\n\r\n**OS version**: 2.6.32-504.12.2.02.el6.x86_64 #1 SMP Tue May 12 11:44:09 CST 2015 x86_64 x86_64 x86_64 GNU/Linux\r\n\r\n**Description of the problem including expected versus actual behavior**:\r\nif nodes greater than 3, the cluster is constantly add and remove nodes, leading to unavailable status.\r\nFrom the following attached log, nodes ping each other timed out often\r\nexpected: a full-nodes started cluster\r\n\r\n**Provide logs (if relevant)**:\r\n```\r\n[2016-12-15T19:39:17,648][INFO ][o.e.c.s.ClusterService   ] [node244] removed {{node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371},}, reason: master_failed ({node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371})\r\n[2016-12-15T19:39:42,556][INFO ][o.e.d.z.ZenDiscovery     ] [node244] failed to send join request to master [{node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371}], reason [RemoteTransportException[[node237][aa.bb.cc.dd:9371][internal:discovery/zen/join]]; nested: IllegalStateException[failure when sending a validation request to node]; nested: NodeDisconnectedException[[node244][ee.ff.gg.hh:9371][internal:discovery/zen/join/validate] disconnected]; ]\r\n[2016-12-15T19:40:02,565][INFO ][o.e.d.z.ZenDiscovery     ] [node244] failed to send join request to master [{node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371}], reason [RemoteTransportException[[node237][aa.bb.cc.dd:9371][internal:discovery/zen/join]]; nested: ConnectTransportException[[node244][ee.ff.gg.hh:9371] connect_timeout[30s]]; nested: IOException[Connection timed out: ee.ff.gg.hh/ee.ff.gg.hh:9371]; ]\r\n[2016-12-15T19:40:25,433][INFO ][o.e.d.z.ZenDiscovery     ] [node244] failed to send join request to master [{node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371}], reason [RemoteTransportException[[node237][aa.bb.cc.dd:9371][internal:discovery/zen/join]]; nested: IllegalStateException[failure when sending a validation request to node]; nested: NodeDisconnectedException[[node244][ee.ff.gg.hh:9371][internal:discovery/zen/join/validate] disconnected]; ]\r\n[2016-12-15T19:40:42,442][INFO ][o.e.d.z.ZenDiscovery     ] [node244] failed to send join request to master [{node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371}], reason [NodeDisconnectedException[[node237][aa.bb.cc.dd:9371][internal:discovery/zen/join] disconnected]]\r\n[2016-12-15T19:41:03,531][INFO ][o.e.c.s.ClusterService   ] [node244] detected_master {node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371}, added {{node241}{gx3rLPdIQvSm_u6mLoIbtQ}{80MXRZOjSN6RAQ7gZGeUcQ}{xx.x.xx.x}{x.x.x.x:9371},{node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371},}, reason: zen-disco-receive(from master [master {node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371} committed version [115]])\r\n[2016-12-15T19:41:09,541][INFO ][o.e.d.z.ZenDiscovery     ] [node244] master_left [{node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371}], reason [failed to ping, tried [3] times, each with  maximum [30s] timeout]\r\n[2016-12-15T19:41:09,541][WARN ][o.e.d.z.ZenDiscovery     ] [node244] master left (reason = failed to ping, tried [3] times, each with  maximum [30s] timeout), current nodes: nodes: \r\n   {node243}{tEv4PrXSTdyecz2nClog4Q}{KQsJYnauSe2W0hxWPte4Iw}{xx.xx.xx.xx}{xx.xx.xx.xx:9371}\r\n   {node239}{oWLoFJf-Rs-3q-kK1Y_2BQ}{dekHpx5zQASxKgIO21n1Yg}{xx.xx.xx.xx}{xx.xx.xx.xx:9371}\r\n   {node244}{stop1h4jQc-zCvM2hBS56Q}{iV8QrIRiQ5y3cvW_en0sMA}{ee.ff.gg.hh}{ee.ff.gg.hh:9371}, local\r\n   {node241}{gx3rLPdIQvSm_u6mLoIbtQ}{80MXRZOjSN6RAQ7gZGeUcQ}{xx.xx.xx.xx}{xx.xx.xx.xx:9371}\r\n   {node242}{k5vCKHcjTSKTK-uhK_MtLw}{DMGU0D-8T-Si-ZpfqmiDTQ}{xx.xx.xx.xx}{xx.xx.xx.xx:9371}\r\n\r\n[2016-12-15T19:41:09,542][INFO ][o.e.c.s.ClusterService   ] [node244] removed {{node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371},}, reason: master_failed ({node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371})\r\n[2016-12-15T19:41:28,555][INFO ][o.e.c.s.ClusterService   ] [node244] master {new {node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371}}, removed {{node241}{gx3rLPdIQvSm_u6mLoIbtQ}{80MXRZOjSN6RAQ7gZGeUcQ}{xx.xx.xx.xx}{x.x.x.x:9371},}, added {{node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371},}, reason: zen-disco-receive(from master [master {node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371} committed version [120]])\r\n[2016-12-15T19:41:34,564][INFO ][o.e.d.z.ZenDiscovery     ] [node244] master_left [{node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371}], reason [failed to ping, tried [3] times, each with  maximum [30s] timeout]\r\n[2016-12-15T19:41:34,564][WARN ][o.e.d.z.ZenDiscovery     ] [node244] master left (reason = failed to ping, tried [3] times, each with  maximum [30s] timeout), current nodes: nodes: \r\n   {node243}{tEv4PrXSTdyecz2nClog4Q}{KQsJYnauSe2W0hxWPte4Iw}{xx.xx.xx.xx}{xx.xx.xx.xx:9371}\r\n   {node239}{oWLoFJf-Rs-3q-kK1Y_2BQ}{dekHpx5zQASxKgIO21n1Yg}{xx.xx.xx.xx}{xx.xx.xx.xx:9371}\r\n   {node244}{stop1h4jQc-zCvM2hBS56Q}{iV8QrIRiQ5y3cvW_en0sMA}{ee.ff.gg.hh}{ee.ff.gg.hh:9371}, local\r\n   {node242}{k5vCKHcjTSKTK-uhK_MtLw}{DMGU0D-8T-Si-ZpfqmiDTQ}{xx.xx.xx.xx}{xx.xx.xx.xx:9371}\r\n\r\n[2016-12-15T19:41:34,565][INFO ][o.e.c.s.ClusterService   ] [node244] removed {{node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371},}, reason: master_failed ({node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371})\r\n[2016-12-15T19:41:55,575][INFO ][o.e.d.z.ZenDiscovery     ] [node244] failed to send join request to master [{node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371}], reason [RemoteTransportException[[node237][aa.bb.cc.dd:9371][internal:discovery/zen/join]]; nested: ConnectTransportException[[node244][ee.ff.gg.hh:9371] connect_timeout[30s]]; nested: IOException[Connection timed out: ee.ff.gg.hh/ee.ff.gg.hh:9371]; ]\r\n[2016-12-15T19:42:15,586][INFO ][o.e.d.z.ZenDiscovery     ] [node244] failed to send join request to master [{node237}{FmyPJDzMR2KRSl10jPGT0Q}{hrOcPGYBTOSEQU5J7hOwVA}{aa.bb.cc.dd}{aa.bb.cc.dd:9371}], reason [RemoteTransportException[[node237][aa.bb.cc.dd:9371][internal:discovery/zen/join]]; nested: ConnectTransportException[[node244][ee.ff.gg.hh:9371] connect_timeout[30s]]; nested: IOException[Connection timed out: ee.ff.gg.hh/ee.ff.gg.hh:9371]; ]\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/267973937","html_url":"https://github.com/elastic/elasticsearch/issues/22189#issuecomment-267973937","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/22189","id":267973937,"user":{"login":"jasontedor","id":4744941,"avatar_url":"https://avatars.githubusercontent.com/u/4744941?v=3","gravatar_id":"","url":"https://api.github.com/users/jasontedor","html_url":"https://github.com/jasontedor","followers_url":"https://api.github.com/users/jasontedor/followers","following_url":"https://api.github.com/users/jasontedor/following{/other_user}","gists_url":"https://api.github.com/users/jasontedor/gists{/gist_id}","starred_url":"https://api.github.com/users/jasontedor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jasontedor/subscriptions","organizations_url":"https://api.github.com/users/jasontedor/orgs","repos_url":"https://api.github.com/users/jasontedor/repos","events_url":"https://api.github.com/users/jasontedor/events{/privacy}","received_events_url":"https://api.github.com/users/jasontedor/received_events","type":"User","site_admin":false},"created_at":"2016-12-19T14:15:58Z","updated_at":"2016-12-19T14:15:58Z","body":"Can you share some more details:\r\n - node specs (especially heap size but general specs as well)\r\n - from your kernel version I think that you're running RHEL 6, can you confirm?\r\n - logs from a node and the master (there is no need to mask the IP addresses, from your configuration I see that you're in 10.0.0.0/8 which is a private IP address space; masking will just make it harder-than-necessary to read)\r\n - are the nodes accessible over HTTP (e.g., `GET /`)?\r\n - what is going on on the cluster when this happens (heavy indexing, searching, etc.)?\r\n"}}
{"actor":{"display_login":"jasontedor"}}