{"action":"created","issue":{"url":"https://api.github.com/repos/rancher/rancher/issues/15269","repository_url":"https://api.github.com/repos/rancher/rancher","labels_url":"https://api.github.com/repos/rancher/rancher/issues/15269/labels{/name}","comments_url":"https://api.github.com/repos/rancher/rancher/issues/15269/comments","events_url":"https://api.github.com/repos/rancher/rancher/issues/15269/events","html_url":"https://github.com/rancher/rancher/issues/15269","id":354774562,"node_id":"MDU6SXNzdWUzNTQ3NzQ1NjI=","number":15269,"title":"etcd Cluster Not Letting New Nodes Join","user":{"login":"kylegoch","id":5991749,"node_id":"MDQ6VXNlcjU5OTE3NDk=","avatar_url":"https://avatars3.githubusercontent.com/u/5991749?v=4","gravatar_id":"","url":"https://api.github.com/users/kylegoch","html_url":"https://github.com/kylegoch","followers_url":"https://api.github.com/users/kylegoch/followers","following_url":"https://api.github.com/users/kylegoch/following{/other_user}","gists_url":"https://api.github.com/users/kylegoch/gists{/gist_id}","starred_url":"https://api.github.com/users/kylegoch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kylegoch/subscriptions","organizations_url":"https://api.github.com/users/kylegoch/orgs","repos_url":"https://api.github.com/users/kylegoch/repos","events_url":"https://api.github.com/users/kylegoch/events{/privacy}","received_events_url":"https://api.github.com/users/kylegoch/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-08-28T15:05:52Z","updated_at":"2018-08-28T15:20:40Z","closed_at":null,"author_association":"NONE","body":"**Rancher versions:**\r\nrancher/server or rancher/rancher: 2.0.8\r\nrancher/agent or rancher/rancher-agent: 2.0.8\r\n\r\n**Infrastructure Stack versions:**\r\nhealthcheck:\r\nipsec:\r\nnetwork-services: canal\r\nscheduler:\r\nkubernetes (if applicable): 1.11.2\r\n\r\n**Docker version: (`docker version`,`docker info` preferred)**\r\n18.3.1\r\n**Operating system and kernel: (`cat /etc/os-release`, `uname -r` preferred)**\r\nContainer Linux by CoreOS 1800.7.0 (Rhyolite)\r\n**Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)**\r\nAWS\r\n**Setup details: (single node rancher vs. HA rancher, internal DB vs. external DB)**\r\nHA Rancher\r\n**Environment Template: (Cattle/Kubernetes/Swarm/Mesos)**\r\nKubernetes\r\n**Steps to Reproduce:**\r\nCustom Kubernetes cluster setup using Rancher Agent and Ignition for CoreOS to provision the nodes.\r\n\r\n3 etcd nodes: m5.large\r\n3 master/controlplane: m5.large\r\nAssorted workers (using spot fleet)\r\n\r\nEtcd and Master are in Autoscaling Groups.\r\n\r\nOvernight, an etcd node died in the Autoscaling Group. The ASG killed it and made a new one. The status in the Rancher Console was:\r\nActive for two existing etcd nodes.\r\nRegistering for the new node\r\nUnavailable for the old node.\r\n\r\nConsole error message of:\r\n`Failed to reconcile etcd plane: Failed to add etcd member [new_node_hostname] to etcd cluster`\r\n\r\nDeleted the old node that had died. Provisioning started again, but then finished with the same status as above, but with the old node gone.\r\n\r\nChecked the logs on an old node and found it saying this:\r\n```\r\n2018-08-28 13:49:25.346896 W | etcdserver: failed to reach the peerURL(https://10.10.200.17:2380) of member 20d735767b9414c (Get https://10.10.200.17:2380/version: dial tcp 10.10.200.17:2380: i/o timeout)\r\n2018-08-28 13:49:25.346922 W | etcdserver: cannot get the version of member 20d735767b9414c (Get https://10.10.200.17:2380/version: dial tcp 10.10.200.17:2380: i/o timeout)\r\n2018-08-28 13:49:25.887047 W | rafthttp: health check for peer 20d735767b9414c could not connect: dial tcp 10.10.200.17:2380: i/o timeout\r\n2018-08-28 13:49:30.887188 W | rafthttp: health check for peer 20d735767b9414c could not connect: dial tcp 10.10.200.17:2380: i/o timeout\r\n2018-08-28 13:49:31.348188 W | etcdserver: failed to reach the peerURL(https://10.10.200.17:2380) of member 20d735767b9414c (Get https://10.10.200.17:2380/version: dial tcp 10.10.200.17:2380: i/o timeout)\r\n2018-08-28 13:49:31.348285 W | etcdserver: cannot get the version of member 20d735767b9414c (Get https://10.10.200.17:2380/version: dial tcp 10.10.200.17:2380: i/o timeout)\r\n2018-08-28 13:49:31.376388 W | etcdserver: not healthy for reconfigure, rejecting member add {ID:71a70df981ef548f RaftAttributes:{PeerURLs:[https://10.10.207.236:2380]} Attributes:{Name: ClientURLs:[]}}\r\n2018-08-28 13:49:31.376410 E | etcdserver/api/v2http: error adding member 71a70df981ef548f (etcdserver: unhealthy cluster)\r\n2018-08-28 13:49:31.376420 E | etcdserver/api/etcdhttp: etcdserver: unhealthy cluster\r\n**Results:**\r\n```\r\n\r\nShortly after checking the logs, both etcd nodes when from Active to Unavailable. Checked the logs again and saw that the etcd container had been restarted on both nodes.\r\nNow both nodes were spewing these logs:\r\n```\r\n2018-08-28 14:02:39.273558 E | rafthttp: request cluster ID mismatch (got 99936c54272f25cf want 422b22b9156bf8ec)\r\n2018-08-28 14:02:39.273848 E | rafthttp: request cluster ID mismatch (got 99936c54272f25cf want 422b22b9156bf8ec)\r\n2018-08-28 14:02:39.311173 E | rafthttp: request cluster ID mismatch (got 99936c54272f25cf want 422b22b9156bf8ec)\r\n2018-08-28 14:02:39.388036 E | rafthttp: request cluster ID mismatch (got 99936c54272f25cf want 422b22b9156bf8ec)\r\n2018-08-28 14:02:39.388300 E | rafthttp: request cluster ID mismatch (got 99936c54272f25cf want 422b22b9156bf8ec)\r\n```\r\n\r\nNo other action has been taken. I will either destroy this cluster and try again or poke at etcd for more logs and potential troubleshooting.\r\n\r\nI should also note that not much is running on this cluster at this time. And that when the etcd nodes went Unavailable the Master/Controlplane nodes started reporting `Kubelet has disk pressure`."},"comment":{"url":"https://api.github.com/repos/rancher/rancher/issues/comments/416627962","html_url":"https://github.com/rancher/rancher/issues/15269#issuecomment-416627962","issue_url":"https://api.github.com/repos/rancher/rancher/issues/15269","id":416627962,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNjYyNzk2Mg==","user":{"login":"superseb","id":2620179,"node_id":"MDQ6VXNlcjI2MjAxNzk=","avatar_url":"https://avatars2.githubusercontent.com/u/2620179?v=4","gravatar_id":"","url":"https://api.github.com/users/superseb","html_url":"https://github.com/superseb","followers_url":"https://api.github.com/users/superseb/followers","following_url":"https://api.github.com/users/superseb/following{/other_user}","gists_url":"https://api.github.com/users/superseb/gists{/gist_id}","starred_url":"https://api.github.com/users/superseb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/superseb/subscriptions","organizations_url":"https://api.github.com/users/superseb/orgs","repos_url":"https://api.github.com/users/superseb/repos","events_url":"https://api.github.com/users/superseb/events{/privacy}","received_events_url":"https://api.github.com/users/superseb/received_events","type":"User","site_admin":false},"created_at":"2018-08-28T15:20:40Z","updated_at":"2018-08-28T15:20:40Z","author_association":"MEMBER","body":"As discussed on Slack, etcd is not a component that is suited for dynamic nodes (e.g. ASG). The cluster needs to provision when the node is deleted, so the cluster gets in a correct state, and then a new node can be added. Lastly, depending on the amount of etcd nodes, you can have x failed nodes. (in this case, 1)."}}
{"actor":{"display_login":"superseb"}}