{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/etcd/issues/5244","repository_url":"https://api.github.com/repos/coreos/etcd","labels_url":"https://api.github.com/repos/coreos/etcd/issues/5244/labels{/name}","comments_url":"https://api.github.com/repos/coreos/etcd/issues/5244/comments","events_url":"https://api.github.com/repos/coreos/etcd/issues/5244/events","html_url":"https://github.com/coreos/etcd/issues/5244","id":152474805,"number":5244,"title":"cluster locks up if two members with same name","user":{"login":"llamahunter","id":843173,"avatar_url":"https://avatars.githubusercontent.com/u/843173?v=3","gravatar_id":"","url":"https://api.github.com/users/llamahunter","html_url":"https://github.com/llamahunter","followers_url":"https://api.github.com/users/llamahunter/followers","following_url":"https://api.github.com/users/llamahunter/following{/other_user}","gists_url":"https://api.github.com/users/llamahunter/gists{/gist_id}","starred_url":"https://api.github.com/users/llamahunter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/llamahunter/subscriptions","organizations_url":"https://api.github.com/users/llamahunter/orgs","repos_url":"https://api.github.com/users/llamahunter/repos","events_url":"https://api.github.com/users/llamahunter/events{/privacy}","received_events_url":"https://api.github.com/users/llamahunter/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":24,"created_at":"2016-05-02T03:13:07Z","updated_at":"2016-05-02T04:49:36Z","closed_at":null,"body":"I am not 100% sure what's going on, but I think there's a bug somewhere in etcd2 where it does not treat different members with the same name very well.  It seems to miscount them.\r\n\r\nI started with three nodes in my cluster.  I then added a 4th node, but typo'd the name such that it had the same name as an existing node, though it was assigned a unique member id.\r\n\r\nWhen I went to manually bring up this node in the cluster using the new ETCD_INITIAL_CLUSTER that listed 4 nodes, it would not come up, claiming\r\n```error validating peerURLs ...: member count is unequal```\r\n\r\nI suspect somewhere deep in etcd2, it is putting members listed in ETCD_INITIAL_CLUSTER into a hash table based on their name rather their member id, and so it comes up with 3 nodes, but when consulting the cluster member list organized by member id, it sees 4.\r\n\r\nAlong the way, one of the nodes died, so, now I have 2 nodes running in a 4 node cluster. But, I can't restart either the failed node or the new node because of the 'member count unequal' problem.  And I can't remove the typo'd duplicate node name via 'etcdctl member remove' (with a unique id) because the cluster is now considered 'unhealthy' due to lack of quorum.\r\n\r\nAny help in getting either of the duplicately named nodes without having to lose the entire cluster and restart from scratch?\r\n"},"comment":{"url":"https://api.github.com/repos/coreos/etcd/issues/comments/216104706","html_url":"https://github.com/coreos/etcd/issues/5244#issuecomment-216104706","issue_url":"https://api.github.com/repos/coreos/etcd/issues/5244","id":216104706,"user":{"login":"llamahunter","id":843173,"avatar_url":"https://avatars.githubusercontent.com/u/843173?v=3","gravatar_id":"","url":"https://api.github.com/users/llamahunter","html_url":"https://github.com/llamahunter","followers_url":"https://api.github.com/users/llamahunter/followers","following_url":"https://api.github.com/users/llamahunter/following{/other_user}","gists_url":"https://api.github.com/users/llamahunter/gists{/gist_id}","starred_url":"https://api.github.com/users/llamahunter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/llamahunter/subscriptions","organizations_url":"https://api.github.com/users/llamahunter/orgs","repos_url":"https://api.github.com/users/llamahunter/repos","events_url":"https://api.github.com/users/llamahunter/events{/privacy}","received_events_url":"https://api.github.com/users/llamahunter/received_events","type":"User","site_admin":false},"created_at":"2016-05-02T04:49:36Z","updated_at":"2016-05-02T04:49:36Z","body":"ah... ok... well, good to know. I tried that, and it *looked* like it worked.  The failing node was able to rejoin, but then, a short time later, all three nodes went down simultaneously due to\r\n\r\n<pre>\r\nMay 02 04:42:19 core03.teab1.tivo.com etcd2[10967]: nodeToMember should never fail: raftAttributes key doesn't exist\r\nMay 02 04:42:20 core03.teab1.tivo.com etcd2[10967]: failed to write 32124f388b63682a on pipeline (net/http: request canceled while waiting for connection)\r\nMay 02 04:42:20 core03.teab1.tivo.com systemd[1]: etcd2.service: Main process exited, code=exited, status=2/INVALIDARGUMENT\r\nMay 02 04:42:20 core03.teab1.tivo.com systemd[1]: etcd2.service: Unit entered failed state.\r\nMay 02 04:42:20 core03.teab1.tivo.com systemd[1]: etcd2.service: Failed with result 'exit-code'.\r\n</pre>\r\n\r\nSo, now I am definitely in disaster recovery mode, as there are no longer *any* members running.  Fortunately, I took an etcd2 backup before doing this, so I'm off to learn about rebuilding a cluster from a backup now.\r\n\r\nThanks for the help."}}