{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/etcd/issues/5244","repository_url":"https://api.github.com/repos/coreos/etcd","labels_url":"https://api.github.com/repos/coreos/etcd/issues/5244/labels{/name}","comments_url":"https://api.github.com/repos/coreos/etcd/issues/5244/comments","events_url":"https://api.github.com/repos/coreos/etcd/issues/5244/events","html_url":"https://github.com/coreos/etcd/issues/5244","id":152474805,"number":5244,"title":"cluster locks up if two members with same name","user":{"login":"llamahunter","id":843173,"avatar_url":"https://avatars.githubusercontent.com/u/843173?v=3","gravatar_id":"","url":"https://api.github.com/users/llamahunter","html_url":"https://github.com/llamahunter","followers_url":"https://api.github.com/users/llamahunter/followers","following_url":"https://api.github.com/users/llamahunter/following{/other_user}","gists_url":"https://api.github.com/users/llamahunter/gists{/gist_id}","starred_url":"https://api.github.com/users/llamahunter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/llamahunter/subscriptions","organizations_url":"https://api.github.com/users/llamahunter/orgs","repos_url":"https://api.github.com/users/llamahunter/repos","events_url":"https://api.github.com/users/llamahunter/events{/privacy}","received_events_url":"https://api.github.com/users/llamahunter/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":10,"created_at":"2016-05-02T03:13:07Z","updated_at":"2016-05-02T04:07:01Z","closed_at":null,"body":"I am not 100% sure what's going on, but I think there's a bug somewhere in etcd2 where it does not treat different members with the same name very well.  It seems to miscount them.\r\n\r\nI started with three nodes in my cluster.  I then added a 4th node, but typo'd the name such that it had the same name as an existing node, though it was assigned a unique member id.\r\n\r\nWhen I went to manually bring up this node in the cluster using the new ETCD_INITIAL_CLUSTER that listed 4 nodes, it would not come up, claiming\r\n```error validating peerURLs ...: member count is unequal```\r\n\r\nI suspect somewhere deep in etcd2, it is putting members listed in ETCD_INITIAL_CLUSTER into a hash table based on their name rather their member id, and so it comes up with 3 nodes, but when consulting the cluster member list organized by member id, it sees 4.\r\n\r\nAlong the way, one of the nodes died, so, now I have 2 nodes running in a 4 node cluster. But, I can't restart either the failed node or the new node because of the 'member count unequal' problem.  And I can't remove the typo'd duplicate node name via 'etcdctl member remove' (with a unique id) because the cluster is now considered 'unhealthy' due to lack of quorum.\r\n\r\nAny help in getting either of the duplicately named nodes without having to lose the entire cluster and restart from scratch?\r\n"},"comment":{"url":"https://api.github.com/repos/coreos/etcd/issues/comments/216101962","html_url":"https://github.com/coreos/etcd/issues/5244#issuecomment-216101962","issue_url":"https://api.github.com/repos/coreos/etcd/issues/5244","id":216101962,"user":{"login":"llamahunter","id":843173,"avatar_url":"https://avatars.githubusercontent.com/u/843173?v=3","gravatar_id":"","url":"https://api.github.com/users/llamahunter","html_url":"https://github.com/llamahunter","followers_url":"https://api.github.com/users/llamahunter/followers","following_url":"https://api.github.com/users/llamahunter/following{/other_user}","gists_url":"https://api.github.com/users/llamahunter/gists{/gist_id}","starred_url":"https://api.github.com/users/llamahunter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/llamahunter/subscriptions","organizations_url":"https://api.github.com/users/llamahunter/orgs","repos_url":"https://api.github.com/users/llamahunter/repos","events_url":"https://api.github.com/users/llamahunter/events{/privacy}","received_events_url":"https://api.github.com/users/llamahunter/received_events","type":"User","site_admin":false},"created_at":"2016-05-02T04:07:01Z","updated_at":"2016-05-02T04:07:01Z","body":"So, I sort of tried that already, but it fails spectacularly:\r\n<pre>\r\n# etcd2 -data-dir /var/lib/etcd2 -listen-client-urls http://0.0.0.0:2379 -advertise-client-urls http://10.243.228.246:2379  -listen-peer-urls http://10.243.228.246:2380 -initial-advertise-peer-urls http://10.243.228.246:2380\r\n2016-05-02 04:03:52.306751 I | flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=XXX=http://10.243.228.246:2380,core01.teab1.tivo.com=http://core01.teab1.tivo.com:2380,15b5345d69494950900fdc08f25e730e=http://10.243.50.139:2380,980e80e760d44f6a8e91bd68471928d5=http://10.243.128.162:2380\r\n2016-05-02 04:03:52.306788 I | flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=existing\r\n2016-05-02 04:03:52.306803 I | flags: recognized and used environment variable ETCD_NAME=XXX\r\n2016-05-02 04:03:52.306872 I | etcdmain: etcd Version: 2.2.3\r\n2016-05-02 04:03:52.306877 I | etcdmain: Git SHA: 05b564a\r\n2016-05-02 04:03:52.306881 I | etcdmain: Go Version: go1.4.3\r\n2016-05-02 04:03:52.306885 I | etcdmain: Go OS/Arch: linux/amd64\r\n2016-05-02 04:03:52.306890 I | etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 4\r\n2016-05-02 04:03:52.306977 I | etcdmain: listening for peers on http://10.243.228.246:2380\r\n2016-05-02 04:03:52.307006 I | etcdmain: listening for client requests on http://0.0.0.0:2379\r\n2016-05-02 04:03:52.310493 I | netutil: resolving core01.teab1.tivo.com:2380 to 10.243.228.246:2380\r\n2016-05-02 04:03:52.310555 I | netutil: resolving core01.teab1.tivo.com:2380 to 10.243.228.246:2380\r\n2016-05-02 04:03:57.311119 W | etcdserver: failed to reach the peerURL(http://core01.teab1.tivo.com:2380) of member 32124f388b63682a (Get http://core01.teab1.tivo.com:2380/version: read tcp 10.243.228.246:2380: i/o timeout)\r\n2016-05-02 04:03:57.311166 W | etcdserver: cannot get the version of member 32124f388b63682a (Get http://core01.teab1.tivo.com:2380/version: read tcp 10.243.228.246:2380: i/o timeout)\r\n2016-05-02 04:03:57.315504 I | etcdserver: name = XXX\r\n2016-05-02 04:03:57.315517 I | etcdserver: data dir = /var/lib/etcd2\r\n2016-05-02 04:03:57.315524 I | etcdserver: member dir = /var/lib/etcd2/member\r\n2016-05-02 04:03:57.315529 I | etcdserver: heartbeat = 100ms\r\n2016-05-02 04:03:57.315538 I | etcdserver: election = 1000ms\r\n2016-05-02 04:03:57.315543 I | etcdserver: snapshot count = 10000\r\n2016-05-02 04:03:57.315550 I | etcdserver: advertise client URLs = http://10.243.228.246:2379\r\n2016-05-02 04:03:57.323968 I | etcdserver: starting member 149291bba7d50725 in cluster 1b0a52aed542d6b0\r\n2016-05-02 04:03:57.324018 I | raft: 149291bba7d50725 became follower at term 0\r\n2016-05-02 04:03:57.324035 I | raft: newRaft 149291bba7d50725 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]\r\n2016-05-02 04:03:57.324050 I | raft: 149291bba7d50725 became follower at term 1\r\n2016-05-02 04:03:57.333377 I | etcdserver: starting server... [version: 2.2.3, cluster version: to_be_decided]\r\n2016-05-02 04:03:57.334909 E | etcdmain: failed to notify systemd for readiness: No socket\r\n2016-05-02 04:03:57.334937 E | etcdmain: forgot to set Type=notify in systemd service file?\r\n2016-05-02 04:03:57.336149 E | rafthttp: streaming request ignored (ID mismatch got 32124f388b63682a want 149291bba7d50725)\r\n2016-05-02 04:03:57.336294 E | rafthttp: streaming request ignored (ID mismatch got 32124f388b63682a want 149291bba7d50725)\r\n2016-05-02 04:03:57.336405 E | rafthttp: streaming request ignored (ID mismatch got 32124f388b63682a want 149291bba7d50725)\r\n2016-05-02 04:03:57.336550 E | rafthttp: streaming request ignored (ID mismatch got 32124f388b63682a want 149291bba7d50725)\r\n2016-05-02 04:03:57.338299 I | raft: 149291bba7d50725 [term: 1] received a MsgHeartbeat message with higher term from 80c09bc33831bc4e [term: 183]\r\n2016-05-02 04:03:57.338336 I | raft: 149291bba7d50725 became follower at term 183\r\n2016-05-02 04:03:57.338386 C | raft: tocommit(31449002) is out of range [lastIndex(0)]. Was the raft log corrupted, truncated, or lost?\r\npanic: tocommit(31449002) is out of range [lastIndex(0)]. Was the raft log corrupted, truncated, or lost?\r\n\r\n... lots and lots of go exception stuff elided ...\r\n</pre>"}}