{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/36555","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/36555/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/36555/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/36555/events","html_url":"https://github.com/kubernetes/kubernetes/issues/36555","id":188412261,"number":36555,"title":"etcd rollback tool panic","user":{"login":"mml","id":3115,"avatar_url":"https://avatars.githubusercontent.com/u/3115?v=3","gravatar_id":"","url":"https://api.github.com/users/mml","html_url":"https://github.com/mml","followers_url":"https://api.github.com/users/mml/followers","following_url":"https://api.github.com/users/mml/following{/other_user}","gists_url":"https://api.github.com/users/mml/gists{/gist_id}","starred_url":"https://api.github.com/users/mml/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mml/subscriptions","organizations_url":"https://api.github.com/users/mml/orgs","repos_url":"https://api.github.com/users/mml/repos","events_url":"https://api.github.com/users/mml/events{/privacy}","received_events_url":"https://api.github.com/users/mml/received_events","type":"User","site_admin":false},"labels":[{"id":136601536,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/component/apiserver","name":"component/apiserver","color":"0052cc","default":false},{"id":127257343,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/dependency/etcd","name":"dependency/etcd","color":"fbfa04","default":false},{"id":105146071,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21","default":false},{"id":433683967,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/non-release-blocker","name":"non-release-blocker","color":"0e8a16","default":false},{"id":114528223,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P1","name":"priority/P1","color":"eb6420","default":false},{"id":173493835,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/CSI-API%20Machinery%20SIG","name":"team/CSI-API Machinery SIG","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":{"login":"xiang90","id":4479947,"avatar_url":"https://avatars.githubusercontent.com/u/4479947?v=3","gravatar_id":"","url":"https://api.github.com/users/xiang90","html_url":"https://github.com/xiang90","followers_url":"https://api.github.com/users/xiang90/followers","following_url":"https://api.github.com/users/xiang90/following{/other_user}","gists_url":"https://api.github.com/users/xiang90/gists{/gist_id}","starred_url":"https://api.github.com/users/xiang90/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xiang90/subscriptions","organizations_url":"https://api.github.com/users/xiang90/orgs","repos_url":"https://api.github.com/users/xiang90/repos","events_url":"https://api.github.com/users/xiang90/events{/privacy}","received_events_url":"https://api.github.com/users/xiang90/received_events","type":"User","site_admin":false},"assignees":[{"login":"xiang90","id":4479947,"avatar_url":"https://avatars.githubusercontent.com/u/4479947?v=3","gravatar_id":"","url":"https://api.github.com/users/xiang90","html_url":"https://github.com/xiang90","followers_url":"https://api.github.com/users/xiang90/followers","following_url":"https://api.github.com/users/xiang90/following{/other_user}","gists_url":"https://api.github.com/users/xiang90/gists{/gist_id}","starred_url":"https://api.github.com/users/xiang90/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xiang90/subscriptions","organizations_url":"https://api.github.com/users/xiang90/orgs","repos_url":"https://api.github.com/users/xiang90/repos","events_url":"https://api.github.com/users/xiang90/events{/privacy}","received_events_url":"https://api.github.com/users/xiang90/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/23","html_url":"https://github.com/kubernetes/kubernetes/milestones/v1.5","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/23/labels","id":1945978,"number":23,"title":"v1.5","description":"","creator":{"login":"lavalamp","id":647318,"avatar_url":"https://avatars.githubusercontent.com/u/647318?v=3","gravatar_id":"","url":"https://api.github.com/users/lavalamp","html_url":"https://github.com/lavalamp","followers_url":"https://api.github.com/users/lavalamp/followers","following_url":"https://api.github.com/users/lavalamp/following{/other_user}","gists_url":"https://api.github.com/users/lavalamp/gists{/gist_id}","starred_url":"https://api.github.com/users/lavalamp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lavalamp/subscriptions","organizations_url":"https://api.github.com/users/lavalamp/orgs","repos_url":"https://api.github.com/users/lavalamp/repos","events_url":"https://api.github.com/users/lavalamp/events{/privacy}","received_events_url":"https://api.github.com/users/lavalamp/received_events","type":"User","site_admin":false},"open_issues":114,"closed_issues":961,"state":"open","created_at":"2016-08-16T21:47:28Z","updated_at":"2016-12-06T02:05:36Z","due_on":"2016-12-09T00:00:00Z","closed_at":null},"comments":13,"created_at":"2016-11-10T03:30:54Z","updated_at":"2016-12-06T02:05:36Z","closed_at":null,"body":"I tried to do a rollback on my own test cluster and got a panic.  I will attach the data-dir tarball.  The script that performs the migration goes from 2.2.1->2.3.7->3.0.14->2.3.7, and it is this last step that barfs.  Script is forthcoming in a PR, but there's a snapshot at https://gist.github.com/mml/42e57c45b333f8b1c8c29bb162394e7d\r\n\r\n```\r\n Performing etcd3 -> etcd2 rollback\r\n + /usr/local/bin/rollback --data-dir /var/etcd/data\r\n 2016-11-10 03:18:07.226168 C | membership: error updating attributes of unknown member cf7d617fc919f571\r\n panic: error updating attributes of unknown member cf7d617fc919f571\r\n\r\n goroutine 1 [running]:\r\n panic(0xae1600, 0xc8211be610)\r\n         /usr/local/go/src/runtime/panic.go:481 +0x3e6\r\n k8s.io/kubernetes/vendor/github.com/coreos/pkg/capnslog.(*PackageLogger).Panicf(0xc820138ac0, 0xe5b340, 0x2e, 0xc8219112e8, 0x1, 0x1)\r\n         /go/src/k8s.io/kubernetes/vendor/github.com/coreos/pkg/capnslog/pkg_logger.go:75 +0x191\r\n k8s.io/kubernetes/vendor/github.com/coreos/etcd/etcdserver/membership.(*RaftCluster).UpdateAttributes(0xc8213c2b40, 0xcf7d617fc919f571, 0xc8211be520, 0xb, 0xc8211c8f80, 0x1, 0x4)\r\n         /go/src/k8s.io/kubernetes/vendor/github.com/coreos/etcd/etcdserver/membership/cluster.go:331 +0x2e5\r\n k8s.io/kubernetes/vendor/github.com/coreos/etcd/etcdserver.(*applierV2store).Put(0xc8202a2be0, 0xc8214d11e0, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n         /go/src/k8s.io/kubernetes/vendor/github.com/coreos/etcd/etcdserver/apply_v2.go:83 +0x765\r\n main.applyRequest(0xc8214d11e0, 0x7f4c439473c0, 0xc8202a2be0)\r\n         /go/src/k8s.io/kubernetes/cluster/images/etcd/rollback/rollback.go:321 +0x1cf\r\n main.rebuild(0x7ffc57679f18, 0xe, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n         /go/src/k8s.io/kubernetes/cluster/images/etcd/rollback/rollback.go:296 +0xe8d\r\n main.main()\r\n         /go/src/k8s.io/kubernetes/cluster/images/etcd/rollback/rollback.go:103 +0x70c\r\n```\r\n\r\n@wojtek-t @hongchaodeng "},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/265039349","html_url":"https://github.com/kubernetes/kubernetes/issues/36555#issuecomment-265039349","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/36555","id":265039349,"user":{"login":"mml","id":3115,"avatar_url":"https://avatars.githubusercontent.com/u/3115?v=3","gravatar_id":"","url":"https://api.github.com/users/mml","html_url":"https://github.com/mml","followers_url":"https://api.github.com/users/mml/followers","following_url":"https://api.github.com/users/mml/following{/other_user}","gists_url":"https://api.github.com/users/mml/gists{/gist_id}","starred_url":"https://api.github.com/users/mml/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mml/subscriptions","organizations_url":"https://api.github.com/users/mml/orgs","repos_url":"https://api.github.com/users/mml/repos","events_url":"https://api.github.com/users/mml/events{/privacy}","received_events_url":"https://api.github.com/users/mml/received_events","type":"User","site_admin":false},"created_at":"2016-12-06T02:05:36Z","updated_at":"2016-12-06T02:05:36Z","body":"\r\nHmm...  I tried re-running this on my test data, and now 2.3.7 **always** panics after the last step, whether I begin with backup data or not.\r\n\r\n```\r\n panic: runtime error: makeslice: len out of range\r\n \r\n goroutine 1 [running]:\r\n panic(0xdc26c0, 0xc8203ec880)\r\n         /usr/local/go/src/runtime/panic.go:481 +0x3e6\r\n github.com/coreos/etcd/wal.(*decoder).decode(0xc82039f860, 0xc8201fdf90, 0x0, 0x0)\r\n         /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/wal/decoder.go:55 +0x142\r\n github.com/coreos/etcd/wal.(*WAL).ReadAll(0xc82034fd40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n         /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/wal/wal.go:237 +0x214\r\n github.com/coreos/etcd/etcdserver.readWAL(0xc82036fca0, 0x19, 0xf268, 0x8, 0x0, 0x0, 0x0, 0xc82034fd40, 0x1, 0xc8203ec6c0, ...)\r\n         /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/etcdserver/storage.go:87 +0x228\r\n github.com/coreos/etcd/etcdserver.restartNode(0xc82008f180, 0xc820081560, 0x29, 0xc8201fe950, 0x0, 0x0, 0x0, 0x0)\r\n         /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/etcdserver/raft.go:334 +0xff\r\n github.com/coreos/etcd/etcdserver.NewServer(0xc82008f180, 0xc82008f238, 0x0, 0x0)\r\n         /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/etcdserver/server.go:333 +0x3be5\r\n github.com/coreos/etcd/etcdmain.startEtcd(0xc820090f00, 0x0, 0x0, 0x0)\r\n         /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/etcdmain/etcd.go:302 +0x1b40\r\n github.com/coreos/etcd/etcdmain.Main()\r\n         /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/etcdmain/etcd.go:118 +0x2142\r\n main.main()\r\n         /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/main.go:29 +0x14\r\n```\r\n\r\nHere is the data\r\n - before running rollback: [snapshot-before-rollback-2.3.7.tar.gz](https://github.com/kubernetes/kubernetes/files/632789/snapshot-before-rollback-2.3.7.tar.gz)\r\n - after running rollback: [snapshot-post-rollback-2.3.7.tar.gz](https://github.com/kubernetes/kubernetes/files/632790/snapshot-post-rollback-2.3.7.tar.gz)"}}
{"actor":{"display_login":"mml"}}