{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/29180","repository_url":"https://api.github.com/repos/docker/docker","labels_url":"https://api.github.com/repos/docker/docker/issues/29180/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/29180/comments","events_url":"https://api.github.com/repos/docker/docker/issues/29180/events","html_url":"https://github.com/docker/docker/issues/29180","id":193800126,"number":29180,"title":"Unable to start dockerd on swarm manager. Error says \"tocommit(150010) is out of range [lastIndex(78775)]. Was the raft log corrupted\"","user":{"login":"herbrandson","id":430142,"avatar_url":"https://avatars.githubusercontent.com/u/430142?v=3","gravatar_id":"","url":"https://api.github.com/users/herbrandson","html_url":"https://github.com/herbrandson","followers_url":"https://api.github.com/users/herbrandson/followers","following_url":"https://api.github.com/users/herbrandson/following{/other_user}","gists_url":"https://api.github.com/users/herbrandson/gists{/gist_id}","starred_url":"https://api.github.com/users/herbrandson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/herbrandson/subscriptions","organizations_url":"https://api.github.com/users/herbrandson/orgs","repos_url":"https://api.github.com/users/herbrandson/repos","events_url":"https://api.github.com/users/herbrandson/events{/privacy}","received_events_url":"https://api.github.com/users/herbrandson/received_events","type":"User","site_admin":false},"labels":[{"id":392678049,"url":"https://api.github.com/repos/docker/docker/labels/area/swarm","name":"area/swarm","color":"fbca04","default":false},{"id":190258329,"url":"https://api.github.com/repos/docker/docker/labels/kind/bug","name":"kind/bug","color":"FF8C94","default":false}],"state":"open","locked":false,"assignee":{"login":"aaronlehmann","id":10601430,"avatar_url":"https://avatars.githubusercontent.com/u/10601430?v=3","gravatar_id":"","url":"https://api.github.com/users/aaronlehmann","html_url":"https://github.com/aaronlehmann","followers_url":"https://api.github.com/users/aaronlehmann/followers","following_url":"https://api.github.com/users/aaronlehmann/following{/other_user}","gists_url":"https://api.github.com/users/aaronlehmann/gists{/gist_id}","starred_url":"https://api.github.com/users/aaronlehmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aaronlehmann/subscriptions","organizations_url":"https://api.github.com/users/aaronlehmann/orgs","repos_url":"https://api.github.com/users/aaronlehmann/repos","events_url":"https://api.github.com/users/aaronlehmann/events{/privacy}","received_events_url":"https://api.github.com/users/aaronlehmann/received_events","type":"User","site_admin":false},"assignees":[{"login":"aaronlehmann","id":10601430,"avatar_url":"https://avatars.githubusercontent.com/u/10601430?v=3","gravatar_id":"","url":"https://api.github.com/users/aaronlehmann","html_url":"https://github.com/aaronlehmann","followers_url":"https://api.github.com/users/aaronlehmann/followers","following_url":"https://api.github.com/users/aaronlehmann/following{/other_user}","gists_url":"https://api.github.com/users/aaronlehmann/gists{/gist_id}","starred_url":"https://api.github.com/users/aaronlehmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aaronlehmann/subscriptions","organizations_url":"https://api.github.com/users/aaronlehmann/orgs","repos_url":"https://api.github.com/users/aaronlehmann/repos","events_url":"https://api.github.com/users/aaronlehmann/events{/privacy}","received_events_url":"https://api.github.com/users/aaronlehmann/received_events","type":"User","site_admin":false}],"milestone":null,"comments":16,"created_at":"2016-12-06T15:23:00Z","updated_at":"2016-12-17T00:49:02Z","closed_at":null,"body":"\r\n\r\n**Description**\r\nI am unable to start docker on one of the manager nodes of my swarm (IMPORTANT: docker swarm mode, not docker swarm).\r\n\r\n**Output of `docker version`:**\r\n```\r\nClient:\r\n Version:      1.12.3\r\n API version:  1.24\r\n Go version:   go1.6.3\r\n Git commit:   6b644ec\r\n Built:        Wed Oct 26 22:01:48 2016\r\n OS/Arch:      linux/amd64\r\nCannot connect to the Docker daemon. Is the docker daemon running on this host?\r\n```\r\n\r\n**Output of `docker info`:**\r\nI'm unable to run this command because docker won't start\r\n\r\n**Additional environment details (AWS, VirtualBox, physical, etc.):**\r\n7 Node swarm with 3 manager nodes. Running in a dedicated VPS on AWS. All nodes are running Ubuntu 16.04.1 LTS\r\n\r\n**Output of `dockerd`:**\r\n```\r\nINFO[0000] libcontainerd: new containerd process, pid: 1693 \r\nWARN[0000] containerd: low RLIMIT_NOFILE changing to max  current=1024 max=65536\r\nINFO[0001] [graphdriver] using prior storage driver \"aufs\" \r\nINFO[0001] Graph migration to content-addressability took 0.00 seconds \r\nWARN[0001] Your kernel does not support swap memory limit. \r\nINFO[0001] Loading containers: start.                   \r\n.................................................................................INFO[0001] Firewalld running: false                     \r\nINFO[0001] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address \r\n\r\nINFO[0001] Loading containers: done.                    \r\nINFO[0001] Listening for local connections               addr=/var/lib/docker/swarm/control.sock proto=unix\r\nINFO[0001] Listening for connections                     addr=[::]:2377 proto=tcp\r\nWARN[0001] ignoring request to join cluster, because raft state already exists \r\nINFO[0001] 52b75a6e6dd83823 became follower at term 2   \r\nINFO[0001] newRaft 52b75a6e6dd83823 [peers: [8475b40f5c3b344,197af42d1ac22e90,52b75a6e6dd83823], term: 2, commit: 78774, applied: 70000, lastindex: 78775, lastterm: 2] \r\nINFO[0002] 52b75a6e6dd83823 [term: 2] received a MsgHeartbeat message with higher term from 197af42d1ac22e90 [term: 14] \r\nINFO[0002] 52b75a6e6dd83823 became follower at term 14  \r\nPANI[0002] tocommit(150010) is out of range [lastIndex(78775)]. Was the raft log corrupted, truncated, or lost? \r\npanic: (*logrus.Entry) (0x1d275e0,0xc824fda400)\r\n\r\ngoroutine 520 [running]:\r\npanic(0x1d275e0, 0xc824fda400)\r\n\t/usr/local/go/src/runtime/panic.go:481 +0x3e6\r\ngithub.com/Sirupsen/logrus.Entry.log(0xc82004c1c0, 0xc8203c68d0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc8246d9a40, ...)\r\n\t/usr/src/docker/vendor/src/github.com/Sirupsen/logrus/entry.go:113 +0x62c\r\ngithub.com/Sirupsen/logrus.(*Entry).Panic(0xc82004da00, 0xc82527c3b8, 0x1, 0x1)\r\n\t/usr/src/docker/vendor/src/github.com/Sirupsen/logrus/entry.go:158 +0x99\r\ngithub.com/Sirupsen/logrus.(*Entry).Panicf(0xc82004da00, 0x20d2dc0, 0x5d, 0xc825267560, 0x2, 0x2)\r\n\t/usr/src/docker/vendor/src/github.com/Sirupsen/logrus/entry.go:206 +0x139\r\ngithub.com/coreos/etcd/raft.(*raftLog).commitTo(0xc82110d260, 0x249fa)\r\n\t/usr/src/docker/vendor/src/github.com/coreos/etcd/raft/log.go:194 +0x1a6\r\ngithub.com/coreos/etcd/raft.(*raft).handleHeartbeat(0xc820d3d2c0, 0x8, 0x52b75a6e6dd83823, 0x197af42d1ac22e90, 0xe, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n\t/usr/src/docker/vendor/src/github.com/coreos/etcd/raft/raft.go:771 +0x44\r\ngithub.com/coreos/etcd/raft.stepFollower(0xc820d3d2c0, 0x8, 0x52b75a6e6dd83823, 0x197af42d1ac22e90, 0xe, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n\t/usr/src/docker/vendor/src/github.com/coreos/etcd/raft/raft.go:736 +0x119c\r\ngithub.com/coreos/etcd/raft.(*raft).Step(0xc820d3d2c0, 0x8, 0x52b75a6e6dd83823, 0x197af42d1ac22e90, 0xe, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n\t/usr/src/docker/vendor/src/github.com/coreos/etcd/raft/raft.go:564 +0x3e0\r\ngithub.com/coreos/etcd/raft.(*node).run(0xc822c5bbd0, 0xc820d3d2c0)\r\n\t/usr/src/docker/vendor/src/github.com/coreos/etcd/raft/node.go:310 +0x90e\r\ncreated by github.com/coreos/etcd/raft.RestartNode\r\n\t/usr/src/docker/vendor/src/github.com/coreos/etcd/raft/node.go:215 +0x2e4\r\n```"},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/267730366","html_url":"https://github.com/docker/docker/issues/29180#issuecomment-267730366","issue_url":"https://api.github.com/repos/docker/docker/issues/29180","id":267730366,"user":{"login":"aaronlehmann","id":10601430,"avatar_url":"https://avatars.githubusercontent.com/u/10601430?v=3","gravatar_id":"","url":"https://api.github.com/users/aaronlehmann","html_url":"https://github.com/aaronlehmann","followers_url":"https://api.github.com/users/aaronlehmann/followers","following_url":"https://api.github.com/users/aaronlehmann/following{/other_user}","gists_url":"https://api.github.com/users/aaronlehmann/gists{/gist_id}","starred_url":"https://api.github.com/users/aaronlehmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aaronlehmann/subscriptions","organizations_url":"https://api.github.com/users/aaronlehmann/orgs","repos_url":"https://api.github.com/users/aaronlehmann/repos","events_url":"https://api.github.com/users/aaronlehmann/events{/privacy}","received_events_url":"https://api.github.com/users/aaronlehmann/received_events","type":"User","site_admin":false},"created_at":"2016-12-17T00:49:01Z","updated_at":"2016-12-17T00:49:01Z","body":"With help from @dongluochen and @cyli, I think I've been able to understand what's happening here, and I've opened some PRs in SwarmKit to address the problems.\r\n\r\nhttps://github.com/docker/swarmkit/pull/1815#issuecomment-267703760 explains a few cases that I think can lead to this panic. The first of them is fixed by that PR. The second is fixed by https://github.com/docker/swarmkit/pull/1820. The third will be fixed by providing a reverse migration tool to support downgrades from 1.13 to 1.12."}}
{"actor":{"display_login":"aaronlehmann"}}