{"action":"created","issue":{"url":"https://api.github.com/repos/rancher/rancher/issues/5626","repository_url":"https://api.github.com/repos/rancher/rancher","labels_url":"https://api.github.com/repos/rancher/rancher/issues/5626/labels{/name}","comments_url":"https://api.github.com/repos/rancher/rancher/issues/5626/comments","events_url":"https://api.github.com/repos/rancher/rancher/issues/5626/events","html_url":"https://github.com/rancher/rancher/issues/5626","id":169193969,"number":5626,"title":" etcd panic with a  3 host Kubernetes set up","user":{"login":"soumyalj","id":18536626,"avatar_url":"https://avatars.githubusercontent.com/u/18536626?v=3","gravatar_id":"","url":"https://api.github.com/users/soumyalj","html_url":"https://github.com/soumyalj","followers_url":"https://api.github.com/users/soumyalj/followers","following_url":"https://api.github.com/users/soumyalj/following{/other_user}","gists_url":"https://api.github.com/users/soumyalj/gists{/gist_id}","starred_url":"https://api.github.com/users/soumyalj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/soumyalj/subscriptions","organizations_url":"https://api.github.com/users/soumyalj/orgs","repos_url":"https://api.github.com/users/soumyalj/repos","events_url":"https://api.github.com/users/soumyalj/events{/privacy}","received_events_url":"https://api.github.com/users/soumyalj/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rancher/rancher/labels/area/kubernetes","name":"area/kubernetes","color":"fbca04"},{"url":"https://api.github.com/repos/rancher/rancher/labels/kind/bug","name":"kind/bug","color":"fc2929"}],"state":"closed","locked":false,"assignee":{"login":"LLParse","id":6145659,"avatar_url":"https://avatars.githubusercontent.com/u/6145659?v=3","gravatar_id":"","url":"https://api.github.com/users/LLParse","html_url":"https://github.com/LLParse","followers_url":"https://api.github.com/users/LLParse/followers","following_url":"https://api.github.com/users/LLParse/following{/other_user}","gists_url":"https://api.github.com/users/LLParse/gists{/gist_id}","starred_url":"https://api.github.com/users/LLParse/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/LLParse/subscriptions","organizations_url":"https://api.github.com/users/LLParse/orgs","repos_url":"https://api.github.com/users/LLParse/repos","events_url":"https://api.github.com/users/LLParse/events{/privacy}","received_events_url":"https://api.github.com/users/LLParse/received_events","type":"User","site_admin":false},"assignees":[{"login":"LLParse","id":6145659,"avatar_url":"https://avatars.githubusercontent.com/u/6145659?v=3","gravatar_id":"","url":"https://api.github.com/users/LLParse","html_url":"https://github.com/LLParse","followers_url":"https://api.github.com/users/LLParse/followers","following_url":"https://api.github.com/users/LLParse/following{/other_user}","gists_url":"https://api.github.com/users/LLParse/gists{/gist_id}","starred_url":"https://api.github.com/users/LLParse/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/LLParse/subscriptions","organizations_url":"https://api.github.com/users/LLParse/orgs","repos_url":"https://api.github.com/users/LLParse/repos","events_url":"https://api.github.com/users/LLParse/events{/privacy}","received_events_url":"https://api.github.com/users/LLParse/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/rancher/rancher/milestones/67","html_url":"https://github.com/rancher/rancher/milestones/Release%201.2.0","labels_url":"https://api.github.com/repos/rancher/rancher/milestones/67/labels","id":1788744,"number":67,"title":"Release 1.2.0","description":"","creator":{"login":"will-chan","id":10134676,"avatar_url":"https://avatars.githubusercontent.com/u/10134676?v=3","gravatar_id":"","url":"https://api.github.com/users/will-chan","html_url":"https://github.com/will-chan","followers_url":"https://api.github.com/users/will-chan/followers","following_url":"https://api.github.com/users/will-chan/following{/other_user}","gists_url":"https://api.github.com/users/will-chan/gists{/gist_id}","starred_url":"https://api.github.com/users/will-chan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/will-chan/subscriptions","organizations_url":"https://api.github.com/users/will-chan/orgs","repos_url":"https://api.github.com/users/will-chan/repos","events_url":"https://api.github.com/users/will-chan/events{/privacy}","received_events_url":"https://api.github.com/users/will-chan/received_events","type":"User","site_admin":false},"open_issues":296,"closed_issues":48,"state":"open","created_at":"2016-05-26T00:51:12Z","updated_at":"2016-09-28T21:09:02Z","due_on":"2016-10-28T07:00:00Z","closed_at":null},"comments":1,"created_at":"2016-08-03T17:58:28Z","updated_at":"2016-09-28T21:09:02Z","closed_at":"2016-09-28T21:09:02Z","body":"**Rancher Version:**: 1.1.2\r\n\r\n**Docker Version:**: 1.10.3\r\n\r\n**OS and where are the hosts located? (cloud, bare metal, etc):**Ubuntu, Cloud\r\n\r\n**Environment Type: (Cattle/Kubernetes/Swarm/Mesos)**: Kuberentes\r\n\r\n**Steps to Reproduce:**\r\n1. Start  Kubernetes setup with 3 hosts using https://github.com/rancher/qa-catalog\r\n2. Create pods/rcs\r\n3. Power down 2 hosts. \r\n4. Power back 2 host after 5 mins\r\n\r\n**Results:** \r\netcds did not self heal.\r\nThere was a panic .  \"8/2/2016 8:22:40 PMpanic: tocommit(22358) is out of range [lastIndex(0)]. Was the raft log corrupted, truncated, or lost?\" (edited)\r\n\r\n```\r\n8/2/2016 8:27:17 PM2016-08-03 03:27:17.412170 I | etcdserver: restarting member 1dce51dc03bae852 in cluster 564dd555b702 at commit index 0\r\n8/2/2016 8:27:17 PM2016-08-03 03:27:17.412717 I | raft: 1dce51dc03bae852 became follower at term 1\r\n8/2/2016 8:27:17 PM2016-08-03 03:27:17.412739 I | raft: newRaft 1dce51dc03bae852 [peers: [], term: 1, commit: 0, applied: 0, lastindex: 0, lastterm: 0]\r\n8/2/2016 8:27:17 PM2016-08-03 03:27:17.413050 I | etcdserver: starting server... [version: 2.3.7, cluster version: to_be_decided]\r\n8/2/2016 8:27:17 PM2016-08-03 03:27:17.429882 I | raft: 1dce51dc03bae852 [term: 1] received a MsgHeartbeat message with higher term from 564dd555b701 [term: 668]\r\n8/2/2016 8:27:17 PM2016-08-03 03:27:17.430044 I | raft: 1dce51dc03bae852 became follower at term 668\r\n8/2/2016 8:27:17 PM2016-08-03 03:27:17.430292 C | raft: tocommit(22358) is out of range [lastIndex(0)]. Was the raft log corrupted, truncated, or lost?\r\n8/2/2016 8:27:17 PMpanic: tocommit(22358) is out of range [lastIndex(0)]. Was the raft log corrupted, truncated, or lost?\r\n8/2/2016 8:27:17 PM\r\n8/2/2016 8:27:17 PMgoroutine 55 [running]:\r\n8/2/2016 8:27:17 PMpanic(0xbeaf00, 0xc82012c3e0)\r\n8/2/2016 8:27:17 PM     /usr/local/go/src/runtime/panic.go:481 +0x3e6\r\n8/2/2016 8:27:17 PMgithub.com/coreos/etcd/Godeps/_workspace/src/github.com/coreos/pkg/capnslog.(*PackageLogger).Panicf(0xc8201a9a80, 0x10de7a0, 0x5d, 0xc8200daa40, 0x2, 0x2)\r\n8/2/2016 8:27:17 PM     /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/Godeps/_workspace/src/github.com/coreos/pkg/capnslog/pkg_logger.go:73 +0x191\r\n8/2/2016 8:27:17 PMgithub.com/coreos/etcd/raft.(*raftLog).commitTo(0xc8201e69a0, 0x5756)\r\n8/2/2016 8:27:17 PM     /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/raft/log.go:194 +0x1a6\r\n8/2/2016 8:27:17 PMgithub.com/coreos/etcd/raft.(*raft).handleHeartbeat(0xc8201c8240, 0x8, 0x1dce51dc03bae852, 0x564dd555b701, 0x29c, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n8/2/2016 8:27:17 PM     /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/raft/raft.go:771 +0x44\r\n8/2/2016 8:27:17 PMgithub.com/coreos/etcd/raft.stepFollower(0xc8201c8240, 0x8, 0x1dce51dc03bae852, 0x564dd555b701, 0x29c, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n8/2/2016 8:27:17 PM     /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/raft/raft.go:736 +0x119c\r\n8/2/2016 8:27:17 PMgithub.com/coreos/etcd/raft.(*raft).Step(0xc8201c8240, 0x8, 0x1dce51dc03bae852, 0x564dd555b701, 0x29c, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n8/2/2016 8:27:17 PM     /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/raft/raft.go:564 +0x3e0\r\n8/2/2016 8:27:17 PMgithub.com/coreos/etcd/raft.(*node).run(0xc8201ec5a0, 0xc8201c8240)\r\n8/2/2016 8:27:17 PM     /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/raft/node.go:310 +0x90e\r\n8/2/2016 8:27:17 PMcreated by github.com/coreos/etcd/raft.RestartNode\r\n8/2/2016 8:27:17 PM     /home/gyuho/go/src/github.com/coreos/etcd/release/etcd/gopath/src/github.com/coreos/etcd/raft/node.go:215 +0x2e4\r\n\r\n```\r\n\r\n**Expected:**\r\nIt should be a self healing scenario\r\n"},"comment":{"url":"https://api.github.com/repos/rancher/rancher/issues/comments/250300705","html_url":"https://github.com/rancher/rancher/issues/5626#issuecomment-250300705","issue_url":"https://api.github.com/repos/rancher/rancher/issues/5626","id":250300705,"user":{"login":"LLParse","id":6145659,"avatar_url":"https://avatars.githubusercontent.com/u/6145659?v=3","gravatar_id":"","url":"https://api.github.com/users/LLParse","html_url":"https://github.com/LLParse","followers_url":"https://api.github.com/users/LLParse/followers","following_url":"https://api.github.com/users/LLParse/following{/other_user}","gists_url":"https://api.github.com/users/LLParse/gists{/gist_id}","starred_url":"https://api.github.com/users/LLParse/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/LLParse/subscriptions","organizations_url":"https://api.github.com/users/LLParse/orgs","repos_url":"https://api.github.com/users/LLParse/repos","events_url":"https://api.github.com/users/LLParse/events{/privacy}","received_events_url":"https://api.github.com/users/LLParse/received_events","type":"User","site_admin":false},"created_at":"2016-09-28T21:09:02Z","updated_at":"2016-09-28T21:09:02Z","body":"Fixed in `etcd:v2.3.7-9` "}}
{"actor":{"display_login":"LLParse"}}