{"action":"created","issue":{"url":"https://api.github.com/repos/cloudfoundry-incubator/consul-release/issues/27","repository_url":"https://api.github.com/repos/cloudfoundry-incubator/consul-release","labels_url":"https://api.github.com/repos/cloudfoundry-incubator/consul-release/issues/27/labels{/name}","comments_url":"https://api.github.com/repos/cloudfoundry-incubator/consul-release/issues/27/comments","events_url":"https://api.github.com/repos/cloudfoundry-incubator/consul-release/issues/27/events","html_url":"https://github.com/cloudfoundry-incubator/consul-release/issues/27","id":162198148,"number":27,"title":"Consul server didn't recover when clients were also down","user":{"login":"youngm","id":438661,"avatar_url":"https://avatars.githubusercontent.com/u/438661?v=3","gravatar_id":"","url":"https://api.github.com/users/youngm","html_url":"https://github.com/youngm","followers_url":"https://api.github.com/users/youngm/followers","following_url":"https://api.github.com/users/youngm/following{/other_user}","gists_url":"https://api.github.com/users/youngm/gists{/gist_id}","starred_url":"https://api.github.com/users/youngm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/youngm/subscriptions","organizations_url":"https://api.github.com/users/youngm/orgs","repos_url":"https://api.github.com/users/youngm/repos","events_url":"https://api.github.com/users/youngm/events{/privacy}","received_events_url":"https://api.github.com/users/youngm/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/cloudfoundry-incubator/consul-release/labels/scheduled","name":"scheduled","color":"BFD4F2"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-06-24T17:54:20Z","updated_at":"2016-07-02T01:17:54Z","closed_at":null,"body":"We experienced a partial datacenter outage recently and while bringing things back up I discovered that I couldn't bring one of my 3 consul servers back up.  From looking at the logs it appears that the server couldn't come up because some of the clients were down.\r\n\r\nOnce all the clients recovered then the server was able to start.\r\n\r\nI haven't yet attempted to duplicate the issue.  If I find the time I will update the issue with some exact steps.  I imagine I could duplicate the issue by disconnecting some clients from the network and then crashing a node of a 3 node cluster.\r\n\r\nHere are the applicable logs:\r\n\r\n```\r\n{\"timestamp\":\"1466775061.845097780\",\"source\":\"confab\",\"message\":\"confab.agent-client.is-last-node.result\",\"log_level\":1,\"data\":{\"actual_members_count\":3,\"expected_members_count\":3,\"is_last_node\":true}}\r\n{\"timestamp\":\"1466775061.845248222\",\"source\":\"confab\",\"message\":\"confab.controller.configure-server.verify-synced\",\"log_level\":1,\"data\":{}}\r\n{\"timestamp\":\"1466775061.845463514\",\"source\":\"confab\",\"message\":\"confab.agent-client.verify-synced.stats.request\",\"log_level\":1,\"data\":{}}\r\n{\"timestamp\":\"1466775061.845934629\",\"source\":\"confab\",\"message\":\"confab.agent-client.verify-synced.stats.response\",\"log_level\":1,\"data\":{\"commit_index\":\"362812\",\"last_log_index\":\"362812\"}}\r\n{\"timestamp\":\"1466775061.846092939\",\"source\":\"confab\",\"message\":\"confab.agent-client.verify-synced.synced\",\"log_level\":1,\"data\":{}}\r\n{\"timestamp\":\"1466775061.846226931\",\"source\":\"confab\",\"message\":\"confab.controller.configure-server.set-keys\",\"log_level\":1,\"data\":{\"keys\":[\"--deleted--\"]}}\r\n{\"timestamp\":\"1466775061.846373320\",\"source\":\"confab\",\"message\":\"confab.agent-client.set-keys.list-keys.request\",\"log_level\":1,\"data\":{}}\r\n    2016/06/24 13:31:01 [INFO] serf: Received list-keys query\r\n    2016/06/24 13:31:01 [INFO] serf: Received list-keys query\r\n{\"timestamp\":\"1466775066.457789183\",\"source\":\"confab\",\"message\":\"confab.agent-client.set-keys.list-keys.response\",\"log_level\":1,\"data\":{\"keys\":[\"--deleted--\"]}}\r\n{\"timestamp\":\"1466775066.457836866\",\"source\":\"confab\",\"message\":\"confab.agent-client.set-keys.install-key.request\",\"log_level\":1,\"data\":{\"key\":\"--deleted--\"}}\r\n    2016/06/24 13:31:06 [INFO] serf: Received install-key query\r\n    2016/06/24 13:31:06 [INFO] serf: Received install-key query\r\n{\"timestamp\":\"1466775070.503915071\",\"source\":\"confab\",\"message\":\"confab.agent-client.set-keys.install-key.request.failed\",\"log_level\":2,\"data\":{\"error\":\"1/108 nodes reported failure\",\"key\":\"--deleted--\"}}\r\n{\"timestamp\":\"1466775070.503953934\",\"source\":\"confab\",\"message\":\"confab.controller.configure-server.set-keys.failed\",\"log_level\":2,\"data\":{\"error\":\"1/108 nodes reported failure\",\"keys\":[\"--deleted--\"]}}\r\n==> /var/vcap/sys/log/consul_agent/consul_agent.stderr.log <==\r\nerror during start: 1/108 nodes reported failure\r\n==> /var/vcap/sys/log/consul_agent/consul_agent.stdout.log <==\r\n    2016/06/24 13:31:10 [INFO] agent.rpc: Accepted client: 127.0.0.1:37087\r\n{\"timestamp\":\"1466775070.505033970\",\"source\":\"confab\",\"message\":\"confab.controller.stop-agent.leave\",\"log_level\":1,\"data\":{}}\r\n{\"timestamp\":\"1466775070.505050182\",\"source\":\"confab\",\"message\":\"confab.agent-client.leave.leave.request\",\"log_level\":1,\"data\":{}}\r\n    2016/06/24 13:31:10 [INFO] agent.rpc: Graceful leave triggered\r\n    2016/06/24 13:31:10 [INFO] consul: server starting leave\r\n    2016/06/24 13:31:10 [INFO] serf: EventMemberLeave: etcd-pz2-0.dc1 --deleted--\r\n    2016/06/24 13:31:10 [INFO] consul: removing server etcd-pz2-0.dc1 (Addr: --deleted--:8300) (DC: dc1)\r\n    2016/06/24 13:31:11 [INFO] serf: EventMemberLeave: etcd-pz2-0 10.61.137.53\r\n    2016/06/24 13:31:11 [INFO] consul: removing server etcd-pz2-0 (Addr: --deleted--:8300) (DC: dc1)\r\n    2016/06/24 13:31:11 [INFO] raft: Removed ourself, transitioning to follower\r\n{\"timestamp\":\"1466775072.029870272\",\"source\":\"confab\",\"message\":\"confab.agent-client.leave.leave.response\",\"log_level\":1,\"data\":{}}\r\n{\"timestamp\":\"1466775072.029910088\",\"source\":\"confab\",\"message\":\"confab.controller.stop-agent.wait\",\"log_level\":1,\"data\":{}}\r\n{\"timestamp\":\"1466775072.029917955\",\"source\":\"confab\",\"message\":\"confab.agent-runner.wait.get-process\",\"log_level\":1,\"data\":{}}\r\n{\"timestamp\":\"1466775072.029926062\",\"source\":\"confab\",\"message\":\"confab.agent-runner.wait.get-process.result\",\"log_level\":1,\"data\":{\"pid\":5852}}\r\n{\"timestamp\":\"1466775072.029937506\",\"source\":\"confab\",\"message\":\"confab.agent-runner.wait.signal\",\"log_level\":1,\"data\":{\"pid\":5852}}\r\n    2016/06/24 13:31:12 [INFO] agent: requesting shutdown\r\n    2016/06/24 13:31:12 [INFO] consul: shutting down server\r\n    2016/06/24 13:31:12 [INFO] agent: shutdown complete\r\n```"},"comment":{"url":"https://api.github.com/repos/cloudfoundry-incubator/consul-release/issues/comments/230075887","html_url":"https://github.com/cloudfoundry-incubator/consul-release/issues/27#issuecomment-230075887","issue_url":"https://api.github.com/repos/cloudfoundry-incubator/consul-release/issues/27","id":230075887,"user":{"login":"christianang","id":539826,"avatar_url":"https://avatars.githubusercontent.com/u/539826?v=3","gravatar_id":"","url":"https://api.github.com/users/christianang","html_url":"https://github.com/christianang","followers_url":"https://api.github.com/users/christianang/followers","following_url":"https://api.github.com/users/christianang/following{/other_user}","gists_url":"https://api.github.com/users/christianang/gists{/gist_id}","starred_url":"https://api.github.com/users/christianang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/christianang/subscriptions","organizations_url":"https://api.github.com/users/christianang/orgs","repos_url":"https://api.github.com/users/christianang/repos","events_url":"https://api.github.com/users/christianang/events{/privacy}","received_events_url":"https://api.github.com/users/christianang/received_events","type":"User","site_admin":false},"created_at":"2016-07-02T01:17:54Z","updated_at":"2016-07-02T01:17:54Z","body":"@youngm:\r\n\r\nYour consul deployment should fail if any clients return an error because this means something is wrong with your deployment and we don't want to ignore that. In your case, if we don't report that a client returned an error it could be difficult to later realize why your deployment is not working properly. It might also partition your client from the rest of the cluster, which could cause problems.\r\n\r\nChristian Ang @kkallday  @valeriap "}}
{"actor":{"display_login":"christianang"}}