{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9372","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9372/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9372/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9372/events","html_url":"https://github.com/cockroachdb/cockroach/issues/9372","id":176984872,"number":9372,"title":"storage: quiescence leads to more elections","user":{"login":"petermattis","id":6201335,"avatar_url":"https://avatars.githubusercontent.com/u/6201335?v=3","gravatar_id":"","url":"https://api.github.com/users/petermattis","html_url":"https://github.com/petermattis","followers_url":"https://api.github.com/users/petermattis/followers","following_url":"https://api.github.com/users/petermattis/following{/other_user}","gists_url":"https://api.github.com/users/petermattis/gists{/gist_id}","starred_url":"https://api.github.com/users/petermattis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/petermattis/subscriptions","organizations_url":"https://api.github.com/users/petermattis/orgs","repos_url":"https://api.github.com/users/petermattis/repos","events_url":"https://api.github.com/users/petermattis/events{/privacy}","received_events_url":"https://api.github.com/users/petermattis/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"petermattis","id":6201335,"avatar_url":"https://avatars.githubusercontent.com/u/6201335?v=3","gravatar_id":"","url":"https://api.github.com/users/petermattis","html_url":"https://github.com/petermattis","followers_url":"https://api.github.com/users/petermattis/followers","following_url":"https://api.github.com/users/petermattis/following{/other_user}","gists_url":"https://api.github.com/users/petermattis/gists{/gist_id}","starred_url":"https://api.github.com/users/petermattis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/petermattis/subscriptions","organizations_url":"https://api.github.com/users/petermattis/orgs","repos_url":"https://api.github.com/users/petermattis/repos","events_url":"https://api.github.com/users/petermattis/events{/privacy}","received_events_url":"https://api.github.com/users/petermattis/received_events","type":"User","site_admin":false},"assignees":[{"login":"petermattis","id":6201335,"avatar_url":"https://avatars.githubusercontent.com/u/6201335?v=3","gravatar_id":"","url":"https://api.github.com/users/petermattis","html_url":"https://github.com/petermattis","followers_url":"https://api.github.com/users/petermattis/followers","following_url":"https://api.github.com/users/petermattis/following{/other_user}","gists_url":"https://api.github.com/users/petermattis/gists{/gist_id}","starred_url":"https://api.github.com/users/petermattis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/petermattis/subscriptions","organizations_url":"https://api.github.com/users/petermattis/orgs","repos_url":"https://api.github.com/users/petermattis/repos","events_url":"https://api.github.com/users/petermattis/events{/privacy}","received_events_url":"https://api.github.com/users/petermattis/received_events","type":"User","site_admin":false}],"milestone":null,"comments":6,"created_at":"2016-09-14T18:32:52Z","updated_at":"2016-09-14T22:00:00Z","closed_at":null,"body":"I'm able to reproduce the additional elections when quiescing is turned on in a local test environment. Within a single scheduler worker iteration we do the following:\r\n\r\n```\r\n\t\t\t// Process the range ID.\r\n\t\t\tif state&stateRaftTick != 0 {\r\n\t\t\t\t// processRaftTick returns true if the range should perform ready\r\n\t\t\t\t// processing. Do not reorder this below the call to processRaftReady.\r\n\t\t\t\tif s.processor.processTick(id) {\r\n\t\t\t\t\tstate |= stateRaftReady\r\n\t\t\t\t}\r\n\t\t\t}\r\n\t\t\tif state&stateRaftRequest != 0 {\r\n\t\t\t\ts.processor.processRequestQueue(id)\r\n\t\t\t}\r\n\t\t\tif state&stateRaftReady != 0 {\r\n\t\t\t\ts.processor.processReady(id)\r\n\t\t\t}\r\n```\r\n\r\nMy untested hypothesis of what is happening:\r\n1. Follower ticks a replica.\r\n2. Follower processes a \"quiesce\" message from the leader.\r\n3. Followers performs raft ready processing (due to the earlier tick) which unquiesces the replica.\r\n\r\nAt this point, the leader is \"quiesced\" but the follower is not. The follower will tick and then perform an election. What I'm not quite clear on is that exact timing for why this causes a disruption of request processing."},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/247169415","html_url":"https://github.com/cockroachdb/cockroach/issues/9372#issuecomment-247169415","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9372","id":247169415,"user":{"login":"tschottdorf","id":5076964,"avatar_url":"https://avatars.githubusercontent.com/u/5076964?v=3","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"created_at":"2016-09-14T22:00:00Z","updated_at":"2016-09-14T22:00:00Z","body":"This \"lease is not expired\" message is interesting. It appears to be turned on when the raft quorumCheck is activated and in effect implements some kind of raft-internal leader lease which was introduced here: https://github.com/coreos/etcd/pull/5451\r\n\r\nIt's annoying that this is getting in our way. Perhaps we shouldn't use checkQuorum, though I'm not really clear on what its role is at the moment. @bdarnell should weigh in.\r\nProbably you can solve the problems without running into this issue anyway, but would be good to understand this better."}}
{"actor":{"display_login":"tschottdorf"}}