{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9736","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9736/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9736/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9736/events","html_url":"https://github.com/cockroachdb/cockroach/issues/9736","id":181006058,"number":9736,"title":"storage: TestRebalance_3To5Small has leaderless ranges","user":{"login":"cuongdo","id":1564582,"avatar_url":"https://avatars.githubusercontent.com/u/1564582?v=3","gravatar_id":"","url":"https://api.github.com/users/cuongdo","html_url":"https://github.com/cuongdo","followers_url":"https://api.github.com/users/cuongdo/followers","following_url":"https://api.github.com/users/cuongdo/following{/other_user}","gists_url":"https://api.github.com/users/cuongdo/gists{/gist_id}","starred_url":"https://api.github.com/users/cuongdo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cuongdo/subscriptions","organizations_url":"https://api.github.com/users/cuongdo/orgs","repos_url":"https://api.github.com/users/cuongdo/repos","events_url":"https://api.github.com/users/cuongdo/events{/privacy}","received_events_url":"https://api.github.com/users/cuongdo/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"cuongdo","id":1564582,"avatar_url":"https://avatars.githubusercontent.com/u/1564582?v=3","gravatar_id":"","url":"https://api.github.com/users/cuongdo","html_url":"https://github.com/cuongdo","followers_url":"https://api.github.com/users/cuongdo/followers","following_url":"https://api.github.com/users/cuongdo/following{/other_user}","gists_url":"https://api.github.com/users/cuongdo/gists{/gist_id}","starred_url":"https://api.github.com/users/cuongdo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cuongdo/subscriptions","organizations_url":"https://api.github.com/users/cuongdo/orgs","repos_url":"https://api.github.com/users/cuongdo/repos","events_url":"https://api.github.com/users/cuongdo/events{/privacy}","received_events_url":"https://api.github.com/users/cuongdo/received_events","type":"User","site_admin":false},"assignees":[{"login":"cuongdo","id":1564582,"avatar_url":"https://avatars.githubusercontent.com/u/1564582?v=3","gravatar_id":"","url":"https://api.github.com/users/cuongdo","html_url":"https://github.com/cuongdo","followers_url":"https://api.github.com/users/cuongdo/followers","following_url":"https://api.github.com/users/cuongdo/following{/other_user}","gists_url":"https://api.github.com/users/cuongdo/gists{/gist_id}","starred_url":"https://api.github.com/users/cuongdo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cuongdo/subscriptions","organizations_url":"https://api.github.com/users/cuongdo/orgs","repos_url":"https://api.github.com/users/cuongdo/repos","events_url":"https://api.github.com/users/cuongdo/events{/privacy}","received_events_url":"https://api.github.com/users/cuongdo/received_events","type":"User","site_admin":false}],"milestone":null,"comments":7,"created_at":"2016-10-04T21:11:41Z","updated_at":"2016-10-04T22:46:09Z","closed_at":null,"body":"repro:\r\n\r\n```\r\nbuild/builder.sh make build && make test \\\r\n  TESTTIMEOUT=48h \\\r\n  PKG=./acceptance \\\r\n  TESTS=Rebalance_3To5Small \\\r\n  TESTFLAGS='-v -remote -key-name google_compute_engine -cwd terraform -tf.keep-cluster=failed -tf.cockroach-binary=../../cockroach'\r\n```\r\n\r\nThe leader-less ranges are evident when visiting `/raft/ranges`. \r\n<img width=\"691\" alt=\"screenshot 2016-10-04 17 09 58\" src=\"https://cloud.githubusercontent.com/assets/1564582/19092598/6a523ca8-8a55-11e6-8623-0d5f34cb8430.png\">\r\n\r\nThis leaderless state persists for >25 minutes, which encompasses two `replicaScanner` cycles. Anywhere from 7-30 ranges (of 262) have no leaders even after rebalancing has finished."},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/251535558","html_url":"https://github.com/cockroachdb/cockroach/issues/9736#issuecomment-251535558","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/9736","id":251535558,"user":{"login":"tschottdorf","id":5076964,"avatar_url":"https://avatars.githubusercontent.com/u/5076964?v=3","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"created_at":"2016-10-04T22:46:09Z","updated_at":"2016-10-04T22:46:09Z","body":"I don't know if I agree with that statement (the lease could expire, which is what I considered above), but you're right that the queue would try to obtain a leader lease (thus unquiescing).\r\n\r\nYou can `curl host:8080/debug/vmodule/replicate_queue=5,queue=5` which should give you all the verbose logging."}}
{"actor":{"display_login":"tschottdorf"}}