{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/8993","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/8993/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/8993/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/8993/events","html_url":"https://github.com/cockroachdb/cockroach/issues/8993","id":174350545,"number":8993,"title":"stability: slow startup times because of eager replica GC","user":{"login":"tschottdorf","id":5076964,"avatar_url":"https://avatars.githubusercontent.com/u/5076964?v=3","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/cockroachdb/cockroach/labels/stability","name":"stability","color":"b60205"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-08-31T18:56:30Z","updated_at":"2016-09-01T14:41:49Z","closed_at":null,"body":"Forked from #8694, where this was initially discovered and discussed as it exposed GRPC problems which lead to persistent RPC connection failures.\r\n\r\n![screen shot 2016-08-31 at 2 52 42 pm](https://cloud.githubusercontent.com/assets/5076964/18142251/af673c2a-6f8a-11e6-9d7d-d634a029bccc.png)\r\n\r\nI believe that the large number of replicas here suggests that the gc queue isn't properly doing its job, though that is only a theory. If substantiated, we should be spending less time here in regular cases.\r\n\r\nOne way to make this iteration faster is parallelism (i.e. destroy multiple replicas in parallel, to keep RocksDB busy). We must, however, finish it before continuing with the startup process (to avoid racing with the Store, which might access overlapping keyspace)."},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/244100620","html_url":"https://github.com/cockroachdb/cockroach/issues/8993#issuecomment-244100620","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/8993","id":244100620,"user":{"login":"tschottdorf","id":5076964,"avatar_url":"https://avatars.githubusercontent.com/u/5076964?v=3","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"created_at":"2016-09-01T14:41:49Z","updated_at":"2016-09-01T14:41:49Z","body":"The remaining part here is keeping an eye out for the GC queue getting stuck (or whatever other reason there is for having that many replicas gc'able at boot time)."}}
{"actor":{"display_login":"tschottdorf"}}