{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/28132","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/28132/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/28132/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/28132/events","html_url":"https://github.com/cockroachdb/cockroach/issues/28132","id":346476315,"node_id":"MDU6SXNzdWUzNDY0NzYzMTU=","number":28132,"title":"storage: TestReplicaIDChangePending failed under stress","user":{"login":"cockroach-teamcity","id":20559710,"node_id":"MDQ6VXNlcjIwNTU5NzEw","avatar_url":"https://avatars0.githubusercontent.com/u/20559710?v=4","gravatar_id":"","url":"https://api.github.com/users/cockroach-teamcity","html_url":"https://github.com/cockroach-teamcity","followers_url":"https://api.github.com/users/cockroach-teamcity/followers","following_url":"https://api.github.com/users/cockroach-teamcity/following{/other_user}","gists_url":"https://api.github.com/users/cockroach-teamcity/gists{/gist_id}","starred_url":"https://api.github.com/users/cockroach-teamcity/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cockroach-teamcity/subscriptions","organizations_url":"https://api.github.com/users/cockroach-teamcity/orgs","repos_url":"https://api.github.com/users/cockroach-teamcity/repos","events_url":"https://api.github.com/users/cockroach-teamcity/events{/privacy}","received_events_url":"https://api.github.com/users/cockroach-teamcity/received_events","type":"User","site_admin":false},"labels":[{"id":912737912,"node_id":"MDU6TGFiZWw5MTI3Mzc5MTI=","url":"https://api.github.com/repos/cockroachdb/cockroach/labels/A-core-kv","name":"A-core-kv","color":"c2e0c6","default":false},{"id":191587409,"node_id":"MDU6TGFiZWwxOTE1ODc0MDk=","url":"https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure","name":"C-test-failure","color":"c5def5","default":false},{"id":335217860,"node_id":"MDU6TGFiZWwzMzUyMTc4NjA=","url":"https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot","name":"O-robot","color":"666ad1","default":false}],"state":"closed","locked":false,"assignee":{"login":"tschottdorf","id":5076964,"node_id":"MDQ6VXNlcjUwNzY5NjQ=","avatar_url":"https://avatars1.githubusercontent.com/u/5076964?v=4","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"assignees":[{"login":"tschottdorf","id":5076964,"node_id":"MDQ6VXNlcjUwNzY5NjQ=","avatar_url":"https://avatars1.githubusercontent.com/u/5076964?v=4","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/cockroachdb/cockroach/milestones/13","html_url":"https://github.com/cockroachdb/cockroach/milestone/13","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/milestones/13/labels","id":2745639,"node_id":"MDk6TWlsZXN0b25lMjc0NTYzOQ==","number":13,"title":"2.1","description":"","creator":{"login":"knz","id":642886,"node_id":"MDQ6VXNlcjY0Mjg4Ng==","avatar_url":"https://avatars3.githubusercontent.com/u/642886?v=4","gravatar_id":"","url":"https://api.github.com/users/knz","html_url":"https://github.com/knz","followers_url":"https://api.github.com/users/knz/followers","following_url":"https://api.github.com/users/knz/following{/other_user}","gists_url":"https://api.github.com/users/knz/gists{/gist_id}","starred_url":"https://api.github.com/users/knz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/knz/subscriptions","organizations_url":"https://api.github.com/users/knz/orgs","repos_url":"https://api.github.com/users/knz/repos","events_url":"https://api.github.com/users/knz/events{/privacy}","received_events_url":"https://api.github.com/users/knz/received_events","type":"User","site_admin":false},"open_issues":479,"closed_issues":1628,"state":"open","created_at":"2017-09-05T15:30:49Z","updated_at":"2018-09-20T10:08:44Z","due_on":null,"closed_at":null},"comments":13,"created_at":"2018-08-01T07:15:14Z","updated_at":"2018-09-20T10:08:43Z","closed_at":"2018-09-20T10:08:43Z","author_association":"MEMBER","body":"SHA: https://github.com/cockroachdb/cockroach/commits/cbe3d10fd28f7a1a1025028b9f840e2f6e437133\n\nParameters:\n```\nTAGS=\nGOFLAGS=-race\n```\n\nFailed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=809459&tab=buildLog\n\n```\n\nI180801 06:46:12.236226 37808 storage/store.go:1496  [s1] [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available\npanic: test timed out after 30m0s\n\ngoroutine 37950 [running]:\ntesting.(*M).startAlarm.func1()\n\t/usr/local/go/src/testing/testing.go:1240 +0x146\ncreated by time.goFunc\n\t/usr/local/go/src/time/sleep.go:172 +0x52\n\ngoroutine 1 [chan receive, 28 minutes]:\ntesting.(*T).Run(0xc42024d4a0, 0x31aebd6, 0x1a, 0x32c4ac8, 0xc4205b5901)\n\t/usr/local/go/src/testing/testing.go:825 +0x597\ntesting.runTests.func1(0xc42024d4a0)\n\t/usr/local/go/src/testing/testing.go:1063 +0xa5\ntesting.tRunner(0xc42024d4a0, 0xc4205b5b20)\n\t/usr/local/go/src/testing/testing.go:777 +0x16e\ntesting.runTests(0xc4202c4140, 0x498efc0, 0x204, 0x204, 0x4a)\n\t/usr/local/go/src/testing/testing.go:1061 +0x4e2\ntesting.(*M).Run(0xc42017e200, 0x0)\n\t/usr/local/go/src/testing/testing.go:978 +0x2ce\ngithub.com/cockroachdb/cockroach/pkg/storage_test.TestMain(0xc42017e200)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/main_test.go:57 +0x2a9\nmain.main()\n\t_testmain.go:1088 +0x22b\n\ngoroutine 20 [syscall, 30 minutes]:\nos/signal.signal_recv(0x896011)\n\t/usr/local/go/src/runtime/sigqueue.go:139 +0xa6\nos/signal.loop()\n\t/usr/local/go/src/os/signal/signal_unix.go:22 +0x30\ncreated by os/signal.init.0\n\t/usr/local/go/src/os/signal/signal_unix.go:28 +0x4f\n\ngoroutine 8 [chan receive]:\ngithub.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1171 +0x132\ncreated by github.com/cockroachdb/cockroach/pkg/util/log.init.0\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:590 +0x189\n\ngoroutine 9 [chan receive, 30 minutes]:\ngithub.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:597 +0xf2\ncreated by github.com/cockroachdb/cockroach/pkg/util/log.init.0\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:591 +0x1a1\n\ngoroutine 25 [select, 30 minutes, locked to thread]:\nruntime.gopark(0x32c8000, 0x0, 0x318a467, 0x6, 0x18, 0x1)\n\t/usr/local/go/src/runtime/proc.go:291 +0xf9\nruntime.selectgo(0xc42008ff50, 0xc4204d4300)\n\t/usr/local/go/src/runtime/select.go:392 +0x11d4\nruntime.ensureSigM.func1()\n\t/usr/local/go/src/runtime/signal_unix.go:549 +0x19f\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1\n\ngoroutine 37921 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x2f)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc421da0570)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc421da0570)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4214028e0, 0xc420bb47e0, 0xc4214028d0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37922 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x30)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc420562cc0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc420562cc0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402900, 0xc420bb47e0, 0xc4214028f0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37889 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x12)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc4205627e0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc4205627e0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402480, 0xc420bb47e0, 0xc421402470)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37935 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x3d)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc421da07b0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc421da07b0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402aa0, 0xc420bb47e0, 0xc421402a90)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37942 [semacquire, 28 minutes]:\nsync.runtime_Semacquire(0xc4200f0b84)\n\t/usr/local/go/src/runtime/sema.go:56 +0x39\nsync.(*RWMutex).RLock(0xc4200f0b78)\n\t/usr/local/go/src/sync/rwmutex.go:50 +0x95\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Replica).Desc(0xc4200f0a80, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1560 +0x54\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Replica).endKey(0xc4200f0a80, 0x3, 0xc420327900, 0xc420327900)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:6477 +0x3d\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Replica).Less(0xc4200f0a80, 0x397fc20, 0xc420371100, 0x18)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:6482 +0x3d\ngithub.com/cockroachdb/cockroach/vendor/github.com/google/btree.(*node).iterate(0xc42005dc00, 0x1, 0x397fc20, 0xc420371100, 0x0, 0x0, 0x2ee0001, 0xc420ce7bd8, 0xc420371100)\n\t/go/src/github.com/cockroachdb/cockroach/vendor/github.com/google/btree/btree.go:507 +0x4df\ngithub.com/cockroachdb/cockroach/vendor/github.com/google/btree.(*BTree).AscendGreaterOrEqual(0xc420370aa0, 0x397fc20, 0xc420371100, 0xc420ce7bd8)\n\t/go/src/github.com/cockroachdb/cockroach/vendor/github.com/google/btree/btree.go:759 +0xaa\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).visitReplicasLocked(0xc420327900, 0x49945e0, 0x0, 0x0, 0x4994616, 0x2, 0x2, 0xc420ce7cb8)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:2049 +0x1d7\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).LookupReplica(0xc420327900, 0x49945e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:2007 +0x16c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x39a6160, 0xc420ba0c90)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1589 +0x157\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402b70, 0xc420bb47e0, 0xc420c16c80)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37911 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x25)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc421da0270)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc421da0270)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4214027a0, 0xc420bb47e0, 0xc421402790)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37884 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0xa)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc4205624e0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc4205624e0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4214023e0, 0xc420bb47e0, 0xc4214023d0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37902 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x1c)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc4205629c0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc4205629c0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402660, 0xc420bb47e0, 0xc421402650)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37909 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x23)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc421da0180)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc421da0180)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402760, 0xc420bb47e0, 0xc421402750)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37664 [select, 28 minutes]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x39a6160, 0xc4205630b0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:586 +0x1c8\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421afe040, 0xc420bb47e0, 0xc4202df900)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37906 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x20)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc420562a20)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc420562a20)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4214026e0, 0xc420bb47e0, 0xc4214026d0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37903 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x1d)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc42051f2f0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc42051f2f0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402680, 0xc420bb47e0, 0xc421402670)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37886 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0xc)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc420562600)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc420562600)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402420, 0xc420bb47e0, 0xc421402410)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37913 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x27)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc421da0300)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc421da0300)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4214027e0, 0xc420bb47e0, 0xc4214027d0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37496 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc421416470, 0x4)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc421416460)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc421416340)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:655 +0xb3\ncreated by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:642 +0x607\n\ngoroutine 37937 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x3f)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc421da0810)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc421da0810)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402ae0, 0xc420bb47e0, 0xc421402ad0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37919 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x2d)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc421da0540)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc421da0540)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4214028a0, 0xc420bb47e0, 0xc421402890)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37895 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x17)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc4205628d0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc4205628d0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402540, 0xc420bb47e0, 0xc421402530)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37660 [select, 28 minutes]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x39a6160, 0xc420562fc0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:586 +0x1c8\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421afe000, 0xc420bb47e0, 0xc4202df6c0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37916 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x2a)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc420562ba0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc420562ba0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402840, 0xc420bb47e0, 0xc421402830)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37917 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x2b)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc421da0360)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc421da0360)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402860, 0xc420bb47e0, 0xc421402850)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37892 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x13)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc420562810)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc420562810)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4214024e0, 0xc420bb47e0, 0xc4214024d0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37927 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x35)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc421da06c0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc421da06c0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4214029a0, 0xc420bb47e0, 0xc421402990)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37901 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x19)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc420562930)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc420562930)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402640, 0xc420bb47e0, 0xc421402630)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37885 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0xb)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc420562510)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc420562510)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402400, 0xc420bb47e0, 0xc4214023f0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37933 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x3b)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc421da0750)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc421da0750)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402a60, 0xc420bb47e0, 0xc421402a50)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37915 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x29)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc421da0330)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc421da0330)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402820, 0xc420bb47e0, 0xc421402810)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37930 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x38)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc420562d80)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc420562d80)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402a00, 0xc420bb47e0, 0xc4214029f0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37497 [semacquire, 28 minutes]:\nsync.runtime_Semacquire(0xc4200f0b84)\n\t/usr/local/go/src/runtime/sema.go:56 +0x39\nsync.(*RWMutex).RLock(0xc4200f0b78)\n\t/usr/local/go/src/sync/rwmutex.go:50 +0x95\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Replica).Desc(0xc4200f0a80, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1560 +0x54\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Replica).endKey(0xc4200f0a80, 0x845cc9, 0x2b1d151faba8, 0xc4203bc000)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:6477 +0x3d\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Replica).Less(0xc4200f0a80, 0x397fc20, 0xc4203bc000, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:6482 +0x3d\ngithub.com/cockroachdb/cockroach/vendor/github.com/google/btree.(*node).iterate(0xc42005dc00, 0x1, 0x397fc20, 0xc4203bc000, 0x0, 0x0, 0x2ee0001, 0xc420ce5bd8, 0xc4203bc000)\n\t/go/src/github.com/cockroachdb/cockroach/vendor/github.com/google/btree/btree.go:507 +0x4df\ngithub.com/cockroachdb/cockroach/vendor/github.com/google/btree.(*BTree).AscendGreaterOrEqual(0xc420370aa0, 0x397fc20, 0xc4203bc000, 0xc420ce5bd8)\n\t/go/src/github.com/cockroachdb/cockroach/vendor/github.com/google/btree/btree.go:759 +0xaa\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).visitReplicasLocked(0xc420327900, 0xc4202ead90, 0x1, 0x8, 0x4994616, 0x2, 0x2, 0xc420ce5cb8)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:2049 +0x1d7\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).LookupReplica(0xc420327900, 0xc4202ead90, 0x1, 0x8, 0x0, 0x0, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:2007 +0x16c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x39a6160, 0xc420562f30)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1589 +0x157\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4208d6580, 0xc420bb47e0, 0xc421cfcc30)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37938 [semacquire, 28 minutes]:\nsync.runtime_Semacquire(0xc420bb4aa4)\n\t/usr/local/go/src/runtime/sema.go:56 +0x39\nsync.(*WaitGroup).Wait(0xc420bb4a98)\n\t/usr/local/go/src/sync/waitgroup.go:129 +0xb3\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait(0xc420bb4a20, 0x39a6160, 0xc421da0780)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:172 +0x3f\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait-fm(0x39a6160, 0xc421da0780)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3830 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402b00, 0xc420bb47e0, 0xc421402af0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37808 [chan send, 28 minutes]:\ngithub.com/cockroachdb/cockroach/pkg/storage.TestReplicaIDChangePending.func2(0xc421470540, 0xc421c90100, 0x31cb7e8)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_test.go:7804 +0xf8\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Replica).submitProposalLocked(0xc4200f0a80, 0xc421470540, 0x31cb7e8, 0x24)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:3603 +0xb9\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Replica).refreshProposalsLocked(0xc4200f0a80, 0x0, 0x4)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:4720 +0xf05\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Replica).setReplicaIDRaftMuLockedMuLocked(0xc4200f0a80, 0x2, 0xc4200f0b78, 0xd1380a)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1001 +0x4b0\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Replica).setReplicaID(0xc4200f0a80, 0x2, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:916 +0xd3\ngithub.com/cockroachdb/cockroach/pkg/storage.TestReplicaIDChangePending(0xc42024d590)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_test.go:7811 +0x9bd\ntesting.tRunner(0xc42024d590, 0x32c4ac8)\n\t/usr/local/go/src/testing/testing.go:777 +0x16e\ncreated by testing.(*T).Run\n\t/usr/local/go/src/testing/testing.go:824 +0x565\n\ngoroutine 37891 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x11)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc4205627b0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc4205627b0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4214024c0, 0xc420bb47e0, 0xc4214024b0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37883 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0xd)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc420562630)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc420562630)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4214023c0, 0xc420bb47e0, 0xc4214023b0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37908 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x22)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc420562a50)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc420562a50)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402740, 0xc420bb47e0, 0xc421402730)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37907 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x21)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc421da0150)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc421da0150)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402720, 0xc420bb47e0, 0xc4214026f0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37955 [select, 28 minutes]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x39a6160, 0xc4205630e0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:586 +0x1c8\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421afe070, 0xc420bb47e0, 0xc4202dfa00)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37934 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x3c)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc420562de0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc420562de0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402a80, 0xc420bb47e0, 0xc421402a70)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37663 [select, 28 minutes]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x39a6160, 0xc420563050)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:586 +0x1c8\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421afe030, 0xc420bb47e0, 0xc4202df8a0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37914 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x28)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc420562b70)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc420562b70)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402800, 0xc420bb47e0, 0xc4214027f0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37662 [select, 28 minutes]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x39a6160, 0xc420563020)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:586 +0x1c8\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421afe020, 0xc420bb47e0, 0xc4202df860)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37924 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x32)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc420562cf0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc420562cf0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402940, 0xc420bb47e0, 0xc421402930)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37905 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x1f)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc421da0000)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc421da0000)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4214026c0, 0xc420bb47e0, 0xc4214026b0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37945 [semacquire, 28 minutes]:\nsync.runtime_Semacquire(0xc4200f0b84)\n\t/usr/local/go/src/runtime/sema.go:56 +0x39\nsync.(*RWMutex).RLock(0xc4200f0b78)\n\t/usr/local/go/src/sync/rwmutex.go:50 +0x95\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Replica).leaseGoodToGo(0xc4200f0a80, 0x39a6160, 0xc420a702d0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1305 +0x104\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Replica).redirectOnOrAcquireLease(0xc4200f0a80, 0x39a6160, 0xc420a702d0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1340 +0xd6\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).startLeaseRenewer.func1(0x39a6160, 0xc420ba19b0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1640 +0x215\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402c20, 0xc420bb47e0, 0xc421402c10)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37498 [semacquire, 28 minutes]:\nsync.runtime_Semacquire(0xc4200f0b84)\n\t/usr/local/go/src/runtime/sema.go:56 +0x39\nsync.(*RWMutex).RLock(0xc4200f0b78)\n\t/usr/local/go/src/sync/rwmutex.go:50 +0x95\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Replica).Desc(0xc4200f0a80, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1560 +0x54\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Replica).endKey(0xc4200f0a80, 0x3, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:6477 +0x3d\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Replica).Less(0xc4200f0a80, 0x397fc20, 0xc420371120, 0x18)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:6482 +0x3d\ngithub.com/cockroachdb/cockroach/vendor/github.com/google/btree.(*node).iterate(0xc42005dc00, 0x1, 0x397fc20, 0xc420371120, 0x0, 0x0, 0x2ee0001, 0xc420ce3bd8, 0xc420371120)\n\t/go/src/github.com/cockroachdb/cockroach/vendor/github.com/google/btree/btree.go:507 +0x4df\ngithub.com/cockroachdb/cockroach/vendor/github.com/google/btree.(*BTree).AscendGreaterOrEqual(0xc420370aa0, 0x397fc20, 0xc420371120, 0xc420ce3bd8)\n\t/go/src/github.com/cockroachdb/cockroach/vendor/github.com/google/btree/btree.go:759 +0xaa\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).visitReplicasLocked(0xc420327900, 0xc4202eac60, 0xb, 0xb, 0x4994616, 0x2, 0x2, 0xc420ce3cb8)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:2049 +0x1d7\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).LookupReplica(0xc420327900, 0xc4202eac60, 0xb, 0xb, 0x0, 0x0, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:2007 +0x16c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x39a6160, 0xc421da08d0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1589 +0x157\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4208d6590, 0xc420bb47e0, 0xc421cfcc80)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37920 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x2e)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc420562c00)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc420562c00)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4214028c0, 0xc420bb47e0, 0xc4214028b0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37890 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x10)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc4205626f0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc4205626f0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4214024a0, 0xc420bb47e0, 0xc421402490)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37899 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x1b)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc420562990)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc420562990)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402600, 0xc420bb47e0, 0xc4214025f0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37887 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0xa95d265100000040)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc420562000)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc420562000)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402440, 0xc420bb47e0, 0xc421402430)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37931 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x39)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc421da0720)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc421da0720)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402a20, 0xc420bb47e0, 0xc421402a10)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37925 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x33)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc421da0690)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc421da0690)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402960, 0xc420bb47e0, 0xc421402950)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37900 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x1a)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc420562960)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc420562960)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402620, 0xc420bb47e0, 0xc421402610)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37882 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x9)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc4205624b0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc4205624b0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4214023a0, 0xc420bb47e0, 0xc421402390)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37661 [select, 28 minutes]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x39a6160, 0xc420562ff0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:586 +0x1c8\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421afe010, 0xc420bb47e0, 0xc4202df840)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37877 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x4)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc4205621b0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc4205621b0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402260, 0xc420bb47e0, 0xc421402250)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37874 [semacquire, 28 minutes]:\nsync.runtime_SemacquireMutex(0xc4200f0b14, 0x900000000)\n\t/usr/local/go/src/runtime/sema.go:71 +0x3d\nsync.(*Mutex).Lock(0xc4200f0b10)\n\t/usr/local/go/src/sync/mutex.go:134 +0x172\ngithub.com/cockroachdb/cockroach/pkg/util/syncutil.(*Mutex).Lock(0xc4200f0b10)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/mutex_sync.go:32 +0x3b\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Replica).tick(0xc4200f0a80, 0x0, 0x0, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:4260 +0xfd\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).processTick(0xc420327900, 0x39a6160, 0xc420562060, 0x1, 0xc420a8f080)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3792 +0xef\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc420562060)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:210 +0x396\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc420562060)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4214021a0, 0xc420bb47e0, 0xc421402190)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37939 [select]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).raftTickLoop(0xc420327900, 0x39a6160, 0xc421da0870)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3846 +0x3aa\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).(github.com/cockroachdb/cockroach/pkg/storage.raftTickLoop)-fm(0x39a6160, 0xc421da0870)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3832 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402b20, 0xc420bb47e0, 0xc421402b10)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37918 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x2c)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc420562bd0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc420562bd0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402880, 0xc420bb47e0, 0xc421402870)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37936 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x3e)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc421da07e0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc421da07e0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402ac0, 0xc420bb47e0, 0xc421402ab0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37875 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x2)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc420562090)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc420562090)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4214021c0, 0xc420bb47e0, 0xc4214021b0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37809 [chan receive, 28 minutes]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func1(0x39a6160, 0xc420562030)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:156 +0x63\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402180, 0xc420bb47e0, 0xc4203121a0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37495 [chan receive, 28 minutes]:\ngithub.com/cockroachdb/cockroach/pkg/rpc.NewContext.func1(0x39a6160, 0xc42051e4e0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:329 +0x6c\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4208d60b0, 0xc420bb47e0, 0xc420370420)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37897 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x15)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc420562870)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc420562870)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4214025c0, 0xc420bb47e0, 0xc4214025b0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37878 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x8)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc4205623f0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc4205623f0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402280, 0xc420bb47e0, 0xc421402270)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37910 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x24)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc420562a80)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc420562a80)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402780, 0xc420bb47e0, 0xc421402770)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37954 [select, 28 minutes]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x39a6160, 0xc420563230)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:586 +0x1c8\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421afe060, 0xc420bb47e0, 0xc4202df9c0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37946 [select, 28 minutes]:\ngithub.com/cockroachdb/cockroach/pkg/storage/compactor.(*Compactor).Start.func1.1(0x39a6160, 0xc420ba0e40)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/compactor/compactor.go:127 +0x22d\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402c30, 0xc420bb47e0, 0xc4203132a0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37941 [select, 28 minutes]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).Start.func2(0x39a6160, 0xc421da0840)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1466 +0x149\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402b60, 0xc420bb47e0, 0xc420312420)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0x14b\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xbb\n\ngoroutine 37926 [semacquire, 28 minutes]:\nsync.runtime_notifyListWait(0xc420ad5110, 0x34)\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\nsync.(*Cond).Wait(0xc420ad5100)\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb4a20, 0x39a6160, 0xc420562d20)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x39a6160, 0xc420562d20)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421402980, 0xc420bb47e0, 0xc421402970)\n```"},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/423125364","html_url":"https://github.com/cockroachdb/cockroach/issues/28132#issuecomment-423125364","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/28132","id":423125364,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMzEyNTM2NA==","user":{"login":"tschottdorf","id":5076964,"node_id":"MDQ6VXNlcjUwNzY5NjQ=","avatar_url":"https://avatars1.githubusercontent.com/u/5076964?v=4","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"created_at":"2018-09-20T10:08:43Z","updated_at":"2018-09-20T10:08:43Z","author_association":"MEMBER","body":"Ah, we started passing `-p 4` to `stress`:\r\n\r\n```\r\n2018/09/08 05:06:50 created teamcity build (buildID=Cockroach_Nightlies_Stress, branch=master, opts=map[env.GOFLAGS:-race env.STRESSFLAGS:-p 4 env.PKG:github.com/cockroachdb/cockroach/pkg/storage]): Build 892843, 1 state=queued\r\n```\r\n\r\nThe double reproposal probably requires really slamming the machine and now we're not doing that any more.\r\n\r\nOk, I'm satisfied. Closing in anticipation of #30452."}}
{"actor":{"display_login":"tschottdorf"}}