{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/25746","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/25746/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/25746/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/25746/events","html_url":"https://github.com/cockroachdb/cockroach/issues/25746","id":324765027,"node_id":"MDU6SXNzdWUzMjQ3NjUwMjc=","number":25746,"title":"teamcity: failed tests on master: testrace/TestStoreMetrics","user":{"login":"cockroach-teamcity","id":20559710,"node_id":"MDQ6VXNlcjIwNTU5NzEw","avatar_url":"https://avatars0.githubusercontent.com/u/20559710?v=4","gravatar_id":"","url":"https://api.github.com/users/cockroach-teamcity","html_url":"https://github.com/cockroach-teamcity","followers_url":"https://api.github.com/users/cockroach-teamcity/followers","following_url":"https://api.github.com/users/cockroach-teamcity/following{/other_user}","gists_url":"https://api.github.com/users/cockroach-teamcity/gists{/gist_id}","starred_url":"https://api.github.com/users/cockroach-teamcity/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cockroach-teamcity/subscriptions","organizations_url":"https://api.github.com/users/cockroach-teamcity/orgs","repos_url":"https://api.github.com/users/cockroach-teamcity/repos","events_url":"https://api.github.com/users/cockroach-teamcity/events{/privacy}","received_events_url":"https://api.github.com/users/cockroach-teamcity/received_events","type":"User","site_admin":false},"labels":[{"id":912737912,"node_id":"MDU6TGFiZWw5MTI3Mzc5MTI=","url":"https://api.github.com/repos/cockroachdb/cockroach/labels/A-core-kv","name":"A-core-kv","color":"c2e0c6","default":false},{"id":191587409,"node_id":"MDU6TGFiZWwxOTE1ODc0MDk=","url":"https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure","name":"C-test-failure","color":"c5def5","default":false},{"id":335217860,"node_id":"MDU6TGFiZWwzMzUyMTc4NjA=","url":"https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot","name":"O-robot","color":"666ad1","default":false}],"state":"open","locked":false,"assignee":{"login":"nvanbenschoten","id":5438456,"node_id":"MDQ6VXNlcjU0Mzg0NTY=","avatar_url":"https://avatars3.githubusercontent.com/u/5438456?v=4","gravatar_id":"","url":"https://api.github.com/users/nvanbenschoten","html_url":"https://github.com/nvanbenschoten","followers_url":"https://api.github.com/users/nvanbenschoten/followers","following_url":"https://api.github.com/users/nvanbenschoten/following{/other_user}","gists_url":"https://api.github.com/users/nvanbenschoten/gists{/gist_id}","starred_url":"https://api.github.com/users/nvanbenschoten/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nvanbenschoten/subscriptions","organizations_url":"https://api.github.com/users/nvanbenschoten/orgs","repos_url":"https://api.github.com/users/nvanbenschoten/repos","events_url":"https://api.github.com/users/nvanbenschoten/events{/privacy}","received_events_url":"https://api.github.com/users/nvanbenschoten/received_events","type":"User","site_admin":false},"assignees":[{"login":"nvanbenschoten","id":5438456,"node_id":"MDQ6VXNlcjU0Mzg0NTY=","avatar_url":"https://avatars3.githubusercontent.com/u/5438456?v=4","gravatar_id":"","url":"https://api.github.com/users/nvanbenschoten","html_url":"https://github.com/nvanbenschoten","followers_url":"https://api.github.com/users/nvanbenschoten/followers","following_url":"https://api.github.com/users/nvanbenschoten/following{/other_user}","gists_url":"https://api.github.com/users/nvanbenschoten/gists{/gist_id}","starred_url":"https://api.github.com/users/nvanbenschoten/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nvanbenschoten/subscriptions","organizations_url":"https://api.github.com/users/nvanbenschoten/orgs","repos_url":"https://api.github.com/users/nvanbenschoten/repos","events_url":"https://api.github.com/users/nvanbenschoten/events{/privacy}","received_events_url":"https://api.github.com/users/nvanbenschoten/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/cockroachdb/cockroach/milestones/13","html_url":"https://github.com/cockroachdb/cockroach/milestone/13","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/milestones/13/labels","id":2745639,"node_id":"MDk6TWlsZXN0b25lMjc0NTYzOQ==","number":13,"title":"2.1","description":"","creator":{"login":"knz","id":642886,"node_id":"MDQ6VXNlcjY0Mjg4Ng==","avatar_url":"https://avatars3.githubusercontent.com/u/642886?v=4","gravatar_id":"","url":"https://api.github.com/users/knz","html_url":"https://github.com/knz","followers_url":"https://api.github.com/users/knz/followers","following_url":"https://api.github.com/users/knz/following{/other_user}","gists_url":"https://api.github.com/users/knz/gists{/gist_id}","starred_url":"https://api.github.com/users/knz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/knz/subscriptions","organizations_url":"https://api.github.com/users/knz/orgs","repos_url":"https://api.github.com/users/knz/repos","events_url":"https://api.github.com/users/knz/events{/privacy}","received_events_url":"https://api.github.com/users/knz/received_events","type":"User","site_admin":false},"open_issues":571,"closed_issues":644,"state":"open","created_at":"2017-09-05T15:30:49Z","updated_at":"2018-06-19T14:59:13Z","due_on":null,"closed_at":null},"comments":1,"created_at":"2018-05-21T01:59:37Z","updated_at":"2018-06-19T14:59:32Z","closed_at":null,"author_association":"MEMBER","body":"The following tests appear to have failed:\n\n[#669819](https://teamcity.cockroachdb.com/viewLog.html?buildId=669819):\n\n\n```\n--- FAIL: testrace/TestStoreMetrics (0.000s)\nTest ended in panic.\n\n------- Stdout: -------\nI180521 01:50:36.478031 50471 storage/engine/rocksdb.go:565  opening rocksdb instance at \"/go/src/github.com/cockroachdb/cockroach/artifacts/TestStoreMetrics602880603\"\r\nI180521 01:50:36.627628 50471 storage/store.go:1443  [s1] [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available\r\nI180521 01:50:36.628445 50471 gossip/gossip.go:333  [n1] NodeDescriptor set to node_id:1 address:<network_field:\"tcp\" address_field:\"127.0.0.1:33309\" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > \r\nI180521 01:50:36.674540 50471 storage/engine/rocksdb.go:565  opening rocksdb instance at \"/go/src/github.com/cockroachdb/cockroach/artifacts/TestStoreMetrics165337598\"\r\nW180521 01:50:36.713142 50471 gossip/gossip.go:1293  [n2] no incoming or outgoing connections\r\nI180521 01:50:36.715565 50736 gossip/client.go:129  [n2] started gossip client to 127.0.0.1:33309\r\nI180521 01:50:36.751855 50471 gossip/gossip.go:333  [n2] NodeDescriptor set to node_id:2 address:<network_field:\"tcp\" address_field:\"127.0.0.1:37741\" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > \r\nI180521 01:50:36.788124 50471 storage/engine/rocksdb.go:565  opening rocksdb instance at \"/go/src/github.com/cockroachdb/cockroach/artifacts/TestStoreMetrics333823301\"\r\nW180521 01:50:36.830639 50471 gossip/gossip.go:1293  [n3] no incoming or outgoing connections\r\nI180521 01:50:36.833257 50862 gossip/client.go:129  [n3] started gossip client to 127.0.0.1:33309\r\nI180521 01:50:36.846315 50471 storage/store.go:1443  [s3] [n3,s3]: failed initial metrics computation: [n3,s3]: system config not yet available\r\nI180521 01:50:36.863341 50471 gossip/gossip.go:333  [n3] NodeDescriptor set to node_id:3 address:<network_field:\"tcp\" address_field:\"127.0.0.1:34281\" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > \r\nI180521 01:50:36.944337 50471 storage/replica_command.go:863  [s1,r1/1:/M{in-ax}] initiating a split of this range at key \"m\" [r2]\r\nE180521 01:50:36.976524 50961 storage/queue.go:778  [replicate,s1,r1/1:{/Min-m}] range requires a replication change, but lacks a quorum of live replicas (0/1)\r\nE180521 01:50:36.977473 50962 storage/queue.go:778  [replicate,s1,r2/1:{m-/Max}] range requires a replication change, but lacks a quorum of live replicas (0/1)\r\nI180521 01:50:37.058657 50748 storage/replica_proposal.go:202  [s1,r2/1:{m-/Max}] new range lease repl=(n1,s1):1 seq=2 start=0.000000123,151 epo=1 pro=0.000000123,168 following repl=(n1,s1):1 seq=1 start=0.000000000,0 exp=0.900000123,5 pro=0.000000123,6\r\nI180521 01:50:37.160383 51239 storage/replica_raftstorage.go:520  [s1,r2/1:{m-/Max}] generated preemptive snapshot 8596775c at index 13\r\nI180521 01:50:37.165838 51239 storage/store_snapshot.go:605  [s1,r2/1:{m-/Max}] streamed snapshot to (n2,s2):?: kv pairs: 40, log entries: 3, rate-limit: 2.0 MiB/sec, 5ms\r\nI180521 01:50:37.171068 50609 storage/replica_raftstorage.go:730  [s2,r2/?:{-}] applying preemptive snapshot at index 13 (id=8596775c, encoded size=7431, 1 rocksdb batches, 3 log entries)\r\nI180521 01:50:37.180853 50609 storage/replica_raftstorage.go:736  [s2,r2/?:{m-/Max}] applied preemptive snapshot in 9ms [clear=1ms batch=0ms entries=2ms commit=6ms]\r\nI180521 01:50:37.185532 51239 storage/replica_command.go:1777  [s1,r2/1:{m-/Max}] change replicas (ADD_REPLICA (n2,s2):2): read existing descriptor r2:{m-/Max} [(n1,s1):1, next=2]\r\nI180521 01:50:37.243442 50999 storage/replica_proposal.go:202  [s1,r2/1:{m-/Max}] new range lease repl=(n1,s1):1 seq=3 start=0.000000123,151 epo=2 pro=0.000000123,313 following repl=(n1,s1):1 seq=2 start=0.000000123,151 epo=1 pro=0.000000123,168\r\nI180521 01:50:37.248141 51288 storage/replica.go:3312  [txn=af500a0c,s1,r2/1:{m-/Max}] proposing ADD_REPLICA((n2,s2):2): updated=[(n1,s1):1 (n2,s2):2] next=3\r\nI180521 01:50:37.263293 51321 storage/replica_raftstorage.go:520  [s1,r2/1:{m-/Max}] generated preemptive snapshot 2df75ffb at index 16\r\nI180521 01:50:37.268138 51321 storage/store_snapshot.go:605  [s1,r2/1:{m-/Max}] streamed snapshot to (n3,s3):?: kv pairs: 42, log entries: 6, rate-limit: 2.0 MiB/sec, 4ms\r\nI180521 01:50:37.270150 51332 storage/replica_raftstorage.go:730  [s3,r2/?:{-}] applying preemptive snapshot at index 16 (id=2df75ffb, encoded size=8468, 1 rocksdb batches, 6 log entries)\r\nI180521 01:50:37.275933 51332 storage/replica_raftstorage.go:736  [s3,r2/?:{m-/Max}] applied preemptive snapshot in 6ms [clear=0ms batch=0ms entries=2ms commit=3ms]\r\nI180521 01:50:37.287451 51321 storage/replica_command.go:1777  [s1,r2/1:{m-/Max}] change replicas (ADD_REPLICA (n3,s3):3): read existing descriptor r2:{m-/Max} [(n1,s1):1, (n2,s2):2, next=3]\r\nI180521 01:50:37.306845 51324 storage/raft_transport.go:459  raft transport stream to node 1 established\r\nI180521 01:50:37.357633 51296 storage/replica.go:3312  [txn=273c3595,s1,r2/1:{m-/Max}] proposing ADD_REPLICA((n3,s3):3): updated=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4\r\nI180521 01:50:37.714080 51485 storage/replica_proposal.go:202  [s3,r2/3:{m-/Max}] new range lease repl=(n3,s3):3 seq=4 start=0.000000123,518 epo=2 pro=0.000000123,519 following repl=(n1,s1):1 seq=3 start=0.000000123,151 epo=2 pro=0.000000123,313\r\nI180521 01:50:37.729748 51200 storage/client_test.go:1237  test clock advanced to: 1.800000125,0\r\nW180521 01:50:37.730274 51445 storage/store.go:1539  [s1,r2/1:{m-/Max}] could not gossip system config: [NotLeaseHolderError] r2: replica (n1,s1):1 not lease holder; lease holder unknown\r\nI180521 01:50:37.767707 51497 storage/replica_proposal.go:202  [s2,r2/2:{m-/Max}] new range lease repl=(n2,s2):2 seq=5 start=1.800000125,7 epo=3 pro=1.800000125,8 following repl=(n3,s3):3 seq=4 start=0.000000123,518 epo=2 pro=0.000000123,519\r\nE180521 01:50:37.793316 51654 storage/replica_range_lease.go:292  [s1,r2/1:{m-/Max}] not incrementing epoch on n2 because next leaseholder (n1) not live (err = <nil>)\r\nW180521 01:50:37.814202 51645 storage/store.go:1539  [s3,r2/3:{m-/Max}] could not gossip system config: [NotLeaseHolderError] r2: replica (n3,s3):3 not lease holder; lease holder unknown\r\nI180521 01:50:37.842332 51502 storage/replica_proposal.go:202  [s2,r2/2:{m-/Max}] new range lease repl=(n2,s2):2 seq=5 start=1.800000125,7 epo=3 pro=1.800000125,32 following repl=(n2,s2):2 seq=5 start=1.800000125,7 epo=3 pro=1.800000125,8\r\nI180521 01:50:38.064642 51822 storage/replica_proposal.go:202  [s2,r2/2:{m-/Max}] new range lease repl=(n2,s2):2 seq=6 start=1.800000125,204 epo=3 pro=1.800000125,212 following repl=(n2,s2):2 seq=5 start=1.800000125,7 epo=3 pro=1.800000125,32\r\nI180521 01:50:38.109477 51999 util/stop/stopper.go:471  quiescing; tasks left:\r\n1      [async] storage.multiTestContextKVTransport: calling next replica\r\nW180521 01:50:38.114677 51887 storage/node_liveness.go:465  [hb] failed node liveness heartbeat: node unavailable; try another peer\r\npanic: first Range is not present on any store in the multiTestContext.\r\n\r\ngoroutine 51911 [running]:\r\ngithub.com/cockroachdb/cockroach/pkg/storage_test.(*multiTestContext).FirstRange(0xc4201043c0, 0x0, 0x0, 0x0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_test.go:612 +0x339\r\ngithub.com/cockroachdb/cockroach/pkg/kv.(*RangeDescriptorCache).performRangeLookup(0xc420960180, 0x38c8140, 0xc421692b40, 0xc420468da0, 0xf, 0xf, 0x38f4300, 0xc420b48380, 0x0, 0x6, ...)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/range_cache.go:427 +0x218\r\ngithub.com/cockroachdb/cockroach/pkg/kv.(*RangeDescriptorCache).lookupRangeDescriptorInternal.func3(0x0, 0x0, 0x0, 0x0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/range_cache.go:310 +0x168\r\ngithub.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).doCall(0xc4209601c0, 0xc420e407d0, 0xc4200f7800, 0x15, 0xc42072ff40)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:118 +0x4d\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).DoChan\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:111 +0x4d1\r\n\r\ngoroutine 1 [chan receive]:\r\ntesting.(*T).Run(0xc420244960, 0x32cdc27, 0x10, 0x33ef798, 0xc4205af901)\r\n\t/usr/local/go/src/testing/testing.go:825 +0x597\r\ntesting.runTests.func1(0xc420244960)\r\n\t/usr/local/go/src/testing/testing.go:1063 +0xa5\r\ntesting.tRunner(0xc420244960, 0xc4205afb20)\r\n\t/usr/local/go/src/testing/testing.go:777 +0x16e\r\ntesting.runTests(0xc420383720, 0x496b560, 0x1f1, 0x1f1, 0xc42036c098)\r\n\t/usr/local/go/src/testing/testing.go:1061 +0x4e2\r\ntesting.(*M).Run(0xc420562100, 0x0)\r\n\t/usr/local/go/src/testing/testing.go:978 +0x2ce\r\ngithub.com/cockroachdb/cockroach/pkg/storage_test.TestMain(0xc420562100)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/main_test.go:57 +0x2a9\r\nmain.main()\r\n\t_testmain.go:1050 +0x22b\r\n\r\ngoroutine 20 [syscall, 1 minutes]:\r\nos/signal.signal_recv(0xa703c1)\r\n\t/usr/local/go/src/runtime/sigqueue.go:139 +0xa6\r\nos/signal.loop()\r\n\t/usr/local/go/src/os/signal/signal_unix.go:22 +0x30\r\ncreated by os/signal.init.0\r\n\t/usr/local/go/src/os/signal/signal_unix.go:28 +0x4f\r\n\r\ngoroutine 4 [chan receive]:\r\ngithub.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1158 +0x132\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/log.init.0\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:591 +0x1b5\r\n\r\ngoroutine 5 [chan receive, 1 minutes]:\r\ngithub.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:598 +0xf2\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/log.init.0\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:592 +0x1cd\r\n\r\ngoroutine 6 [select, 1 minutes, locked to thread]:\r\nruntime.gopark(0x33f1ab8, 0x0, 0x32be23f, 0x6, 0x18, 0x1)\r\n\t/usr/local/go/src/runtime/proc.go:291 +0xf9\r\nruntime.selectgo(0xc420489f50, 0xc4202edaa0)\r\n\t/usr/local/go/src/runtime/select.go:392 +0x11d4\r\nruntime.ensureSigM.func1()\r\n\t/usr/local/go/src/runtime/signal_unix.go:549 +0x19f\r\nruntime.goexit()\r\n\t/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1\r\n\r\ngoroutine 51898 [runnable]:\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x38c8140, 0xc4209c1320)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:586 +0x1c8\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42040a380, 0xc420bb2240, 0xc4203b62c0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 50512 [semacquire]:\r\nsync.runtime_notifyListWait(0xc420a577e8, 0x24)\r\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\r\nsync.(*Cond).Wait(0xc420a577d8)\r\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\r\ngithub.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc420a576c0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:647 +0xb3\r\ncreated by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:634 +0x607\r\n\r\ngoroutine 50853 [select]:\r\ngithub.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc4215050c0)\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:122 +0x1d4\r\ncreated by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:113 +0x291\r\n\r\ngoroutine 51973 [runnable]:\r\nsync.runtime_notifyListWait(0xc42072fa10, 0x33)\r\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\r\nsync.(*Cond).Wait(0xc42072fa00)\r\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb2750, 0x38c8140, 0xc4209c0f30)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38c8140, 0xc4209c0f30)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42154c130, 0xc420bb2240, 0xc42154c110)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 51958 [runnable]:\r\nsync.runtime_notifyListWait(0xc42072fa10, 0x24)\r\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\r\nsync.(*Cond).Wait(0xc42072fa00)\r\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb2750, 0x38c8140, 0xc4209c0d50)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38c8140, 0xc4209c0d50)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421a33ee0, 0xc420bb2240, 0xc421a33ed0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 51988 [runnable]:\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).coalescedHeartbeatsLoop(0xc4208a4400, 0x38c8140, 0xc4218ffbf0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3717 +0x1a6\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).(github.com/cockroachdb/cockroach/pkg/storage.coalescedHeartbeatsLoop)-fm(0x38c8140, 0xc4218ffbf0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3663 +0x56\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42154c450, 0xc420bb2240, 0xc42154c380)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 51954 [runnable]:\r\nsync.runtime_notifyListWait(0xc42072fa10, 0x20)\r\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\r\nsync.(*Cond).Wait(0xc42072fa00)\r\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb2750, 0x38c8140, 0xc4209c0cf0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38c8140, 0xc4209c0cf0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421a33e60, 0xc420bb2240, 0xc421a33e50)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 50711 [select]:\r\ngithub.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.loopyWriter(0x38c8140, 0xc4216af9b0, 0xc4216af8f0, 0xc42007cfb8)\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:727 +0x420\r\ngithub.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server.func2(0xc421a77080)\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:261 +0xb7\r\ncreated by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:260 +0x177a\r\n\r\ngoroutine 51926 [runnable]:\r\nsync.runtime_notifyListWait(0xc42072fa10, 0x3)\r\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\r\nsync.(*Cond).Wait(0xc42072fa00)\r\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb2750, 0x38c8140, 0xc420dbfd10)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38c8140, 0xc420dbfd10)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421a33ae0, 0xc420bb2240, 0xc421a33ad0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 51930 [runnable]:\r\nsync.runtime_notifyListWait(0xc42072fa10, 0x7)\r\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\r\nsync.(*Cond).Wait(0xc42072fa00)\r\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb2750, 0x38c8140, 0xc4218ff650)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38c8140, 0xc4218ff650)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421a33b60, 0xc420bb2240, 0xc421a33b50)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 50734 [select]:\r\ngithub.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1(0x38c8140, 0xc420768180)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1152 +0x43f\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42154dad0, 0xc421516090, 0xc42154dac0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 51983 [runnable]:\r\nsync.runtime_notifyListWait(0xc42072fa10, 0x3d)\r\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\r\nsync.(*Cond).Wait(0xc42072fa00)\r\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb2750, 0x38c8140, 0xc4209c1050)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38c8140, 0xc4209c1050)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42154c2f0, 0xc420bb2240, 0xc42154c2e0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 50429 [select]:\r\ngithub.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.loopyWriter(0x38c8080, 0xc420b380c0, 0xc4218b2210, 0xc420486fb8)\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:727 +0x420\r\ngithub.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client.func3(0xc420dd8f00)\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:305 +0xb7\r\ncreated by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:304 +0x16c7\r\n\r\ngoroutine 51972 [runnable]:\r\nsync.runtime_notifyListWait(0xc42072fa10, 0x32)\r\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\r\nsync.(*Cond).Wait(0xc42072fa00)\r\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb2750, 0x38c8140, 0xc4218ffa70)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38c8140, 0xc4218ffa70)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42154c100, 0xc420bb2240, 0xc42154c0d0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 51986 [semacquire]:\r\nsync.runtime_Semacquire(0xc420bb27d4)\r\n\t/usr/local/go/src/runtime/sema.go:56 +0x39\r\nsync.(*WaitGroup).Wait(0xc420bb27c8)\r\n\t/usr/local/go/src/sync/waitgroup.go:129 +0xb3\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait(0xc420bb2750, 0x38c8140, 0xc4218ffbc0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:172 +0x3f\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait-fm(0x38c8140, 0xc4218ffbc0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3660 +0x56\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42154c350, 0xc420bb2240, 0xc42154c340)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 50621 [select]:\r\ngithub.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc420b382c0)\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:122 +0x1d4\r\ncreated by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:113 +0x291\r\n\r\ngoroutine 51990 [runnable]:\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x38c8140, 0xc4209c1230)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1551 +0x499\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42154c4c0, 0xc420bb2240, 0xc420efba90)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 51935 [runnable]:\r\nsync.runtime_notifyListWait(0xc42072fa10, 0xe)\r\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\r\nsync.(*Cond).Wait(0xc42072fa00)\r\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb2750, 0x38c8140, 0xc420dbfdd0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38c8140, 0xc420dbfdd0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421a33c00, 0xc420bb2240, 0xc421a33bf0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 51924 [runnable]:\r\nsync.runtime_notifyListWait(0xc42072fa10, 0x0)\r\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\r\nsync.(*Cond).Wait(0xc42072fa00)\r\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb2750, 0x38c8140, 0xc420dbfc50)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38c8140, 0xc420dbfc50)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421a33aa0, 0xc420bb2240, 0xc421a33a90)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 51955 [runnable]:\r\nsync.runtime_notifyListWait(0xc42072fa10, 0x21)\r\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\r\nsync.(*Cond).Wait(0xc42072fa00)\r\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb2750, 0x38c8140, 0xc4218ff890)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38c8140, 0xc4218ff890)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421a33e80, 0xc420bb2240, 0xc421a33e70)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 51897 [runnable]:\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x38c8140, 0xc4209c12f0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:586 +0x1c8\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42040a370, 0xc420bb2240, 0xc4203b6280)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 50736 [select]:\r\ngithub.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip(0xc420bad2b0, 0x38c8140, 0xc420768240, 0xc421708600, 0x38e1ce0, 0xc42154dba0, 0xc421516090, 0xc420374fc0, 0x0, 0x0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:323 +0x400\r\ngithub.com/cockroachdb/cockroach/pkg/gossip.(*client).startLocked.func1(0x38c8140, 0xc420768240)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:130 +0x5fd\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42154db60, 0xc421516090, 0xc421ba4f00)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 51969 [runnable]:\r\nsync.runtime_notifyListWait(0xc42072fa10, 0x2f)\r\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\r\nsync.(*Cond).Wait(0xc42072fa00)\r\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb2750, 0x38c8140, 0xc4209c0ea0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38c8140, 0xc4209c0ea0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42154c060, 0xc420bb2240, 0xc42154c050)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 51936 [runnable]:\r\nsync.runtime_notifyListWait(0xc42072fa10, 0xf)\r\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\r\nsync.(*Cond).Wait(0xc42072fa00)\r\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb2750, 0x38c8140, 0xc420dbff80)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38c8140, 0xc420dbff80)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421a33c20, 0xc420bb2240, 0xc421a33c10)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 51927 [runnable]:\r\nsync.runtime_notifyListWait(0xc42072fa10, 0x8)\r\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\r\nsync.(*Cond).Wait(0xc42072fa00)\r\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb2750, 0x38c8140, 0xc4218ff680)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38c8140, 0xc4218ff680)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421a33b00, 0xc420bb2240, 0xc421a33af0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 50714 [select]:\r\ngithub.com/cockroachdb/cockroach/pkg/gossip.(*server).Gossip(0xc4216e0a50, 0x38e1d40, 0xc421a32f40, 0x0, 0x0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:185 +0xa61\r\ngithub.com/cockroachdb/cockroach/pkg/gossip._Gossip_Gossip_Handler(0x3245500, 0xc4216e0a50, 0x38d8bc0, 0xc42185b540, 0xc420330088, 0xc42011fb80)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.pb.go:208 +0xd3\r\ngithub.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc420bc8160, 0x38dbfe0, 0xc420a41b00, 0xc42011fb80, 0xc420bfb080, 0x49459e0, 0x0, 0x0, 0x0)\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1053 +0x1221\r\ngithub.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc420bc8160, 0x38dbfe0, 0xc420a41b00, 0xc42011fb80, 0x0)\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1147 +0x12f9\r\ngithub.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc42041c020, 0xc420bc8160, 0x38dbfe0, 0xc420a41b00, 0xc42011fb80)\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:638 +0xad\r\ncreated by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:636 +0xb9\r\n\r\ngoroutine 51967 [runnable]:\r\nsync.runtime_notifyListWait(0xc42072fa10, 0x2d)\r\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\r\nsync.(*Cond).Wait(0xc42072fa00)\r\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb2750, 0x38c8140, 0xc4209c0e70)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38c8140, 0xc4209c0e70)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42154c020, 0xc420bb2240, 0xc421a33ff0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 50616 [select]:\r\ngithub.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).manage.func1(0x38c8140, 0xc4216ae630)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1190 +0x2fe\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420474320, 0xc421516090, 0xc420474300)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 51329 [select]:\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc4216e0780, 0x3, 0xc4209601e0, 0xc421906040, 0x38e2460, 0xc4203d8020, 0x0, 0x0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:517 +0x764\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).connectAndProcess.func1(0xc4216e0780, 0x3, 0x38c8140, 0xc420bfa270, 0x1, 0x0, 0xc42078c140, 0xc4209601e0, 0xc421906040, 0x0, ...)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:462 +0x378\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).connectAndProcess(0xc4216e0780, 0x38c8140, 0xc420bfa270, 0x3, 0xc4209601e0, 0xc421906040)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:463 +0x103\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).SendAsync.func1.1(0x38c8140, 0xc420bfa270)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:591 +0x7c\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203f4360, 0xc421516090, 0xc4207b00f0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 51974 [runnable]:\r\nsync.runtime_notifyListWait(0xc42072fa10, 0x34)\r\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\r\nsync.(*Cond).Wait(0xc42072fa00)\r\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb2750, 0x38c8140, 0xc4218ffaa0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38c8140, 0xc4218ffaa0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42154c150, 0xc420bb2240, 0xc42154c140)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 51900 [runnable]:\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x38c8140, 0xc4209c1290)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:586 +0x1c8\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42040a3a0, 0xc420bb2240, 0xc4203b6340)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 50430 [select]:\r\ngithub.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).keepalive(0xc420dd8f00)\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1323 +0x1aa\r\ncreated by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:309 +0x1750\r\n\r\ngoroutine 50513 [chan receive]:\r\ngithub.com/cockroachdb/cockroach/pkg/util/netutil.ListenAndServeGRPC.func1(0x38c8140, 0xc4214c2ba0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:49 +0x7a\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42179a9f0, 0xc421516090, 0xc4214c2a20)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 50624 [IO wait]:\r\ninternal/poll.runtime_pollWait(0x7f99367dfc90, 0x72, 0x4971090)\r\n\t/usr/local/go/src/runtime/netpoll.go:173 +0x5e\r\ninternal/poll.(*pollDesc).wait(0xc4202b3998, 0x72, 0x38a7200, 0x4971090, 0xffffffffffffffff)\r\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xe5\r\ninternal/poll.(*pollDesc).waitRead(0xc4202b3998, 0xc420978000, 0x8000, 0x8000)\r\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x4b\r\ninternal/poll.(*FD).Read(0xc4202b3980, 0xc420978000, 0x8000, 0x8000, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/internal/poll/fd_unix.go:157 +0x22a\r\nnet.(*netFD).Read(0xc4202b3980, 0xc420978000, 0x8000, 0x8000, 0x0, 0xc421a0d0f8, 0x10a0863)\r\n\t/usr/local/go/src/net/fd_unix.go:202 +0x66\r\nnet.(*conn).Read(0xc4216ca180, 0xc420978000, 0x8000, 0x8000, 0xc42004e500, 0xc42004e500, 0x0)\r\n\t/usr/local/go/src/net/net.go:176 +0x85\r\nbufio.(*Reader).Read(0xc4208da720, 0xc420722038, 0x9, 0x9, 0xc420071800, 0x10, 0xc42184cc50)\r\n\t/usr/local/go/src/bufio/bufio.go:216 +0x740\r\nio.ReadAtLeast(0x38a1740, 0xc4208da720, 0xc420722038, 0x9, 0x9, 0x9, 0x7d, 0xc42004e500, 0xa4d240)\r\n\t/usr/local/go/src/io/io.go:309 +0x94\r\nio.ReadFull(0x38a1740, 0xc4208da720, 0xc420722038, 0x9, 0x9, 0xc4216ca148, 0x38c8140, 0xc4216931d0)\r\n\t/usr/local/go/src/io/io.go:327 +0x73\r\ngithub.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc420722038, 0x9, 0x9, 0x38a1740, 0xc4208da720, 0x0, 0xc400000000, 0xc42184cdb5, 0xc421b6be60)\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x8f\r\ngithub.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc420722000, 0xc421b6be60, 0xc421b6be00, 0x0, 0x0)\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0x10f\r\ngithub.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc420b5e300)\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1173 +0x1b2\r\ncreated by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:273 +0x12ef\r\n\r\ngoroutine 51947 [runnable]:\r\nsync.runtime_notifyListWait(0xc42072fa10, 0x19)\r\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\r\nsync.(*Cond).Wait(0xc42072fa00)\r\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb2750, 0x38c8140, 0xc4218ff7d0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38c8140, 0xc4218ff7d0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421a33d80, 0xc420bb2240, 0xc421a33d70)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 51956 [runnable]:\r\nsync.runtime_notifyListWait(0xc42072fa10, 0x22)\r\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\r\nsync.(*Cond).Wait(0xc42072fa00)\r\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb2750, 0x38c8140, 0xc4209c0d20)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38c8140, 0xc4209c0d20)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421a33ea0, 0xc420bb2240, 0xc421a33e90)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 50859 [chan receive]:\r\ngithub.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func3(0x38c8140, 0xc4214c39b0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:371 +0x83\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42179af90, 0xc421516090, 0xc420467540)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 50851 [select]:\r\ngithub.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccResolverWrapper).watcher(0xc4214c2ae0)\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:110 +0x205\r\ncreated by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccResolverWrapper).start\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:96 +0x4d\r\n\r\ngoroutine 51953 [runnable]:\r\nsync.runtime_notifyListWait(0xc42072fa10, 0x1f)\r\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\r\nsync.(*Cond).Wait(0xc42072fa00)\r\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb2750, 0x38c8140, 0xc4218ff860)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38c8140, 0xc4218ff860)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421a33e40, 0xc420bb2240, 0xc421a33e30)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 51959 [runnable]:\r\nsync.runtime_notifyListWait(0xc42072fa10, 0x25)\r\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\r\nsync.(*Cond).Wait(0xc42072fa00)\r\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb2750, 0x38c8140, 0xc4209c0db0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38c8140, 0xc4209c0db0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421a33f00, 0xc420bb2240, 0xc421a33ef0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 51975 [runnable]:\r\nsync.runtime_notifyListWait(0xc42072fa10, 0x35)\r\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\r\nsync.(*Cond).Wait(0xc42072fa00)\r\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb2750, 0x38c8140, 0xc4209c0f90)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38c8140, 0xc4209c0f90)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42154c1f0, 0xc420bb2240, 0xc42154c160)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 51322 [select]:\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch(0xc4216e0780, 0x38e24c0, 0xc421a334e0, 0x38ae040, 0xc4216e0780)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:359 +0x2db\r\ngithub.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftMessageBatch_Handler(0x327a680, 0xc4216e0780, 0x38d8bc0, 0xc4208f63c0, 0xc420330088, 0xc420d97680)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:415 +0xd3\r\ngithub.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc420012c60, 0x38dbfe0, 0xc421a77080, 0xc420d97680, 0xc4216c57d0, 0x494f2c0, 0x0, 0x0, 0x0)\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1053 +0x1221\r\ngithub.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc420012c60, 0x38dbfe0, 0xc421a77080, 0xc420d97680, 0x0)\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1147 +0x12f9\r\ngithub.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc4202c1c30, 0xc420012c60, 0x38dbfe0, 0xc421a77080, 0xc420d97680)\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:638 +0xad\r\ncreated by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:636 +0xb9\r\n\r\ngoroutine 50737 [select]:\r\ngithub.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func4(0x38e26a0, 0xc420dd8f00, 0xc42065fb80, 0xc421362b40, 0xc421afa500)\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:284 +0x219\r\ncreated by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:283 +0x1088\r\n\r\ngoroutine 39928 [chan receive]:\r\ngithub.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc421362480)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1210 +0x8b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:82 +0x4bb\r\n\r\ngoroutine 50471 [runnable]:\r\nsync.(*WaitGroup).Wait(0xc4203ef280)\r\n\t/usr/local/go/src/sync/waitgroup.go:102 +0x147\r\ngithub.com/cockroachdb/cockroach/pkg/storage_test.verifyStats(0xc4216e02d0, 0xc4201043c0, 0xc4219ddd98, 0x3, 0x3)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_metrics_test.go:83 +0x4d9\r\ngithub.com/cockroachdb/cockroach/pkg/storage_test.TestStoreMetrics(0xc4216e02d0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_metrics_test.go:216 +0xdf6\r\ntesting.tRunner(0xc4216e02d0, 0x33ef798)\r\n\t/usr/local/go/src/testing/testing.go:777 +0x16e\r\ncreated by testing.(*T).Run\r\n\t/usr/local/go/src/testing/testing.go:824 +0x565\r\n\r\ngoroutine 50860 [select]:\r\ngithub.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1(0x38c8140, 0xc4214c3a10)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1152 +0x43f\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42179afe0, 0xc421516090, 0xc42179afa0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 50861 [select]:\r\ngithub.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).manage.func1(0x38c8140, 0xc420768900)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1190 +0x2fe\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42179b020, 0xc421516090, 0xc42179aff0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 50620 [select]:\r\ngithub.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc42011e500, 0xc420924550, 0xc4203ee1e0, 0xf, 0xc420b22de0, 0x0, 0x0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:575 +0x5a1\r\ngithub.com/cockroachdb/cockroach/pkg/rpc.(*Context).GRPCDial.func1.1.1(0x38c8140, 0xc4219c6b10)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:507 +0x8f\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204747b0, 0xc421516090, 0xc4219c6990)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 51966 [runnable]:\r\nsync.runtime_notifyListWait(0xc42072fa10, 0x2c)\r\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\r\nsync.(*Cond).Wait(0xc42072fa00)\r\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb2750, 0x38c8140, 0xc4218ff9e0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38c8140, 0xc4218ff9e0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421a33fe0, 0xc420bb2240, 0xc421a33fd0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 51928 [runnable]:\r\nsync.runtime_notifyListWait(0xc42072fa10, 0x6)\r\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\r\nsync.(*Cond).Wait(0xc42072fa00)\r\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb2750, 0x38c8140, 0xc4218ff620)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38c8140, 0xc4218ff620)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421a33b20, 0xc420bb2240, 0xc421a33b10)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 51896 [runnable]:\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x38c8140, 0xc4209c12c0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:586 +0x1c8\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42040a360, 0xc420bb2240, 0xc4203b6240)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 51931 [runnable]:\r\nsync.runtime_notifyListWait(0xc42072fa10, 0xa)\r\n\t/usr/local/go/src/runtime/sema.go:510 +0x11a\r\nsync.(*Cond).Wait(0xc42072fa00)\r\n\t/usr/local/go/src/sync/cond.go:56 +0x8e\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420bb2750, 0x38c8140, 0xc4218ff6e0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x38c8140, 0xc4218ff6e0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421a33b80, 0xc420bb2240, 0xc421a33b70)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 51324 [select]:\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc4216e0780, 0xc400000001, 0xc420759080, 0xc421ad5080, 0x38e2460, 0xc421a335f0, 0x0, 0x0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:517 +0x764\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).connectAndProcess.func1(0xc4216e0780, 0x1, 0x38c8140, 0xc420b60120, 0x0, 0x0, 0xc4204c8500, 0xc420759080, 0xc421ad5080, 0x0, ...)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:462 +0x378\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).connectAndProcess(0xc4216e0780, 0x38c8140, 0xc420b60120, 0x1, 0xc420759080, 0xc421ad5080)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:463 +0x103\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).SendAsync.func1.1(0x38c8140, 0xc420b60120)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:591 +0x7c\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421a335c0, 0xc421516090, 0xc420b600c0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 51347 [select]:\r\ngithub.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).waitOnHeader(0xc42011e3c0, 0x60, 0x31c8e40)\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:258 +0x1cd\r\ngithub.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).RecvCompress(0xc42011e3c0, 0x58, 0xa23a65)\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:271 +0x3d\r\ngithub.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc421992120, 0x3186c80, 0xc420dbe0c0, 0x0, 0x0)\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:428 +0xce8\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc4203d8020, 0xc420dbe060, 0xc42163eea8, 0xbb2c23)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:366 +0x83\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x38e2460, 0xc4203d8020, 0xc421906040, 0xc4216e0780, 0x38c8140, 0xc420dbe060, 0xa3bafd, 0xc42199b240)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:491 +0x5a\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1(0x38c8140, 0xc420dbe060)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:506 +0x7e\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d8060, 0xc421516090, 0xc420bfa8d0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 51304 [select]:\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc4216e0780, 0x2, 0xc420875d40, 0xc4217067c0, 0x38e2460, 0xc420994e80, 0x0, 0x0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:517 +0x764\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).connectAndProcess.func1(0xc4216e0780, 0x2, 0x38c8140, 0xc420b3eb70, 0x1, 0x0, 0xc42042b360, 0xc420875d40, 0xc4217067c0, 0x0, ...)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:462 +0x378\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).connectAndProcess(0xc4216e0780, 0x38c8140, 0xc420b3eb70, 0x2, 0xc420875d40, 0xc4217067c0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:463 +0x103\r\ngithub.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).SendAsync.func1.1(0x38c8140, 0xc420b3eb70)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:591 +0x7c\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d93e0, 0xc421516090, 0xc4216ad200)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb\r\n\r\ngoroutine 50843 [IO wait]:\r\ninternal/poll.runtime_pollWait(0x7f99367df950, 0x72, 0x4971090)\r\n\t/usr/local/go/src/runtime/netpoll.go:173 +0x5e\r\ninternal/poll.(*pollDesc).wait(0xc4202b3c98, 0x72, 0x38a7200, 0x4971090, 0xffffffffffffffff)\r\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xe5\r\ninternal/poll.(*pollDesc).waitRead(0xc4202b3c98, 0xc42087e000, 0x8000, 0x8000)\r\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x4b\r\ninternal/poll.(*FD).Read(0xc4202b3c80, 0xc42087e000, 0x8000, 0x8000, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/internal/poll/fd_unix.go:157 +0x22a\r\nnet.(*netFD).Read(0xc4202b3c80, 0xc42087e000, 0x8000, 0x8000, 0x0, 0xc421517188, 0x10a0863)\r\n\t/usr/local/go/src/net/fd_unix.go:202 +0x66\r\nnet.(*conn).Read(0xc4216ca2f0, 0xc42087e000, 0x8000, 0x8000, 0xc42004c000, 0xc42004c000, 0x0)\r\n\t/usr/local/go/src/net/net.go:176 +0x85\r\nbufio.(*Reader).Read(0xc4206271a0, 0xc420722b98, 0x9, 0x9, 0xc42048bc00, 0x10, 0xc420a26c50)\r\n\t/usr/local/go/src/bufio/bufio.go:216 +0x740\r\nio.ReadAtLeast(0x38a1740, 0xc4206271a0, 0xc420722b98, 0x9, 0x9, 0x9, 0x7d, 0xc42004c000, 0xa4d240)\r\n\t/usr/local/go/src/io/io.go:309 +0x94\r\nio.ReadFull(0x38a1740, 0xc4206271a0, 0xc420722b98, 0x9, 0x9, 0xc420b36180, 0x38c8140, 0xc420ba9200)\r\n\t/usr/local/go/src/io/io.go:327 +0x73\r\ngithub.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc420722b98, 0x9, 0x9, 0x38a1740, 0xc4206271a0, 0x0, 0xc400000000, 0xc420a26db5, 0xc420b3e9f0)\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x8f\r\ngithub.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc420722b60, 0xc420b3e9f0, 0xc420b3e900, 0x0, 0x0)\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0x10f\r\ngithub.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc420b5ea80)\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1173 +0x1b2\r\ncreated by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:273 +0x12ef\r\n\r\ngoroutine 50626 [select]:\r\ngithub.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).keepalive(0xc420b5e300)\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1323 +0x1aa\r\ncreated by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:309 +0x1750\r\n\r\ngoroutine 50472 [chan receive]:\r\ngithub.com/cockroachdb/cockroach/pkg/rpc.NewContext.func1(0x38c8140, 0xc420bfa600)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:321 +0x6c\r\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc421a320b0, 0xc421516090, 0xc4215ae400)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b\r\ncreated by github.com/cockroachdb/coc\n```\n\nPlease assign, take a look and update the issue accordingly.\n"},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/398430818","html_url":"https://github.com/cockroachdb/cockroach/issues/25746#issuecomment-398430818","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/25746","id":398430818,"node_id":"MDEyOklzc3VlQ29tbWVudDM5ODQzMDgxOA==","user":{"login":"andreimatei","id":377201,"node_id":"MDQ6VXNlcjM3NzIwMQ==","avatar_url":"https://avatars1.githubusercontent.com/u/377201?v=4","gravatar_id":"","url":"https://api.github.com/users/andreimatei","html_url":"https://github.com/andreimatei","followers_url":"https://api.github.com/users/andreimatei/followers","following_url":"https://api.github.com/users/andreimatei/following{/other_user}","gists_url":"https://api.github.com/users/andreimatei/gists{/gist_id}","starred_url":"https://api.github.com/users/andreimatei/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andreimatei/subscriptions","organizations_url":"https://api.github.com/users/andreimatei/orgs","repos_url":"https://api.github.com/users/andreimatei/repos","events_url":"https://api.github.com/users/andreimatei/events{/privacy}","received_events_url":"https://api.github.com/users/andreimatei/received_events","type":"User","site_admin":false},"created_at":"2018-06-19T14:59:32Z","updated_at":"2018-06-19T14:59:32Z","author_association":"MEMBER","body":"another failure in #26777"}}
{"actor":{"display_login":"andreimatei"}}