{"action":"opened","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/12074","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/12074/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/12074/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/12074/events","html_url":"https://github.com/cockroachdb/cockroach/issues/12074","id":193709572,"number":12074,"title":"github.com/cockroachdb/cockroach/pkg/kv: TestTxnDBWriteSkewAnomaly failed under stress","user":{"login":"cockroach-teamcity","id":20559710,"avatar_url":"https://avatars.githubusercontent.com/u/20559710?v=3","gravatar_id":"","url":"https://api.github.com/users/cockroach-teamcity","html_url":"https://github.com/cockroach-teamcity","followers_url":"https://api.github.com/users/cockroach-teamcity/followers","following_url":"https://api.github.com/users/cockroach-teamcity/following{/other_user}","gists_url":"https://api.github.com/users/cockroach-teamcity/gists{/gist_id}","starred_url":"https://api.github.com/users/cockroach-teamcity/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cockroach-teamcity/subscriptions","organizations_url":"https://api.github.com/users/cockroach-teamcity/orgs","repos_url":"https://api.github.com/users/cockroach-teamcity/repos","events_url":"https://api.github.com/users/cockroach-teamcity/events{/privacy}","received_events_url":"https://api.github.com/users/cockroach-teamcity/received_events","type":"User","site_admin":false},"labels":[{"id":335217860,"url":"https://api.github.com/repos/cockroachdb/cockroach/labels/Robot","name":"Robot","color":"5319e7","default":false},{"id":191587409,"url":"https://api.github.com/repos/cockroachdb/cockroach/labels/test-failure","name":"test-failure","color":"e11d21","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-12-06T08:21:05Z","updated_at":"2016-12-06T08:21:05Z","closed_at":null,"body":"SHA: https://github.com/cockroachdb/cockroach/commits/e39613f6dd5e838085c7738bf9b5febc06001dda\n\nParameters:\n```\nCOCKROACH_PROPOSER_EVALUATED_KV=false\nTAGS=stress\nGOFLAGS=\n```\n\nStress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=81755&tab=buildLog\n\n```\nI161206 08:15:53.528337 12757 gossip/gossip.go:248  [n?] initial resolvers: []\nW161206 08:15:53.528356 12757 gossip/gossip.go:1124  [n?] no resolvers found; use --join to specify a connected node\nI161206 08:15:53.528401 12757 storage/engine/rocksdb.go:340  opening in memory rocksdb instance\nI161206 08:15:53.529480 12757 base/node_id.go:62  NodeID set to 1\nI161206 08:15:53.529829 12757 storage/store.go:1229  [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available\nI161206 08:15:53.529866 12757 gossip/gossip.go:290  [n1] NodeDescriptor set to node_id:1 address:<network_field:\"\" address_field:\"\" > attrs:<> locality:<> \nI161206 08:15:53.529922 12757 kv/txn_correctness_test.go:658  verifying all possible histories for the \"write skew\" anomaly\nI161206 08:15:53.530614 12762 storage/replica_proposal.go:348  [n1,s1,r1/1:/M{in-ax},@c42017bb00] new range lease replica {1 1 1} 1970-01-01 00:00:00 +0000 UTC 950.000123ms following replica {0 0 0} 1970-01-01 00:00:00 +0000 UTC 0s [physicalTime=1970-01-01 00:00:00.000000123 +0000 UTC]\nI161206 08:15:54.325352 12757 util/stop/stopper.go:396  stop has been called, stopping or quiescing all running tasks\nI161206 08:15:54.326796 12757 gossip/gossip.go:248  [n?] initial resolvers: []\nW161206 08:15:54.326822 12757 gossip/gossip.go:1124  [n?] no resolvers found; use --join to specify a connected node\nI161206 08:15:54.326890 12757 storage/engine/rocksdb.go:340  opening in memory rocksdb instance\nI161206 08:15:54.328161 12757 base/node_id.go:62  NodeID set to 1\nI161206 08:15:54.329115 13161 storage/replica_proposal.go:348  [n1,s1,r1/1:/M{in-ax},@c42080c000] new range lease replica {1 1 1} 1970-01-01 00:00:00 +0000 UTC 950.000123ms following replica {0 0 0} 1970-01-01 00:00:00 +0000 UTC 0s [physicalTime=1970-01-01 00:00:00.000000123 +0000 UTC]\nI161206 08:15:54.329453 12757 storage/store.go:1229  [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available\nI161206 08:15:54.329495 12757 gossip/gossip.go:290  [n1] NodeDescriptor set to node_id:1 address:<network_field:\"\" address_field:\"\" > attrs:<> locality:<> \nI161206 08:15:54.329593 12757 kv/txn_correctness_test.go:658  verifying all possible histories for the \"write skew\" anomaly\nI161206 08:15:54.482249 12757 util/stop/stopper.go:396  stop has been called, stopping or quiescing all running tasks\n\tleaktest.go:93: Leaked goroutine: goroutine 759 [chan receive, 3 minutes]:\n\t\tgithub.com/cockroachdb/cockroach/pkg/rpc.NewContext.func1()\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:137 +0x76\n\t\tgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420088090, 0xc4207ce180)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\n\t\tcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\tleaktest.go:93: Leaked goroutine: goroutine 760 [chan receive, 3 minutes]:\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open.func1(0xc42079f4a0)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:373 +0x44\n\t\tcreated by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:374 +0x702\n\tleaktest.go:93: Leaked goroutine: goroutine 762 [chan receive, 3 minutes]:\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func1()\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:171 +0x55\n\t\tgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420088090, 0xc4202a7f20)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\n\t\tcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\tleaktest.go:93: Leaked goroutine: goroutine 763 [semacquire]:\n\t\tsync.runtime_notifyListWait(0xc4207a0ad0, 0xc400004fa6)\n\t\t\t/usr/local/go/src/runtime/sema.go:267 +0x122\n\t\tsync.(*Cond).Wait(0xc4207a0ac0)\n\t\t\t/usr/local/go/src/sync/cond.go:57 +0x80\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420552c60, 0xc420088090)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:212 +0xb2\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\n\t\tgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420088090, 0xc4202a7f40)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\n\t\tcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\tleaktest.go:93: Leaked goroutine: goroutine 764 [semacquire]:\n\t\tsync.runtime_notifyListWait(0xc4207a0ad0, 0xc400004fb0)\n\t\t\t/usr/local/go/src/runtime/sema.go:267 +0x122\n\t\tsync.(*Cond).Wait(0xc4207a0ac0)\n\t\t\t/usr/local/go/src/sync/cond.go:57 +0x80\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420552c60, 0xc420088090)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:212 +0xb2\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\n\t\tgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420088090, 0xc4202a7f60)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\n\t\tcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\tleaktest.go:93: Leaked goroutine: goroutine 765 [semacquire]:\n\t\tsync.runtime_notifyListWait(0xc4207a0ad0, 0xc400004fb3)\n\t\t\t/usr/local/go/src/runtime/sema.go:267 +0x122\n\t\tsync.(*Cond).Wait(0xc4207a0ac0)\n\t\t\t/usr/local/go/src/sync/cond.go:57 +0x80\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420552c60, 0xc420088090)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:212 +0xb2\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\n\t\tgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420088090, 0xc4202a7f80)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\n\t\tcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\tleaktest.go:93: Leaked goroutine: goroutine 766 [semacquire]:\n\t\tsync.runtime_notifyListWait(0xc4207a0ad0, 0xc400004faa)\n\t\t\t/usr/local/go/src/runtime/sema.go:267 +0x122\n\t\tsync.(*Cond).Wait(0xc4207a0ac0)\n\t\t\t/usr/local/go/src/sync/cond.go:57 +0x80\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420552c60, 0xc420088090)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:212 +0xb2\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\n\t\tgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420088090, 0xc4202a7fa0)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\n\t\tcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\tleaktest.go:93: Leaked goroutine: goroutine 767 [semacquire]:\n\t\tsync.runtime_notifyListWait(0xc4207a0ad0, 0xc400004fb1)\n\t\t\t/usr/local/go/src/runtime/sema.go:267 +0x122\n\t\tsync.(*Cond).Wait(0xc4207a0ac0)\n\t\t\t/usr/local/go/src/sync/cond.go:57 +0x80\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420552c60, 0xc420088090)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:212 +0xb2\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\n\t\tgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420088090, 0xc4202a7fc0)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\n\t\tcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\tleaktest.go:93: Leaked goroutine: goroutine 768 [semacquire]:\n\t\tsync.runtime_notifyListWait(0xc4207a0ad0, 0xc400004faf)\n\t\t\t/usr/local/go/src/runtime/sema.go:267 +0x122\n\t\tsync.(*Cond).Wait(0xc4207a0ac0)\n\t\t\t/usr/local/go/src/sync/cond.go:57 +0x80\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420552c60, 0xc420088090)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:212 +0xb2\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\n\t\tgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420088090, 0xc4202a7fe0)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\n\t\tcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\tleaktest.go:93: Leaked goroutine: goroutine 769 [semacquire]:\n\t\tsync.runtime_notifyListWait(0xc4207a0ad0, 0xc400004fa8)\n\t\t\t/usr/local/go/src/runtime/sema.go:267 +0x122\n\t\tsync.(*Cond).Wait(0xc4207a0ac0)\n\t\t\t/usr/local/go/src/sync/cond.go:57 +0x80\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420552c60, 0xc420088090)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:212 +0xb2\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\n\t\tgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420088090, 0xc42012a000)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\n\t\tcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\tleaktest.go:93: Leaked goroutine: goroutine 770 [semacquire]:\n\t\tsync.runtime_notifyListWait(0xc4207a0ad0, 0xc400004fae)\n\t\t\t/usr/local/go/src/runtime/sema.go:267 +0x122\n\t\tsync.(*Cond).Wait(0xc4207a0ac0)\n\t\t\t/usr/local/go/src/sync/cond.go:57 +0x80\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420552c60, 0xc420088090)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:212 +0xb2\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\n\t\tgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420088090, 0xc42012a040)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\n\t\tcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\tleaktest.go:93: Leaked goroutine: goroutine 771 [semacquire]:\n\t\tsync.runtime_notifyListWait(0xc4207a0ad0, 0xc400004fb2)\n\t\t\t/usr/local/go/src/runtime/sema.go:267 +0x122\n\t\tsync.(*Cond).Wait(0xc4207a0ac0)\n\t\t\t/usr/local/go/src/sync/cond.go:57 +0x80\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420552c60, 0xc420088090)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:212 +0xb2\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\n\t\tgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420088090, 0xc42012a060)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\n\t\tcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\tleaktest.go:93: Leaked goroutine: goroutine 772 [semacquire]:\n\t\tsync.runtime_notifyListWait(0xc4207a0ad0, 0xc400004fa7)\n\t\t\t/usr/local/go/src/runtime/sema.go:267 +0x122\n\t\tsync.(*Cond).Wait(0xc4207a0ac0)\n\t\t\t/usr/local/go/src/sync/cond.go:57 +0x80\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420552c60, 0xc420088090)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:212 +0xb2\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\n\t\tgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420088090, 0xc42012a080)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\n\t\tcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\tleaktest.go:93: Leaked goroutine: goroutine 773 [semacquire]:\n\t\tsync.runtime_notifyListWait(0xc4207a0ad0, 0xc400004fad)\n\t\t\t/usr/local/go/src/runtime/sema.go:267 +0x122\n\t\tsync.(*Cond).Wait(0xc4207a0ac0)\n\t\t\t/usr/local/go/src/sync/cond.go:57 +0x80\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420552c60, 0xc420088090)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:212 +0xb2\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\n\t\tgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420088090, 0xc42012a0a0)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\n\t\tcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\tleaktest.go:93: Leaked goroutine: goroutine 774 [semacquire]:\n\t\tsync.runtime_notifyListWait(0xc4207a0ad0, 0xc400004fb4)\n\t\t\t/usr/local/go/src/runtime/sema.go:267 +0x122\n\t\tsync.(*Cond).Wait(0xc4207a0ac0)\n\t\t\t/usr/local/go/src/sync/cond.go:57 +0x80\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420552c60, 0xc420088090)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:212 +0xb2\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\n\t\tgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420088090, 0xc42012a0c0)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\n\t\tcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\tleaktest.go:93: Leaked goroutine: goroutine 775 [semacquire]:\n\t\tsync.runtime_notifyListWait(0xc4207a0ad0, 0xc400004fac)\n\t\t\t/usr/local/go/src/runtime/sema.go:267 +0x122\n\t\tsync.(*Cond).Wait(0xc4207a0ac0)\n\t\t\t/usr/local/go/src/sync/cond.go:57 +0x80\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420552c60, 0xc420088090)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:212 +0xb2\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\n\t\tgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420088090, 0xc42012a0e0)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\n\t\tcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\tleaktest.go:93: Leaked goroutine: goroutine 776 [semacquire]:\n\t\tsync.runtime_notifyListWait(0xc4207a0ad0, 0xc400004fab)\n\t\t\t/usr/local/go/src/runtime/sema.go:267 +0x122\n\t\tsync.(*Cond).Wait(0xc4207a0ac0)\n\t\t\t/usr/local/go/src/sync/cond.go:57 +0x80\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420552c60, 0xc420088090)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:212 +0xb2\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\n\t\tgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420088090, 0xc42012a100)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\n\t\tcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\tleaktest.go:93: Leaked goroutine: goroutine 777 [semacquire]:\n\t\tsync.runtime_notifyListWait(0xc4207a0ad0, 0xc400004fa9)\n\t\t\t/usr/local/go/src/runtime/sema.go:267 +0x122\n\t\tsync.(*Cond).Wait(0xc4207a0ac0)\n\t\t\t/usr/local/go/src/sync/cond.go:57 +0x80\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420552c60, 0xc420088090)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:212 +0xb2\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\n\t\tgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420088090, 0xc42012a120)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\n\t\tcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\tleaktest.go:93: Leaked goroutine: goroutine 778 [semacquire]:\n\t\tsync.runtime_notifyListWait(0xc4207a0ad0, 0xc400004fb5)\n\t\t\t/usr/local/go/src/runtime/sema.go:267 +0x122\n\t\tsync.(*Cond).Wait(0xc4207a0ac0)\n\t\t\t/usr/local/go/src/sync/cond.go:57 +0x80\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420552c60, 0xc420088090)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:212 +0xb2\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\n\t\tgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420088090, 0xc42012a140)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\n\t\tcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\tleaktest.go:93: Leaked goroutine: goroutine 779 [semacquire, 3 minutes]:\n\t\tsync.runtime_Semacquire(0xc420552d04)\n\t\t\t/usr/local/go/src/runtime/sema.go:47 +0x30\n\t\tsync.(*WaitGroup).Wait(0xc420552cf8)\n\t\t\t/usr/local/go/src/sync/waitgroup.go:131 +0x97\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait(0xc420552c60)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:187 +0x34\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaft.func1()\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3354 +0x3a\n\t\tgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420088090, 0xc420794280)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\n\t\tcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\tleaktest.go:93: Leaked goroutine: goroutine 783 [select, 3 minutes]:\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*Store).Start.func3()\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1198 +0x165\n\t\tgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420088090, 0xc42012a1c0)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\n\t\tcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\tleaktest.go:93: Leaked goroutine: goroutine 804 [select, 3 minutes]:\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1()\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:474 +0x383\n\t\tgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420088090, 0xc420616ae0)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\n\t\tcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\tleaktest.go:93: Leaked goroutine: goroutine 805 [select, 3 minutes]:\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1()\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:474 +0x383\n\t\tgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420088090, 0xc420616b00)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\n\t\tcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\tleaktest.go:93: Leaked goroutine: goroutine 806 [select, 3 minutes]:\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1()\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:474 +0x383\n\t\tgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420088090, 0xc420616b20)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\n\t\tcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\tleaktest.go:93: Leaked goroutine: goroutine 807 [select, 3 minutes]:\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1()\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:474 +0x383\n\t\tgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420088090, 0xc420616b40)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\n\t\tcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\tleaktest.go:93: Leaked goroutine: goroutine 808 [select, 3 minutes]:\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1()\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:474 +0x383\n\t\tgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420088090, 0xc420616b60)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\n\t\tcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\tleaktest.go:93: Leaked goroutine: goroutine 809 [select, 3 minutes]:\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1()\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:474 +0x383\n\t\tgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420088090, 0xc420616b80)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\n\t\tcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\tleaktest.go:93: Leaked goroutine: goroutine 810 [select, 3 minutes]:\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).waitAndProcess(0xc42065f400, 0x2ac7d7d29878, 0xc420343560, 0xecfd86a08, 0x34241049, 0x2971080, 0xc4207a0080, 0xc420088090, 0xc420097200, 0xecfd86a08)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:207 +0x254\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1.1(0xc420097200, 0xc420593a10)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:269 +0x8e\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*storeReplicaVisitor).Visit(0xc420593a10, 0xc42060cfa0)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:324 +0x303\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1()\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:271 +0x2ca\n\t\tgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420088090, 0xc420616ba0)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\n\t\tcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\tleaktest.go:93: Leaked goroutine: goroutine 813 [select, 3 minutes]:\n\t\tgithub.com/cockroachdb/cockroach/pkg/storage.(*idAllocator).start.func1()\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/id_alloc.go:133 +0x7c8\n\t\tgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420088090, 0xc42014aa30)\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\n\t\tcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t\t\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n```"}}
{"actor":{"display_login":"cockroach-teamcity"}}