{"action":"opened","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/12312","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/12312/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/12312/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/12312/events","html_url":"https://github.com/cockroachdb/cockroach/issues/12312","id":195214889,"number":12312,"title":"github.com/cockroachdb/cockroach/pkg/storage: TestStoreRangeSystemSplits failed under stress","user":{"login":"cockroach-teamcity","id":20559710,"avatar_url":"https://avatars.githubusercontent.com/u/20559710?v=3","gravatar_id":"","url":"https://api.github.com/users/cockroach-teamcity","html_url":"https://github.com/cockroach-teamcity","followers_url":"https://api.github.com/users/cockroach-teamcity/followers","following_url":"https://api.github.com/users/cockroach-teamcity/following{/other_user}","gists_url":"https://api.github.com/users/cockroach-teamcity/gists{/gist_id}","starred_url":"https://api.github.com/users/cockroach-teamcity/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cockroach-teamcity/subscriptions","organizations_url":"https://api.github.com/users/cockroach-teamcity/orgs","repos_url":"https://api.github.com/users/cockroach-teamcity/repos","events_url":"https://api.github.com/users/cockroach-teamcity/events{/privacy}","received_events_url":"https://api.github.com/users/cockroach-teamcity/received_events","type":"User","site_admin":false},"labels":[{"id":335217860,"url":"https://api.github.com/repos/cockroachdb/cockroach/labels/Robot","name":"Robot","color":"5319e7","default":false},{"id":191587409,"url":"https://api.github.com/repos/cockroachdb/cockroach/labels/test-failure","name":"test-failure","color":"e11d21","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-12-13T10:27:27Z","updated_at":"2016-12-13T10:27:27Z","closed_at":null,"body":"SHA: https://github.com/cockroachdb/cockroach/commits/05ef61ce5a2ee648b16ce65beba890d06c7510eb\n\nParameters:\n```\nCOCKROACH_PROPOSER_EVALUATED_KV=false\nTAGS=deadlock\nGOFLAGS=\n```\n\nStress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=89435&tab=buildLog\n\n```\nI161213 10:27:25.647230 1477326 storage/engine/rocksdb.go:340  opening in memory rocksdb instance\nI161213 10:27:25.647968 1477326 gossip/gossip.go:248  [n?] initial resolvers: []\nW161213 10:27:25.648057 1477326 gossip/gossip.go:1124  [n?] no resolvers found; use --join to specify a connected node\nI161213 10:27:25.648463 1477326 base/node_id.go:62  NodeID set to 1\nI161213 10:27:25.648543 1477326 gossip/gossip.go:290  [n1] NodeDescriptor set to node_id:1 address:<network_field:\"\" address_field:\"\" > attrs:<> locality:<> \nI161213 10:27:25.656459 1477326 storage/store.go:1240  [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available\nI161213 10:27:25.658772 1477594 storage/replica_proposal.go:390  [s1,r1/1:/M{in-ax},@c42d994f00] new range lease repl={1 1 1} start=0.000000000,0 exp=0.900000123,8 pro=0.000000123,9 following repl={0 0 0} start=0.000000000,0 exp=0.000000000,0 [physicalTime=1970-01-01 00:00:00.000000123 +0000 UTC]\nI161213 10:27:25.660356 1477743 storage/split_queue.go:101  [split,s1,r1/1:/M{in-ax},@c42d994f00] splitting at keys [/Table/11/0 /Table/12/0 /Table/13/0 /Table/14/0]\nI161213 10:27:25.666557 1477743 storage/replica_command.go:2389  [split,s1,r1/1:/M{in-ax},@c42d994f00] initiating a split of this range at key /Table/11 [r2]\nE161213 10:27:25.677328 1477743 storage/queue.go:597  [split,s1,r1/1:/{Min-Table/11},@c42d994f00] unable to split [n1,s1,r1/1:/{Min-Table/11}] at key \"/Table/12/0\": key range /Table/12/0-/Table/12/0 outside of bounds of range /Min-/Max\nE161213 10:27:25.677885 1477744 storage/queue.go:608  [replicate,s1,r1/1:/{Min-Table/11},@c42d994f00] purgatory: 0 of 0 stores with an attribute matching []; likely not enough nodes in cluster\nI161213 10:27:25.678121 1477743 storage/split_queue.go:101  [split,s1,r2/1:/{Table/11-Max},@c42d995400] splitting at keys [/Table/12/0 /Table/13/0 /Table/14/0]\nI161213 10:27:25.678206 1477743 storage/replica_command.go:2389  [split,s1,r2/1:/{Table/11-Max},@c42d995400] initiating a split of this range at key /Table/12 [r3]\nW161213 10:27:25.680978 1477743 storage/stores.go:218  range not contained in one range: [/Meta2/Table/12,/Table/12/NULL), but have [/Min,/Table/11)\nW161213 10:27:25.692543 1478966 storage/intent_resolver.go:332  [n1,s1,r1/1:/{Min-Table/11}]: failed to push during intent resolution: failed to push \"storage/replica_command.go:2459 (*Replica).adminSplitWithDescriptor\" id=75139fca key=/Local/Range/\"\\x93\"/RangeDescriptor rw=true pri=0.00537536 iso=SERIALIZABLE stat=PENDING epo=0 ts=0.000000123,55 orig=0.000000123,55 max=0.000000123,55 wto=false rop=false\nE161213 10:27:25.697743 1477744 storage/queue.go:608  [replicate,s1,r2/1:/Table/1{1-2},@c42d995400] purgatory: 0 of 0 stores with an attribute matching []; likely not enough nodes in cluster\nE161213 10:27:25.698046 1477743 storage/queue.go:597  [split,s1,r2/1:/Table/1{1-2},@c42d995400] unable to split [n1,s1,r2/1:/Table/1{1-2}] at key \"/Table/13/0\": key range /Table/13/0-/Table/13/0 outside of bounds of range /Table/11-/Max\nI161213 10:27:25.698492 1477743 storage/split_queue.go:101  [split,s1,r3/1:/{Table/12-Max},@c42ede4f00] splitting at keys [/Table/13/0 /Table/14/0]\nI161213 10:27:25.698696 1477743 storage/replica_command.go:2389  [split,s1,r3/1:/{Table/12-Max},@c42ede4f00] initiating a split of this range at key /Table/13 [r4]\nE161213 10:27:25.729920 1477744 storage/queue.go:608  [replicate,s1,r3/1:/Table/1{2-3},@c42ede4f00] purgatory: 0 of 0 stores with an attribute matching []; likely not enough nodes in cluster\nE161213 10:27:25.730347 1477743 storage/queue.go:597  [split,s1,r3/1:/Table/1{2-3},@c42ede4f00] unable to split [n1,s1,r3/1:/Table/1{2-3}] at key \"/Table/14/0\": key range /Table/14/0-/Table/14/0 outside of bounds of range /Table/12-/Max\nI161213 10:27:25.731054 1477743 storage/split_queue.go:101  [split,s1,r4/1:/{Table/13-Max},@c42ea87400] splitting at keys [/Table/14/0]\nI161213 10:27:25.731156 1477743 storage/replica_command.go:2389  [split,s1,r4/1:/{Table/13-Max},@c42ea87400] initiating a split of this range at key /Table/14 [r5]\nE161213 10:27:25.756042 1477744 storage/queue.go:608  [replicate,s1,r4/1:/Table/1{3-4},@c42ea87400] purgatory: 0 of 0 stores with an attribute matching []; likely not enough nodes in cluster\nE161213 10:27:25.756494 1477744 storage/queue.go:608  [replicate,s1,r5/1:/{Table/14-Max},@c431460a00] purgatory: 0 of 0 stores with an attribute matching []; likely not enough nodes in cluster\nI161213 10:27:25.773787 1477743 storage/split_queue.go:101  [split,s1,r5/1:/{Table/14-Max},@c431460a00] splitting at keys [/Table/50/0 /Table/51/0 /Table/52/0 /Table/53/0 /Table/54/0 /Table/55/0 /Table/56/0 /Table/57/0 /Table/58/0 /Table/59/0 /Table/60/0 /Table/61/0 /Table/62/0 /Table/63/0 /Table/64/0 /Table/65/0 /Table/66/0 /Table/67/0 /Table/68/0]\nI161213 10:27:25.773922 1477743 storage/replica_command.go:2389  [split,s1,r5/1:/{Table/14-Max},@c431460a00] initiating a split of this range at key /Table/50 [r6]\nE161213 10:27:25.789349 1477743 storage/queue.go:597  [split,s1,r5/1:/Table/{14-50},@c431460a00] unable to split [n1,s1,r5/1:/Table/{14-50}] at key \"/Table/51/0\": key range /Table/51/0-/Table/51/0 outside of bounds of range /Table/14-/Max\nI161213 10:27:25.790069 1477743 storage/split_queue.go:101  [split,s1,r6/1:/{Table/50-Max},@c431460f00] splitting at keys [/Table/51/0 /Table/52/0 /Table/53/0 /Table/54/0 /Table/55/0 /Table/56/0 /Table/57/0 /Table/58/0 /Table/59/0 /Table/60/0 /Table/61/0 /Table/62/0 /Table/63/0 /Table/64/0 /Table/65/0 /Table/66/0 /Table/67/0 /Table/68/0]\nI161213 10:27:25.791270 1477743 storage/replica_command.go:2389  [split,s1,r6/1:/{Table/50-Max},@c431460f00] initiating a split of this range at key /Table/51 [r7]\nE161213 10:27:25.836078 1477744 storage/queue.go:608  [replicate,s1,r6/1:/Table/5{0-1},@c431460f00] purgatory: 0 of 0 stores with an attribute matching []; likely not enough nodes in cluster\nE161213 10:27:25.836446 1477743 storage/queue.go:597  [split,s1,r6/1:/Table/5{0-1},@c431460f00] unable to split [n1,s1,r6/1:/Table/5{0-1}] at key \"/Table/52/0\": key range /Table/52/0-/Table/52/0 outside of bounds of range /Table/50-/Max\nI161213 10:27:25.836873 1477743 storage/split_queue.go:101  [split,s1,r7/1:/{Table/51-Max},@c42ea87900] splitting at keys [/Table/52/0 /Table/53/0 /Table/54/0 /Table/55/0 /Table/56/0 /Table/57/0 /Table/58/0 /Table/59/0 /Table/60/0 /Table/61/0 /Table/62/0 /Table/63/0 /Table/64/0 /Table/65/0 /Table/66/0 /Table/67/0 /Table/68/0]\nI161213 10:27:25.836956 1477743 storage/replica_command.go:2389  [split,s1,r7/1:/{Table/51-Max},@c42ea87900] initiating a split of this range at key /Table/52 [r8]\nE161213 10:27:25.861735 1477744 storage/queue.go:608  [replicate,s1,r7/1:/Table/5{1-2},@c42ea87900] purgatory: 0 of 0 stores with an attribute matching []; likely not enough nodes in cluster\nE161213 10:27:25.862081 1477743 storage/queue.go:597  [split,s1,r7/1:/Table/5{1-2},@c42ea87900] unable to split [n1,s1,r7/1:/Table/5{1-2}] at key \"/Table/53/0\": key range /Table/53/0-/Table/53/0 outside of bounds of range /Table/51-/Max\nI161213 10:27:25.863491 1477743 storage/split_queue.go:101  [split,s1,r8/1:/{Table/52-Max},@c430a2e500] splitting at keys [/Table/53/0 /Table/54/0 /Table/55/0 /Table/56/0 /Table/57/0 /Table/58/0 /Table/59/0 /Table/60/0 /Table/61/0 /Table/62/0 /Table/63/0 /Table/64/0 /Table/65/0 /Table/66/0 /Table/67/0 /Table/68/0]\nI161213 10:27:25.863577 1477743 storage/replica_command.go:2389  [split,s1,r8/1:/{Table/52-Max},@c430a2e500] initiating a split of this range at key /Table/53 [r9]\nE161213 10:27:25.902853 1477744 storage/queue.go:608  [replicate,s1,r8/1:/Table/5{2-3},@c430a2e500] purgatory: 0 of 0 stores with an attribute matching []; likely not enough nodes in cluster\nE161213 10:27:25.903308 1477743 storage/queue.go:597  [split,s1,r8/1:/Table/5{2-3},@c430a2e500] unable to split [n1,s1,r8/1:/Table/5{2-3}] at key \"/Table/54/0\": key range /Table/54/0-/Table/54/0 outside of bounds of range /Table/52-/Max\nI161213 10:27:25.903844 1477743 storage/split_queue.go:101  [split,s1,r9/1:/{Table/53-Max},@c42b553900] splitting at keys [/Table/54/0 /Table/55/0 /Table/56/0 /Table/57/0 /Table/58/0 /Table/59/0 /Table/60/0 /Table/61/0 /Table/62/0 /Table/63/0 /Table/64/0 /Table/65/0 /Table/66/0 /Table/67/0 /Table/68/0]\nI161213 10:27:25.903931 1477743 storage/replica_command.go:2389  [split,s1,r9/1:/{Table/53-Max},@c42b553900] initiating a split of this range at key /Table/54 [r10]\nE161213 10:27:25.918750 1477744 storage/queue.go:608  [replicate,s1,r9/1:/Table/5{3-4},@c42b553900] purgatory: 0 of 0 stores with an attribute matching []; likely not enough nodes in cluster\nE161213 10:27:25.919146 1477743 storage/queue.go:597  [split,s1,r9/1:/Table/5{3-4},@c42b553900] unable to split [n1,s1,r9/1:/Table/5{3-4}] at key \"/Table/55/0\": key range /Table/55/0-/Table/55/0 outside of bounds of range /Table/53-/Max\nI161213 10:27:25.920823 1477743 storage/split_queue.go:101  [split,s1,r10/1:/{Table/54-Max},@c430a2f400] splitting at keys [/Table/55/0 /Table/56/0 /Table/57/0 /Table/58/0 /Table/59/0 /Table/60/0 /Table/61/0 /Table/62/0 /Table/63/0 /Table/64/0 /Table/65/0 /Table/66/0 /Table/67/0 /Table/68/0]\nI161213 10:27:25.920909 1477743 storage/replica_command.go:2389  [split,s1,r10/1:/{Table/54-Max},@c430a2f400] initiating a split of this range at key /Table/55 [r11]\nE161213 10:27:25.942480 1477744 storage/queue.go:608  [replicate,s1,r10/1:/Table/5{4-5},@c430a2f400] purgatory: 0 of 0 stores with an attribute matching []; likely not enough nodes in cluster\nE161213 10:27:25.942947 1477743 storage/queue.go:597  [split,s1,r10/1:/Table/5{4-5},@c430a2f400] unable to split [n1,s1,r10/1:/Table/5{4-5}] at key \"/Table/56/0\": key range /Table/56/0-/Table/56/0 outside of bounds of range /Table/54-/Max\nI161213 10:27:25.944217 1477743 storage/split_queue.go:101  [split,s1,r11/1:/{Table/55-Max},@c42cdaef00] splitting at keys [/Table/56/0 /Table/57/0 /Table/58/0 /Table/59/0 /Table/60/0 /Table/61/0 /Table/62/0 /Table/63/0 /Table/64/0 /Table/65/0 /Table/66/0 /Table/67/0 /Table/68/0]\nI161213 10:27:25.944420 1477743 storage/replica_command.go:2389  [split,s1,r11/1:/{Table/55-Max},@c42cdaef00] initiating a split of this range at key /Table/56 [r12]\nE161213 10:27:25.962408 1477744 storage/queue.go:608  [replicate,s1,r11/1:/Table/5{5-6},@c42cdaef00] purgatory: 0 of 0 stores with an attribute matching []; likely not enough nodes in cluster\nE161213 10:27:25.963792 1477743 storage/queue.go:597  [split,s1,r11/1:/Table/5{5-6},@c42cdaef00] unable to split [n1,s1,r11/1:/Table/5{5-6}] at key \"/Table/57/0\": key range /Table/57/0-/Table/57/0 outside of bounds of range /Table/55-/Max\nI161213 10:27:25.964353 1477743 storage/split_queue.go:101  [split,s1,r12/1:/{Table/56-Max},@c43225a500] splitting at keys [/Table/57/0 /Table/58/0 /Table/59/0 /Table/60/0 /Table/61/0 /Table/62/0 /Table/63/0 /Table/64/0 /Table/65/0 /Table/66/0 /Table/67/0 /Table/68/0]\nI161213 10:27:25.964474 1477743 storage/replica_command.go:2389  [split,s1,r12/1:/{Table/56-Max},@c43225a500] initiating a split of this range at key /Table/57 [r13]\nE161213 10:27:26.015482 1477743 storage/queue.go:597  [split,s1,r12/1:/Table/5{6-7},@c43225a500] unable to split [n1,s1,r12/1:/Table/5{6-7}] at key \"/Table/58/0\": key range /Table/58/0-/Table/58/0 outside of bounds of range /Table/56-/Max\nE161213 10:27:26.015593 1477744 storage/queue.go:608  [replicate,s1,r12/1:/Table/5{6-7},@c43225a500] purgatory: 0 of 0 stores with an attribute matching []; likely not enough nodes in cluster\nI161213 10:27:26.016119 1477743 storage/split_queue.go:101  [split,s1,r13/1:/{Table/57-Max},@c42d404a00] splitting at keys [/Table/58/0 /Table/59/0 /Table/60/0 /Table/61/0 /Table/62/0 /Table/63/0 /Table/64/0 /Table/65/0 /Table/66/0 /Table/67/0 /Table/68/0]\nI161213 10:27:26.016204 1477743 storage/replica_command.go:2389  [split,s1,r13/1:/{Table/57-Max},@c42d404a00] initiating a split of this range at key /Table/58 [r14]\nE161213 10:27:26.033419 1477743 storage/queue.go:597  [split,s1,r13/1:/Table/5{7-8},@c42d404a00] unable to split [n1,s1,r13/1:/Table/5{7-8}] at key \"/Table/59/0\": key range /Table/59/0-/Table/59/0 outside of bounds of range /Table/57-/Max\nE161213 10:27:26.033728 1477744 storage/queue.go:608  [replicate,s1,r13/1:/Table/5{7-8},@c42d404a00] purgatory: 0 of 0 stores with an attribute matching []; likely not enough nodes in cluster\nI161213 10:27:26.035702 1477743 storage/split_queue.go:101  [split,s1,r14/1:/{Table/58-Max},@c42d404f00] splitting at keys [/Table/59/0 /Table/60/0 /Table/61/0 /Table/62/0 /Table/63/0 /Table/64/0 /Table/65/0 /Table/66/0 /Table/67/0 /Table/68/0]\nI161213 10:27:26.035845 1477743 storage/replica_command.go:2389  [split,s1,r14/1:/{Table/58-Max},@c42d404f00] initiating a split of this range at key /Table/59 [r15]\nE161213 10:27:26.057838 1477743 storage/queue.go:597  [split,s1,r14/1:/Table/5{8-9},@c42d404f00] unable to split [n1,s1,r14/1:/Table/5{8-9}] at key \"/Table/60/0\": key range /Table/60/0-/Table/60/0 outside of bounds of range /Table/58-/Max\nE161213 10:27:26.058313 1477744 storage/queue.go:608  [replicate,s1,r14/1:/Table/5{8-9},@c42d404f00] purgatory: 0 of 0 stores with an attribute matching []; likely not enough nodes in cluster\nI161213 10:27:26.060667 1477743 storage/split_queue.go:101  [split,s1,r15/1:/{Table/59-Max},@c42ede5400] splitting at keys [/Table/60/0 /Table/61/0 /Table/62/0 /Table/63/0 /Table/64/0 /Table/65/0 /Table/66/0 /Table/67/0 /Table/68/0]\nI161213 10:27:26.060909 1477743 storage/replica_command.go:2389  [split,s1,r15/1:/{Table/59-Max},@c42ede5400] initiating a split of this range at key /Table/60 [r16]\nE161213 10:27:26.099797 1477743 storage/queue.go:597  [split,s1,r15/1:/Table/{59-60},@c42ede5400] unable to split [n1,s1,r15/1:/Table/{59-60}] at key \"/Table/61/0\": key range /Table/61/0-/Table/61/0 outside of bounds of range /Table/59-/Max\nE161213 10:27:26.119610 1477744 storage/queue.go:608  [replicate,s1,r15/1:/Table/{59-60},@c42ede5400] purgatory: 0 of 0 stores with an attribute matching []; likely not enough nodes in cluster\nI161213 10:27:26.119917 1477743 storage/split_queue.go:101  [split,s1,r16/1:/{Table/60-Max},@c430844000] splitting at keys [/Table/61/0 /Table/62/0 /Table/63/0 /Table/64/0 /Table/65/0 /Table/66/0 /Table/67/0 /Table/68/0]\nI161213 10:27:26.120028 1477743 storage/replica_command.go:2389  [split,s1,r16/1:/{Table/60-Max},@c430844000] initiating a split of this range at key /Table/61 [r17]\nE161213 10:27:26.142951 1477743 storage/queue.go:597  [split,s1,r16/1:/Table/6{0-1},@c430844000] unable to split [n1,s1,r16/1:/Table/6{0-1}] at key \"/Table/62/0\": key range /Table/62/0-/Table/62/0 outside of bounds of range /Table/60-/Max\nI161213 10:27:26.146654 1477743 storage/split_queue.go:101  [split,s1,r17/1:/{Table/61-Max},@c42d405900] splitting at keys [/Table/62/0 /Table/63/0 /Table/64/0 /Table/65/0 /Table/66/0 /Table/67/0 /Table/68/0]\nI161213 10:27:26.146755 1477743 storage/replica_command.go:2389  [split,s1,r17/1:/{Table/61-Max},@c42d405900] initiating a split of this range at key /Table/62 [r18]\nE161213 10:27:26.152500 1477744 storage/queue.go:608  [replicate,s1,r16/1:/Table/6{0-1},@c430844000] purgatory: 0 of 0 stores with an attribute matching []; likely not enough nodes in cluster\nE161213 10:27:26.201048 1477744 storage/queue.go:608  [replicate,s1,r17/1:/Table/6{1-2},@c42d405900] purgatory: 0 of 0 stores with an attribute matching []; likely not enough nodes in cluster\nE161213 10:27:26.201456 1477743 storage/queue.go:597  [split,s1,r17/1:/Table/6{1-2},@c42d405900] unable to split [n1,s1,r17/1:/Table/6{1-2}] at key \"/Table/63/0\": key range /Table/63/0-/Table/63/0 outside of bounds of range /Table/61-/Max\nI161213 10:27:26.202820 1477743 storage/split_queue.go:101  [split,s1,r18/1:/{Table/62-Max},@c42eff0500] splitting at keys [/Table/63/0 /Table/64/0 /Table/65/0 /Table/66/0 /Table/67/0 /Table/68/0]\nI161213 10:27:26.202925 1477743 storage/replica_command.go:2389  [split,s1,r18/1:/{Table/62-Max},@c42eff0500] initiating a split of this range at key /Table/63 [r19]\nE161213 10:27:26.229331 1477743 storage/queue.go:597  [split,s1,r18/1:/Table/6{2-3},@c42eff0500] unable to split [n1,s1,r18/1:/Table/6{2-3}] at key \"/Table/64/0\": key range /Table/64/0-/Table/64/0 outside of bounds of range /Table/62-/Max\nE161213 10:27:26.229869 1477744 storage/queue.go:608  [replicate,s1,r18/1:/Table/6{2-3},@c42eff0500] purgatory: 0 of 0 stores with an attribute matching []; likely not enough nodes in cluster\nI161213 10:27:26.238884 1477743 storage/split_queue.go:101  [split,s1,r19/1:/{Table/63-Max},@c430844500] splitting at keys [/Table/64/0 /Table/65/0 /Table/66/0 /Table/67/0 /Table/68/0]\nI161213 10:27:26.239004 1477743 storage/replica_command.go:2389  [split,s1,r19/1:/{Table/63-Max},@c430844500] initiating a split of this range at key /Table/64 [r20]\nE161213 10:27:26.276235 1477744 storage/queue.go:608  [replicate,s1,r19/1:/Table/6{3-4},@c430844500] purgatory: 0 of 0 stores with an attribute matching []; likely not enough nodes in cluster\nE161213 10:27:26.276401 1477743 storage/queue.go:597  [split,s1,r19/1:/Table/6{3-4},@c430844500] unable to split [n1,s1,r19/1:/Table/6{3-4}] at key \"/Table/65/0\": key range /Table/65/0-/Table/65/0 outside of bounds of range /Table/63-/Max\nI161213 10:27:26.277167 1477743 storage/split_queue.go:101  [split,s1,r20/1:/{Table/64-Max},@c42eff0f00] splitting at keys [/Table/65/0 /Table/66/0 /Table/67/0 /Table/68/0]\nI161213 10:27:26.277288 1477743 storage/replica_command.go:2389  [split,s1,r20/1:/{Table/64-Max},@c42eff0f00] initiating a split of this range at key /Table/65 [r21]\nE161213 10:27:26.306532 1477743 storage/queue.go:597  [split,s1,r20/1:/Table/6{4-5},@c42eff0f00] unable to split [n1,s1,r20/1:/Table/6{4-5}] at key \"/Table/66/0\": key range /Table/66/0-/Table/66/0 outside of bounds of range /Table/64-/Max\nE161213 10:27:26.306611 1477744 storage/queue.go:608  [replicate,s1,r20/1:/Table/6{4-5},@c42eff0f00] purgatory: 0 of 0 stores with an attribute matching []; likely not enough nodes in cluster\nI161213 10:27:26.306876 1477743 storage/split_queue.go:101  [split,s1,r21/1:/{Table/65-Max},@c42cf78a00] splitting at keys [/Table/66/0 /Table/67/0 /Table/68/0]\nI161213 10:27:26.306968 1477743 storage/replica_command.go:2389  [split,s1,r21/1:/{Table/65-Max},@c42cf78a00] initiating a split of this range at key /Table/66 [r22]\nE161213 10:27:26.342304 1477744 storage/queue.go:608  [replicate,s1,r21/1:/Table/6{5-6},@c42cf78a00] purgatory: 0 of 0 stores with an attribute matching []; likely not enough nodes in cluster\nE161213 10:27:26.357024 1477743 storage/queue.go:597  [split,s1,r21/1:/Table/6{5-6},@c42cf78a00] unable to split [n1,s1,r21/1:/Table/6{5-6}] at key \"/Table/67/0\": key range /Table/67/0-/Table/67/0 outside of bounds of range /Table/65-/Max\nI161213 10:27:26.363198 1477743 storage/split_queue.go:101  [split,s1,r22/1:/{Table/66-Max},@c42cf78f00] splitting at keys [/Table/67/0 /Table/68/0]\nI161213 10:27:26.363307 1477743 storage/replica_command.go:2389  [split,s1,r22/1:/{Table/66-Max},@c42cf78f00] initiating a split of this range at key /Table/67 [r23]\nE161213 10:27:26.389084 1477744 storage/queue.go:608  [replicate,s1,r22/1:/Table/6{6-7},@c42cf78f00] purgatory: 0 of 0 stores with an attribute matching []; likely not enough nodes in cluster\nE161213 10:27:26.389448 1477743 storage/queue.go:597  [split,s1,r22/1:/Table/6{6-7},@c42cf78f00] unable to split [n1,s1,r22/1:/Table/6{6-7}] at key \"/Table/68/0\": key range /Table/68/0-/Table/68/0 outside of bounds of range /Table/66-/Max\nI161213 10:27:26.389911 1477743 storage/split_queue.go:101  [split,s1,r23/1:/{Table/67-Max},@c42b7aaa00] splitting at keys [/Table/68/0]\nI161213 10:27:26.390206 1477743 storage/replica_command.go:2389  [split,s1,r23/1:/{Table/67-Max},@c42b7aaa00] initiating a split of this range at key /Table/68 [r24]\nE161213 10:27:26.430631 1477744 storage/queue.go:608  [replicate,s1,r23/1:/Table/6{7-8},@c42b7aaa00] purgatory: 0 of 0 stores with an attribute matching []; likely not enough nodes in cluster\nE161213 10:27:26.431714 1477744 storage/queue.go:608  [replicate,s1,r24/1:/{Table/68-Max},@c42f2d0a00] purgatory: 0 of 0 stores with an attribute matching []; likely not enough nodes in cluster\nI161213 10:27:26.498183 1477743 storage/split_queue.go:101  [split,s1,r24/1:/{Table/68-Max},@c42f2d0a00] splitting at keys [/Table/69/0 /Table/70/0 /Table/71/0]\nI161213 10:27:26.499463 1477743 storage/replica_command.go:2389  [split,s1,r24/1:/{Table/68-Max},@c42f2d0a00] initiating a split of this range at key /Table/69 [r25]\nE161213 10:27:26.554880 1477743 storage/queue.go:597  [split,s1,r24/1:/Table/6{8-9},@c42f2d0a00] unable to split [n1,s1,r24/1:/Table/6{8-9}] at key \"/Table/70/0\": key range /Table/70/0-/Table/70/0 outside of bounds of range /Table/68-/Max\nI161213 10:27:26.556962 1477743 storage/split_queue.go:101  [split,s1,r25/1:/{Table/69-Max},@c42cf79400] splitting at keys [/Table/70/0 /Table/71/0]\nI161213 10:27:26.557303 1477743 storage/replica_command.go:2389  [split,s1,r25/1:/{Table/69-Max},@c42cf79400] initiating a split of this range at key /Table/70 [r26]\nE161213 10:27:26.584267 1477743 storage/queue.go:597  [split,s1,r25/1:/Table/{69-70},@c42cf79400] unable to split [n1,s1,r25/1:/Table/{69-70}] at key \"/Table/71/0\": key range /Table/71/0-/Table/71/0 outside of bounds of range /Table/69-/Max\nE161213 10:27:26.585319 1477744 storage/queue.go:608  [replicate,s1,r25/1:/Table/{69-70},@c42cf79400] purgatory: 0 of 0 stores with an attribute matching []; likely not enough nodes in cluster\nI161213 10:27:26.594014 1477743 storage/split_queue.go:101  [split,s1,r26/1:/{Table/70-Max},@c42c977400] splitting at keys [/Table/71/0]\nI161213 10:27:26.594133 1477743 storage/replica_command.go:2389  [split,s1,r26/1:/{Table/70-Max},@c42c977400] initiating a split of this range at key /Table/71 [r27]\nE161213 10:27:26.642031 1477744 storage/queue.go:608  [replicate,s1,r26/1:/Table/7{0-1},@c42c977400] purgatory: 0 of 0 stores with an attribute matching []; likely not enough nodes in cluster\nE161213 10:27:26.642783 1477744 storage/queue.go:608  [replicate,s1,r27/1:/{Table/71-Max},@c432177900] purgatory: 0 of 0 stores with an attribute matching []; likely not enough nodes in cluster\npanic: runtime error: invalid memory address or nil pointer dereference [recovered]\n\tpanic: runtime error: invalid memory address or nil pointer dereference\n[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x692cf1]\n\ngoroutine 1477748 [running]:\npanic(0x18d8440, 0xc420014120)\n\t/usr/local/go/src/runtime/panic.go:500 +0x1a1\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc4360b43f0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0x6e\npanic(0x18d8440, 0xc420014120)\n\t/usr/local/go/src/runtime/panic.go:458 +0x243\ngithub.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).GetLiveness(0x0, 0x1, 0x0, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/node_liveness.go:264 +0x41\ngithub.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).IsLive(0x0, 0xc400000001, 0x0, 0x0, 0x8bb2c97000)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/node_liveness.go:134 +0x37\ngithub.com/cockroachdb/cockroach/pkg/storage.(*consistencyQueue).shouldQueue(0xc42d622f00, 0x2ab39959be78, 0xc427ec59b0, 0x7b, 0xc400000802, 0xc42d994f00, 0xc432131000, 0x1d, 0x20, 0x3, ...)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/consistency_queue.go:77 +0xc6\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).MaybeAdd(0xc42b675b30, 0xc42d994f00, 0x7b, 0x802)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:358 +0x24b\ngithub.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).waitAndProcess(0xc42dd57680, 0x2ab3995990e0, 0xc420014578, 0xecfe1c40d, 0x272ee00a, 0x2b288c0, 0xc42fa30fc0, 0xc4360b43f0, 0xc42d994f00, 0xecfe1c40d)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:221 +0x324\ngithub.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1.1(0xc42d994f00, 0xc435f52000)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:269 +0x8e\ngithub.com/cockroachdb/cockroach/pkg/storage.(*storeReplicaVisitor).Visit(0xc435f52000, 0xc432ee4eb0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:325 +0x303\ngithub.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:271 +0x2ca\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc42d837e80)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1 [chan receive]:\ntesting.(*T).Run(0xc420164480, 0x1a84920, 0x1a, 0x1c779a0, 0xc420289a01)\n\t/usr/local/go/src/testing/testing.go:647 +0x316\ntesting.RunTests.func1(0xc420164480)\n\t/usr/local/go/src/testing/testing.go:793 +0x6d\ntesting.tRunner(0xc420164480, 0xc420309bc0)\n\t/usr/local/go/src/testing/testing.go:610 +0x81\ntesting.RunTests(0x1c79460, 0x2649ce0, 0x168, 0x168, 0xc4202f2bf0)\n\t/usr/local/go/src/testing/testing.go:799 +0x2f5\ntesting.(*M).Run(0xc420309ee8, 0xc420289d30)\n\t/usr/local/go/src/testing/testing.go:743 +0x85\ngithub.com/cockroachdb/cockroach/pkg/storage_test.TestMain(0xc420309ee8)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/main_test.go:57 +0x1ff\nmain.main()\n\tgithub.com/cockroachdb/cockroach/pkg/storage/_test/_testmain.go:784 +0xc6\n\ngoroutine 17 [syscall, locked to thread]:\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1\n\ngoroutine 20 [chan receive]:\ngithub.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).flushDaemon(0x2b29e20)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1016 +0x77\ncreated by github.com/cockroachdb/cockroach/pkg/util/log.init.1\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:581 +0x95\n\ngoroutine 1477326 [select]:\ngithub.com/cockroachdb/cockroach/pkg/util/retry.(*Retry).Next(0xc426ea4c30, 0xa06395)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:128 +0x1a5\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).Send.func2(0xc426ea4c30, 0x7b)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:2441 +0x8d\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).Send(0xc423ff4380, 0x2ab39959be78, 0xc438913920, 0x7b, 0x7c0, 0x100000001, 0x1, 0x1, 0x0, 0x0, ...)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:2448 +0x5e5\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Stores).Send(0xc42ebb73e0, 0x2ab39959be78, 0xc438913890, 0x0, 0x0, 0x100000001, 0x1, 0x1, 0x0, 0x0, ...)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/stores.go:187 +0x202\ngithub.com/cockroachdb/cockroach/pkg/kv.(*senderTransport).SendNext(0xc4224a5220, 0xc4263e5c20)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:309 +0x1f8\ngithub.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc42a8fb950, 0x2ab39959be00, 0xc4286240c0, 0x1dcd6500, 0xc431317470, 0xc42a8fb988, 0x1, 0xc4295802c0, 0x1, 0x1, ...)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1142 +0x35d\ngithub.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc42a8fb950, 0x2ab39959be00, 0xc4286240c0, 0x1, 0xc4295802c0, 0x1, 0x1, 0x0, 0x0, 0x0, ...)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:410 +0x372\ngithub.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc42a8fb950, 0x2ab39959be78, 0xc438913770, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:479 +0x16e\ngithub.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc42a8fb950, 0x2ab39959be78, 0xc438913770, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:931 +0x2c7\ngithub.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc42a8fb950, 0x2ab39959be78, 0xc438913770, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:810 +0x41f\ngithub.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc42a8fb950, 0x2ab39959be78, 0xc438913770, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:625 +0x2d8\ngithub.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).Send(0xc42dd56fa0, 0x2ab39959be78, 0xc438913770, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:418 +0x7e9\ngithub.com/cockroachdb/cockroach/pkg/internal/client.(*DB).send(0xc435ee59a0, 0x2ab3995990e0, 0xc420014590, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:513 +0xbf\ngithub.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run.func1(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:447 +0x6e\ngithub.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0xc429580160, 0xc436071800, 0x0, 0xc429580120)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:418 +0x168\ngithub.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run(0xc435ee59a0, 0x2ab3995990e0, 0xc420014590, 0xc436071800, 0xc429580120, 0x7f)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:449 +0xac\ngithub.com/cockroachdb/cockroach/pkg/internal/client.(*DB).scan(0xc435ee59a0, 0x2ab3995990e0, 0xc420014590, 0x199c6c0, 0xc429580100, 0x199c6c0, 0xc429580120, 0x0, 0x29580100, 0xc430dc5978, ...)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:307 +0x105\ngithub.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Scan(0xc435ee59a0, 0x2ab3995990e0, 0xc420014590, 0x199c6c0, 0xc429580100, 0x199c6c0, 0xc429580120, 0x0, 0x1c70afe, 0xc41fbcd947, ...)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:318 +0x9a\ngithub.com/cockroachdb/cockroach/pkg/storage_test.TestStoreRangeSystemSplits.func2.1(0xecfe1c40e, 0xc424a861f1)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_split_test.go:867 +0x15e\ngithub.com/cockroachdb/cockroach/pkg/util.RetryForDuration(0xa7a358200, 0xc430dc5c90, 0xc430dc5bc8, 0x270)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/retry.go:31 +0xcd\ngithub.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoonDepth(0x1, 0x2671c00, 0xc42e896840, 0xc430dc5c90)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:41 +0x41\ngithub.com/cockroachdb/cockroach/pkg/storage_test.TestStoreRangeSystemSplits.func2(0x16)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_split_test.go:879 +0x67d\ngithub.com/cockroachdb/cockroach/pkg/storage_test.TestStoreRangeSystemSplits(0xc42e896840)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_split_test.go:897 +0x448\ntesting.tRunner(0xc42e896840, 0x1c779a0)\n\t/usr/local/go/src/testing/testing.go:610 +0x81\ncreated by testing.(*T).Run\n\t/usr/local/go/src/testing/testing.go:646 +0x2ec\n\ngoroutine 1477611 [select]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).raftTickLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3415 +0x334\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc4325acd30)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1477608 [runnable]:\nsync.runtime_notifyListWait(0xc430986990, 0xc400000196)\n\t/usr/local/go/src/runtime/sema.go:267 +0x122\nsync.(*Cond).Wait(0xc430986980)\n\t/usr/local/go/src/sync/cond.go:57 +0x80\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc431a5d1e0, 0xc4360b43f0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:212 +0xb2\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc42d836920)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1477744 [select]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:474 +0x383\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc42d837e00)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1477747 [select]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:474 +0x383\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc42d837e60)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1477336 [chan receive]:\ngithub.com/cockroachdb/cockroach/pkg/rpc.NewContext.func1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:137 +0x76\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc42c908f40)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1477745 [select]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:474 +0x383\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc42d837e20)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1477746 [runnable]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:474 +0x383\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc42d837e40)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1477601 [runnable]:\nsync.runtime_notifyListWait(0xc430986990, 0xc40000019f)\n\t/usr/local/go/src/runtime/sema.go:267 +0x122\nsync.(*Cond).Wait(0xc430986980)\n\t/usr/local/go/src/sync/cond.go:57 +0x80\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc431a5d1e0, 0xc4360b43f0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:212 +0xb2\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc42d836840)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1477595 [runnable]:\nsync.runtime_notifyListWait(0xc430986990, 0xc400000193)\n\t/usr/local/go/src/runtime/sema.go:267 +0x122\nsync.(*Cond).Wait(0xc430986980)\n\t/usr/local/go/src/sync/cond.go:57 +0x80\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc431a5d1e0, 0xc4360b43f0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:212 +0xb2\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc42d836780)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1478666 [select]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).maybeAddToPurgatory.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:632 +0xaef\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc430c5a400)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1477600 [runnable]:\nsync.runtime_notifyListWait(0xc430986990, 0xc400000198)\n\t/usr/local/go/src/runtime/sema.go:267 +0x122\nsync.(*Cond).Wait(0xc430986980)\n\t/usr/local/go/src/sync/cond.go:57 +0x80\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc431a5d1e0, 0xc4360b43f0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:212 +0xb2\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc42d836820)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1477607 [runnable]:\nsync.runtime_notifyListWait(0xc430986990, 0xc40000019e)\n\t/usr/local/go/src/runtime/sema.go:267 +0x122\nsync.(*Cond).Wait(0xc430986980)\n\t/usr/local/go/src/sync/cond.go:57 +0x80\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc431a5d1e0, 0xc4360b43f0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:212 +0xb2\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc42d836900)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1477966 [select]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*idAllocator).start.func1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/id_alloc.go:133 +0x7c8\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc4308c0150)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1477658 [select]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1295 +0x39b\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc435f52b10)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1477604 [runnable]:\nsync.runtime_notifyListWait(0xc430986990, 0xc400000195)\n\t/usr/local/go/src/runtime/sema.go:267 +0x122\nsync.(*Cond).Wait(0xc430986980)\n\t/usr/local/go/src/sync/cond.go:57 +0x80\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc431a5d1e0, 0xc4360b43f0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:212 +0xb2\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc42d8368a0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1477598 [runnable]:\nsync.runtime_notifyListWait(0xc430986990, 0xc400000192)\n\t/usr/local/go/src/runtime/sema.go:267 +0x122\nsync.(*Cond).Wait(0xc430986980)\n\t/usr/local/go/src/sync/cond.go:57 +0x80\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc431a5d1e0, 0xc4360b43f0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:212 +0xb2\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc42d8367e0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1477334 [chan receive]:\ngithub.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open.func1(0xc435ee48c0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:373 +0x44\ncreated by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:374 +0x702\n\ngoroutine 1477599 [runnable]:\nsync.runtime_notifyListWait(0xc430986990, 0xc400000197)\n\t/usr/local/go/src/runtime/sema.go:267 +0x122\nsync.(*Cond).Wait(0xc430986980)\n\t/usr/local/go/src/sync/cond.go:57 +0x80\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc431a5d1e0, 0xc4360b43f0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:212 +0xb2\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc42d836800)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1523004 [runnable]:\ngithub.com/cockroachdb/cockroach/vendor/github.com/sasha-s/go-deadlock.lock.func1(0xc42d4f0c70, 0xc4235e3320)\n\t/go/src/github.com/cockroachdb/cockroach/vendor/github.com/sasha-s/go-deadlock/deadlock.go:150\ncreated by github.com/cockroachdb/cockroach/vendor/github.com/sasha-s/go-deadlock.lock\n\t/go/src/github.com/cockroachdb/cockroach/vendor/github.com/sasha-s/go-deadlock/deadlock.go:153 +0xf3\n\ngoroutine 1477603 [runnable]:\ntime.NewTimer(0x45d964b800, 0x1c78ef8)\n\t/usr/local/go/src/time/sleep.go:82 +0x6c\ngithub.com/cockroachdb/cockroach/vendor/github.com/sasha-s/go-deadlock.lock(0xc42d4f0c70, 0x18e0fa0, 0xc4317af060)\n\t/go/src/github.com/cockroachdb/cockroach/vendor/github.com/sasha-s/go-deadlock/deadlock.go:155 +0x103\ngithub.com/cockroachdb/cockroach/vendor/github.com/sasha-s/go-deadlock.(*Mutex).Lock(0xc4317af060)\n\t/go/src/github.com/cockroachdb/cockroach/vendor/github.com/sasha-s/go-deadlock/deadlock.go:55 +0x78\ngithub.com/cockroachdb/cockroach/pkg/util/metric.(*Histogram).RecordValue(0xc4317af020, 0x35e8)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/metric/metric.go:217 +0x3a\ngithub.com/cockroachdb/cockroach/pkg/storage.newReplica.func4(0x35e8)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:603 +0x46\ngithub.com/cockroachdb/cockroach/pkg/util/syncutil.ThresholdLogger.func1(0x35e8)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/timedmutex.go:53 +0x6c\ngithub.com/cockroachdb/cockroach/pkg/util/syncutil.(*TimedMutex).Unlock(0xc4321779f0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/timedmutex.go:97 +0xb6\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Replica).tickRaftMuLocked(0xc432177900, 0x1c77d00, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2728 +0x221\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Replica).tick(0xc432177900, 0xc431317f00, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:2678 +0x85\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).processTick(0xc423ff4380, 0x1b, 0xc42e3adef0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3371 +0xfb\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc431a5d1e0, 0xc4360b43f0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:225 +0x2b6\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc42d836880)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1477596 [runnable]:\nsync.runtime_notifyListWait(0xc430986990, 0xc4000001a0)\n\t/usr/local/go/src/runtime/sema.go:267 +0x122\nsync.(*Cond).Wait(0xc430986980)\n\t/usr/local/go/src/sync/cond.go:57 +0x80\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc431a5d1e0, 0xc4360b43f0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:212 +0xb2\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc42d8367a0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1477656 [select]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).Start.func3()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1209 +0x165\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc42d836f40)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1477605 [runnable]:\nsync.runtime_notifyListWait(0xc430986990, 0xc40000019b)\n\t/usr/local/go/src/runtime/sema.go:267 +0x122\nsync.(*Cond).Wait(0xc430986980)\n\t/usr/local/go/src/sync/cond.go:57 +0x80\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc431a5d1e0, 0xc4360b43f0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:212 +0xb2\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc42d8368c0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1477593 [chan receive]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:171 +0x55\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc42d836740)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1477597 [runnable]:\nsync.runtime_notifyListWait(0xc430986990, 0xc400000194)\n\t/usr/local/go/src/runtime/sema.go:267 +0x122\nsync.(*Cond).Wait(0xc430986980)\n\t/usr/local/go/src/sync/cond.go:57 +0x80\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc431a5d1e0, 0xc4360b43f0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:212 +0xb2\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc42d8367c0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1477375 [select]:\ngithub.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).printStatsLoop(0xc42dd56fa0, 0x2ab3995990e0, 0xc420014578)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:209 +0xbd3\ngithub.com/cockroachdb/cockroach/pkg/kv.NewTxnCoordSender.func1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:193 +0x98\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc43007a020)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1477612 [select]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).startCoalescedHeartbeatsLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3446 +0x167\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc4325acd50)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1477610 [semacquire]:\nsync.runtime_Semacquire(0xc431a5d284)\n\t/usr/local/go/src/runtime/sema.go:47 +0x30\nsync.(*WaitGroup).Wait(0xc431a5d278)\n\t/usr/local/go/src/sync/waitgroup.go:131 +0x97\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait(0xc431a5d1e0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:187 +0x34\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaft.func1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3389 +0x3a\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc4325acd20)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1477609 [runnable]:\nsync.runtime_notifyListWait(0xc430986990, 0xc40000019d)\n\t/usr/local/go/src/runtime/sema.go:267 +0x122\nsync.(*Cond).Wait(0xc430986980)\n\t/usr/local/go/src/sync/cond.go:57 +0x80\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc431a5d1e0, 0xc4360b43f0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:212 +0xb2\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc42d836940)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1477602 [runnable]:\nsync.runtime_notifyListWait(0xc430986990, 0xc40000019a)\n\t/usr/local/go/src/runtime/sema.go:267 +0x122\nsync.(*Cond).Wait(0xc430986980)\n\t/usr/local/go/src/sync/cond.go:57 +0x80\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc431a5d1e0, 0xc4360b43f0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:212 +0xb2\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc42d836860)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1477743 [select]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:474 +0x383\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc42d837de0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1477606 [runnable]:\nsync.runtime_notifyListWait(0xc430986990, 0xc400000199)\n\t/usr/local/go/src/runtime/sema.go:267 +0x122\nsync.(*Cond).Wait(0xc430986980)\n\t/usr/local/go/src/sync/cond.go:57 +0x80\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc431a5d1e0, 0xc4360b43f0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:212 +0xb2\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc42d8368e0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1477742 [select]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:474 +0x383\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc42d837dc0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1477657 [select]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1295 +0x39b\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc435f52a80)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 1477594 [runnable]:\nsync.runtime_notifyListWait(0xc430986990, 0xc40000019c)\n\t/usr/local/go/src/runtime/sema.go:267 +0x122\nsync.(*Cond).Wait(0xc430986980)\n\t/usr/local/go/src/sync/cond.go:57 +0x80\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc431a5d1e0, 0xc4360b43f0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:212 +0xb2\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:181 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4360b43f0, 0xc42d836760)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\n\nERROR: exit status 2\n\nmake: *** [stress] Error 1\n1 runs completed, 1 failures, over 49s\nMakefile:147: recipe for target 'stress' failed\n```"}}
{"actor":{"display_login":"cockroach-teamcity"}}