{"action":"closed","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/6341","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/6341/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/6341/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/6341/events","html_url":"https://github.com/cockroachdb/cockroach/issues/6341","id":151366169,"number":6341,"title":"stress: failed test in cockroach/sql/sql.test: TestLeasesOnDeletedTableAreReleasedImmediately","user":{"login":"mberhault","id":5901480,"avatar_url":"https://avatars.githubusercontent.com/u/5901480?v=3","gravatar_id":"","url":"https://api.github.com/users/mberhault","html_url":"https://github.com/mberhault","followers_url":"https://api.github.com/users/mberhault/followers","following_url":"https://api.github.com/users/mberhault/following{/other_user}","gists_url":"https://api.github.com/users/mberhault/gists{/gist_id}","starred_url":"https://api.github.com/users/mberhault/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mberhault/subscriptions","organizations_url":"https://api.github.com/users/mberhault/orgs","repos_url":"https://api.github.com/users/mberhault/repos","events_url":"https://api.github.com/users/mberhault/events{/privacy}","received_events_url":"https://api.github.com/users/mberhault/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/cockroachdb/cockroach/labels/Robot","name":"Robot","color":"5319e7"},{"url":"https://api.github.com/repos/cockroachdb/cockroach/labels/test-failure","name":"test-failure","color":"e11d21"}],"state":"closed","locked":false,"assignee":{"login":"andreimatei","id":377201,"avatar_url":"https://avatars.githubusercontent.com/u/377201?v=3","gravatar_id":"","url":"https://api.github.com/users/andreimatei","html_url":"https://github.com/andreimatei","followers_url":"https://api.github.com/users/andreimatei/followers","following_url":"https://api.github.com/users/andreimatei/following{/other_user}","gists_url":"https://api.github.com/users/andreimatei/gists{/gist_id}","starred_url":"https://api.github.com/users/andreimatei/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andreimatei/subscriptions","organizations_url":"https://api.github.com/users/andreimatei/orgs","repos_url":"https://api.github.com/users/andreimatei/repos","events_url":"https://api.github.com/users/andreimatei/events{/privacy}","received_events_url":"https://api.github.com/users/andreimatei/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/cockroachdb/cockroach/milestones/1","html_url":"https://github.com/cockroachdb/cockroach/milestones/Beta","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/milestones/1/labels","id":778698,"number":1,"title":"Beta","description":"","creator":{"login":"andybons","id":365204,"avatar_url":"https://avatars.githubusercontent.com/u/365204?v=3","gravatar_id":"","url":"https://api.github.com/users/andybons","html_url":"https://github.com/andybons","followers_url":"https://api.github.com/users/andybons/followers","following_url":"https://api.github.com/users/andybons/following{/other_user}","gists_url":"https://api.github.com/users/andybons/gists{/gist_id}","starred_url":"https://api.github.com/users/andybons/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andybons/subscriptions","organizations_url":"https://api.github.com/users/andybons/orgs","repos_url":"https://api.github.com/users/andybons/repos","events_url":"https://api.github.com/users/andybons/events{/privacy}","received_events_url":"https://api.github.com/users/andybons/received_events","type":"User","site_admin":false},"open_issues":9,"closed_issues":677,"state":"closed","created_at":"2014-09-04T16:42:18Z","updated_at":"2016-05-02T16:02:05Z","due_on":null,"closed_at":"2016-04-07T18:26:58Z"},"comments":3,"created_at":"2016-04-27T11:48:21Z","updated_at":"2016-05-02T16:02:05Z","closed_at":"2016-05-02T16:02:05Z","body":"Binary: cockroach/static-tests.tar.gz sha: https://github.com/cockroachdb/cockroach/commits/e23c02052c15dede1efc4ae6b285940814f3dcd5\n\nStress build found a failed test:\n\n```\n=== RUN   TestLeasesOnDeletedTableAreReleasedImmediately\nW160427 06:47:08.354895 gossip/gossip.go:892  not connected to cluster; use --join to specify a connected node\nI160427 06:47:08.355521 storage/engine/rocksdb.go:137  opening in memory rocksdb instance\nW160427 06:47:08.357855 gossip/gossip.go:892  not connected to cluster; use --join to specify a connected node\nI160427 06:47:08.358712 server/node.go:375  store store=0:0 ([]=) not bootstrapped\nI160427 06:47:08.360531 storage/replica_command.go:1439  range 1: new leader lease replica {1 1 1} 1970-01-01 00:00:00 +0000 UTC 406038h47m9.360066646s following replica {0 0 0} 1970-01-01 00:00:00 +0000 UTC 0 [physicalTime=2016-04-27 06:47:08.360438892 +0000 UTC]\nI160427 06:47:08.361061 server/node.go:325  **** cluster {c3434c6a-2874-4b6f-890b-d7e5df153593} has been created\nI160427 06:47:08.361099 server/node.go:326  **** add additional nodes by specifying --join=127.0.0.1:33722\nW160427 06:47:08.361115 gossip/gossip.go:892  not connected to cluster; use --join to specify a connected node\nI160427 06:47:08.361897 server/node.go:388  initialized store store=1:1 ([]=): {Capacity:10365558784 Available:8217833472 RangeCount:0}\nI160427 06:47:08.361919 server/node.go:300  node ID 1 initialized\nI160427 06:47:08.361960 storage/stores.go:287  read 0 node addresses from persistent storage\nI160427 06:47:08.362009 server/node.go:509  connecting to gossip network to verify cluster ID...\nI160427 06:47:08.362477 server/node.go:530  node connected via gossip and verified as part of cluster {\"c3434c6a-2874-4b6f-890b-d7e5df153593\"}\nI160427 06:47:08.362508 server/node.go:353  [node=1] Started node with [[]=] engine(s) and attributes []\nI160427 06:47:08.362531 server/server.go:368  starting https server at 127.0.0.1:41325\nI160427 06:47:08.362546 server/server.go:369  starting grpc/postgres server at 127.0.0.1:33722\nI160427 06:47:08.362882 storage/split_queue.go:100  splitting range=1 [/Min-/Max) at keys [/Table/11/0 /Table/12/0 /Table/13/0 /Table/14/0]\nI160427 06:47:08.363500 server/updates.go:147  No previous updates check time.\nI160427 06:47:08.395256 storage/replica_command.go:1901  initiating a split of range=1 [/Min-/Max) at key /Table/11\nI160427 06:47:08.443738 storage/replica_command.go:1439  range 2: new leader lease replica {1 1 1} 1970-01-01 00:00:00 +0000 UTC 406038h47m9.49351341s following replica {0 0 0} 1970-01-01 00:00:00 +0000 UTC 0 [physicalTime=2016-04-27 06:47:08.443695721 +0000 UTC]\nI160427 06:47:08.443882 storage/replica_command.go:1901  initiating a split of range=2 [/Table/11-/Max) at key /Table/12\nI160427 06:47:08.447768 storage/replica_command.go:1439  range 3: new leader lease replica {1 1 1} 1970-01-01 00:00:00 +0000 UTC 406038h47m9.497612887s following replica {0 0 0} 1970-01-01 00:00:00 +0000 UTC 0 [physicalTime=2016-04-27 06:47:08.447742439 +0000 UTC]\nI160427 06:47:08.447850 storage/replica_command.go:1901  initiating a split of range=3 [/Table/12-/Max) at key /Table/13\nI160427 06:47:08.470911 storage/replica_command.go:1439  range 4: new leader lease replica {1 1 1} 1970-01-01 00:00:00 +0000 UTC 406038h47m9.520732163s following replica {0 0 0} 1970-01-01 00:00:00 +0000 UTC 0 [physicalTime=2016-04-27 06:47:08.4708801 +0000 UTC]\nI160427 06:47:08.470991 storage/replica_command.go:1901  initiating a split of range=4 [/Table/13-/Max) at key /Table/14\nI160427 06:47:08.541679 storage/replica_command.go:1439  range 5: new leader lease replica {1 1 1} 1970-01-01 00:00:00 +0000 UTC 406038h47m9.591185904s following replica {0 0 0} 1970-01-01 00:00:00 +0000 UTC 0 [physicalTime=2016-04-27 06:47:08.541630863 +0000 UTC]\nI160427 06:47:08.541832 storage/split_queue.go:100  splitting range=5 [/Table/14-/Max) at keys [/Table/50/0]\nI160427 06:47:08.542136 storage/replica_command.go:1901  initiating a split of range=5 [/Table/14-/Max) at key /Table/50\nI160427 06:47:08.557936 storage/replica_command.go:1439  range 6: new leader lease replica {1 1 1} 1970-01-01 00:00:00 +0000 UTC 406038h47m9.607717764s following replica {0 0 0} 1970-01-01 00:00:00 +0000 UTC 0 [physicalTime=2016-04-27 06:47:08.557895686 +0000 UTC]\nI160427 06:47:08.558066 storage/split_queue.go:100  splitting range=6 [/Table/50-/Max) at keys [/Table/51/0]\nI160427 06:47:08.558344 storage/replica_command.go:1901  initiating a split of range=6 [/Table/50-/Max) at key /Table/51\npanic: close of closed channel\n\ngoroutine 1327 [running]:\npanic(0x14dbc00, 0xc82060a6f0)\n\t/usr/local/go/src/runtime/panic.go:464 +0x3e6\ngithub.com/cockroachdb/cockroach/sql_test.TestLeasesOnDeletedTableAreReleasedImmediately.func1(0xc820dff800, 0x17, 0x20)\n\t/go/src/github.com/cockroachdb/cockroach/sql/lease_test.go:494 +0xae\ngithub.com/cockroachdb/cockroach/sql.(*LeaseManager).RefreshLeases.func1()\n\t/go/src/github.com/cockroachdb/cockroach/sql/lease.go:753 +0x452\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc820510540)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1 [chan receive]:\ntesting.RunTests(0x1c3c950, 0x23c92a0, 0x4f, 0x4f, 0xc820323d01)\n\t/usr/local/go/src/testing/testing.go:583 +0x8d2\ntesting.(*M).Run(0xc820040f08, 0x11)\n\t/usr/local/go/src/testing/testing.go:515 +0x81\ngithub.com/cockroachdb/cockroach/sql_test.TestMain(0xc820040f08)\n\t/go/src/github.com/cockroachdb/cockroach/sql/main_test.go:241 +0x26\nmain.main()\n\tgithub.com/cockroachdb/cockroach/sql/_test/_testmain.go:492 +0x114\n\ngoroutine 17 [syscall, locked to thread]:\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1998 +0x1\n\ngoroutine 1404 [runnable]:\ngithub.com/cockroachdb/cockroach/sql_test.TestLeasesOnDeletedTableAreReleasedImmediately(0xc820f18ab0)\n\t/go/src/github.com/cockroachdb/cockroach/sql/lease_test.go:559 +0xab0\ntesting.tRunner(0xc820f18ab0, 0x23c9630)\n\t/usr/local/go/src/testing/testing.go:473 +0x98\ncreated by testing.RunTests\n\t/usr/local/go/src/testing/testing.go:582 +0x892\n\ngoroutine 35 [chan receive]:\ngithub.com/cockroachdb/cockroach/util/log.(*loggingT).flushDaemon(0x26a2640)\n\t/go/src/github.com/cockroachdb/cockroach/util/log/clog.go:1011 +0x64\ncreated by github.com/cockroachdb/cockroach/util/log.init.1\n\t/go/src/github.com/cockroachdb/cockroach/util/log/clog.go:598 +0x8a\n\ngoroutine 1461 [select]:\ngithub.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/storage/queue.go:356 +0x303\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc820522fc0)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1454 [select]:\ngithub.com/cockroachdb/cockroach/server.(*Server).periodicallyCheckForUpdates.func1()\n\t/go/src/github.com/cockroachdb/cockroach/server/updates.go:99 +0x1a2\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc820877510)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1431 [chan receive]:\ngithub.com/cockroachdb/cmux.(*muxListener).Accept(0xc8204fc660, 0x0, 0x0, 0x0, 0x0)\n\t<autogenerated>:9 +0x125\ngoogle.golang.org/grpc.(*Server).Serve(0xc8206ef300, 0x7f70b229f4f8, 0xc8204fc660, 0x0, 0x0)\n\t/go/src/google.golang.org/grpc/server.go:252 +0x1b5\ngithub.com/cockroachdb/cockroach/server.(*Server).Start.func5()\n\t/go/src/github.com/cockroachdb/cockroach/server/server.go:311 +0x43\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc8204fcac0)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1439 [select]:\ngithub.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/storage/queue.go:356 +0x303\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc82089af00)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1435 [select]:\ngithub.com/cockroachdb/cockroach/gossip.(*Gossip).manage.func1()\n\t/go/src/github.com/cockroachdb/cockroach/gossip/gossip.go:791 +0x33b\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc8204fcb80)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1407 [select]:\ngithub.com/cockroachdb/cockroach/storage.(*Store).Start.func3()\n\t/go/src/github.com/cockroachdb/cockroach/storage/store.go:811 +0x13d\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc820522520)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1434 [select]:\ngithub.com/cockroachdb/cockroach/gossip.(*Gossip).bootstrap.func1()\n\t/go/src/github.com/cockroachdb/cockroach/gossip/gossip.go:754 +0x26b\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc8204fcb60)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1462 [select]:\ngithub.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/storage/queue.go:356 +0x303\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc820522fe0)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1458 [select]:\ngithub.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/storage/queue.go:356 +0x303\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc820522f60)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1309 [IO wait]:\nnet.runtime_pollWait(0x7f70b229f348, 0x72, 0xc8205fc800)\n\t/usr/local/go/src/runtime/netpoll.go:160 +0x60\nnet.(*pollDesc).Wait(0xc820271330, 0x72, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\nnet.(*pollDesc).WaitRead(0xc820271330, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36\nnet.(*netFD).Read(0xc8202712d0, 0xc8205fc800, 0x400, 0x400, 0x0, 0x7f70b22d9050, 0xc82000e068)\n\t/usr/local/go/src/net/fd_unix.go:250 +0x23a\nnet.(*conn).Read(0xc82015c0c8, 0xc8205fc800, 0x400, 0x400, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:172 +0xe4\ncrypto/tls.(*block).readFromUntil(0xc82065e150, 0x7f70b22a1670, 0xc82015c0c8, 0x5, 0x0, 0x0)\n\t/usr/local/go/src/crypto/tls/conn.go:460 +0xcc\ncrypto/tls.(*Conn).readRecord(0xc820322000, 0x1c3d217, 0x0, 0x0)\n\t/usr/local/go/src/crypto/tls/conn.go:562 +0x2d1\ncrypto/tls.(*Conn).Read(0xc820322000, 0xc820808000, 0x8000, 0x8000, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/crypto/tls/conn.go:939 +0x167\nbufio.(*Reader).fill(0xc8206875c0)\n\t/usr/local/go/src/bufio/bufio.go:97 +0x1e9\nbufio.(*Reader).Read(0xc8206875c0, 0xc8200973b8, 0x9, 0x9, 0x9, 0x0, 0x0)\n\t/usr/local/go/src/bufio/bufio.go:207 +0x260\nio.ReadAtLeast(0x7f70b225b1d0, 0xc8206875c0, 0xc8200973b8, 0x9, 0x9, 0x9, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/io/io.go:297 +0xe6\nio.ReadFull(0x7f70b225b1d0, 0xc8206875c0, 0xc8200973b8, 0x9, 0x9, 0x7cc13079dbcfbd01, 0x0, 0x0)\n\t/usr/local/go/src/io/io.go:315 +0x62\ngolang.org/x/net/http2.readFrameHeader(0xc8200973b8, 0x9, 0x9, 0x7f70b225b1d0, 0xc8206875c0, 0xb2000000, 0xc800000000, 0x0, 0x0)\n\t/go/src/golang.org/x/net/http2/frame.go:236 +0xa5\ngolang.org/x/net/http2.(*Framer).ReadFrame(0xc820097380, 0x0, 0x0, 0x0, 0x0)\n\t/go/src/golang.org/x/net/http2/frame.go:463 +0x106\ngoogle.golang.org/grpc/transport.(*framer).readFrame(0xc820da17a0, 0x0, 0x0, 0x0, 0x0)\n\t/go/src/google.golang.org/grpc/transport/http_util.go:406 +0x3d\ngoogle.golang.org/grpc/transport.(*http2Client).reader(0xc8206401e0)\n\t/go/src/google.golang.org/grpc/transport/http2_client.go:757 +0x109\ncreated by google.golang.org/grpc/transport.newHTTP2Client\n\t/go/src/google.golang.org/grpc/transport/http2_client.go:200 +0x159a\n\ngoroutine 1422 [IO wait]:\nnet.runtime_pollWait(0x7f70b229ee08, 0x72, 0xc8205fc400)\n\t/usr/local/go/src/runtime/netpoll.go:160 +0x60\nnet.(*pollDesc).Wait(0xc82058df70, 0x72, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\nnet.(*pollDesc).WaitRead(0xc82058df70, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36\nnet.(*netFD).Read(0xc82058df10, 0xc8205fc400, 0x400, 0x400, 0x0, 0x7f70b22d9050, 0xc82000e068)\n\t/usr/local/go/src/net/fd_unix.go:250 +0x23a\nnet.(*conn).Read(0xc820034490, 0xc8205fc400, 0x400, 0x400, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:172 +0xe4\ngithub.com/cockroachdb/cmux.(*MuxConn).Read(0xc82051d290, 0xc8205fc400, 0x400, 0x400, 0x0, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cmux/cmux.go:204 +0x117\ncrypto/tls.(*block).readFromUntil(0xc82079e330, 0x7f70b22e7018, 0xc82051d290, 0x5, 0x0, 0x0)\n\t/usr/local/go/src/crypto/tls/conn.go:460 +0xcc\ncrypto/tls.(*Conn).readRecord(0xc820d84000, 0x1c3d217, 0x0, 0x0)\n\t/usr/local/go/src/crypto/tls/conn.go:562 +0x2d1\ncrypto/tls.(*Conn).Read(0xc820d84000, 0xc8208df000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/crypto/tls/conn.go:939 +0x167\nbufio.(*Reader).fill(0xc8202275c0)\n\t/usr/local/go/src/bufio/bufio.go:97 +0x1e9\nbufio.(*Reader).ReadByte(0xc8202275c0, 0x6, 0x0, 0x0)\n\t/usr/local/go/src/bufio/bufio.go:229 +0x7a\ngithub.com/cockroachdb/cockroach/sql/pgwire.(*readBuffer).readTypedMsg(0xc820d9f828, 0x7f70b222d518, 0xc8202275c0, 0x0, 0x0, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/sql/pgwire/encoding.go:93 +0x3a\ngithub.com/cockroachdb/cockroach/sql/pgwire.(*v3Conn).serve(0xc820d9f800, 0xc8207eec60, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/sql/pgwire/v3.go:246 +0x9a5\ngithub.com/cockroachdb/cockroach/sql/pgwire.(*Server).ServeConn(0xc8206a76d8, 0x7f70b225b0f8, 0xc820d84000, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/sql/pgwire/server.go:157 +0x8df\ngithub.com/cockroachdb/cockroach/server.(*Server).Start.func6.1(0x7f70b22e6fb8, 0xc82051d290)\n\t/go/src/github.com/cockroachdb/cockroach/server/server.go:316 +0x4b\ngithub.com/cockroachdb/cockroach/util.ServeHandler.func3.1(0xc8206ef480, 0x7f70b22e6fb8, 0xc82051d290, 0xc82067e020)\n\t/go/src/github.com/cockroachdb/cockroach/util/net.go:125 +0x5a\ncreated by github.com/cockroachdb/cockroach/util.ServeHandler.func3\n\t/go/src/github.com/cockroachdb/cockroach/util/net.go:127 +0x31f\n\ngoroutine 1344 [select]:\ngoogle.golang.org/grpc/transport.(*http2Client).controller(0xc82025a000)\n\t/go/src/google.golang.org/grpc/transport/http2_client.go:835 +0x5da\ncreated by google.golang.org/grpc/transport.newHTTP2Client\n\t/go/src/google.golang.org/grpc/transport/http2_client.go:194 +0x153b\n\ngoroutine 1421 [chan receive]:\ndatabase/sql.(*DB).connectionOpener(0xc820253ce0)\n\t/usr/local/go/src/database/sql/sql.go:727 +0x45\ncreated by database/sql.Open\n\t/usr/local/go/src/database/sql/sql.go:493 +0x33f\n\ngoroutine 1448 [select]:\ngithub.com/cockroachdb/cockroach/server.(*Node).startComputePeriodicMetrics.func1()\n\t/go/src/github.com/cockroachdb/cockroach/server/node.go:578 +0x23b\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc820522d00)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1440 [select]:\ngithub.com/cockroachdb/cockroach/storage.(*replicaScanner).waitAndProcess(0xc8202705b0, 0xeceb2576c, 0x159e89e8, 0x26a1c20, 0xc82061ad40, 0xc8207a2070, 0xc820596000, 0xc820e6c698)\n\t/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:162 +0x568\ngithub.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1.1(0xc820596000, 0xc8207cc0c0)\n\t/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:204 +0x78\ngithub.com/cockroachdb/cockroach/storage.(*storeRangeSet).Visit(0xc8207cc360, 0xc82064b700)\n\t/go/src/github.com/cockroachdb/cockroach/storage/store.go:239 +0x246\ngithub.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:206 +0x21e\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc82089af20)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1463 [select]:\ngithub.com/cockroachdb/cockroach/storage.(*replicaScanner).waitAndProcess(0xc820270230, 0xeceb2576c, 0x159e577c, 0x26a1c20, 0xc82061ad40, 0xc8207a2070, 0xc820596000, 0xc820457698)\n\t/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:162 +0x568\ngithub.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1.1(0xc820596000, 0xc8207cc0c0)\n\t/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:204 +0x78\ngithub.com/cockroachdb/cockroach/storage.(*storeRangeSet).Visit(0xc8207cc1b0, 0xc8206f0e80)\n\t/go/src/github.com/cockroachdb/cockroach/storage/store.go:239 +0x246\ngithub.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:206 +0x21e\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc820523000)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1329 [chan receive]:\ngithub.com/cockroachdb/cockroach/storage/engine.(*RocksDB).Open.func1(0xc8205e04e0)\n\t/go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:164 +0x3a\ncreated by github.com/cockroachdb/cockroach/storage/engine.(*RocksDB).Open\n\t/go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:165 +0x8f4\n\ngoroutine 1514 [runnable]:\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunLimitedAsyncTask.func1(0xc8207a2070, 0x20c88d5, 0x1a, 0x14b, 0xc820226180, 0xc8205e3240)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:215\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunLimitedAsyncTask\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:219 +0x1e2\n\ngoroutine 1476 [IO wait]:\nnet.runtime_pollWait(0x7f70b229f288, 0x72, 0xc8206f8000)\n\t/usr/local/go/src/runtime/netpoll.go:160 +0x60\nnet.(*pollDesc).Wait(0xc820683aa0, 0x72, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\nnet.(*pollDesc).WaitRead(0xc820683aa0, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36\nnet.(*netFD).Read(0xc820683a40, 0xc8206f8000, 0x400, 0x400, 0x0, 0x7f70b22d9050, 0xc82000e068)\n\t/usr/local/go/src/net/fd_unix.go:250 +0x23a\nnet.(*conn).Read(0xc8208162a8, 0xc8206f8000, 0x400, 0x400, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:172 +0xe4\ngithub.com/cockroachdb/cmux.(*MuxConn).Read(0xc8204cebb0, 0xc8206f8000, 0x400, 0x400, 0xd, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cmux/cmux.go:204 +0x117\ncrypto/tls.(*block).readFromUntil(0xc820d8ecc0, 0x7f70b22e7018, 0xc8204cebb0, 0x5, 0x0, 0x0)\n\t/usr/local/go/src/crypto/tls/conn.go:460 +0xcc\ncrypto/tls.(*Conn).readRecord(0xc8207e4f00, 0x1c3d217, 0x0, 0x0)\n\t/usr/local/go/src/crypto/tls/conn.go:562 +0x2d1\ncrypto/tls.(*Conn).Read(0xc8207e4f00, 0xc820d5c000, 0x8000, 0x8000, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/crypto/tls/conn.go:939 +0x167\nbufio.(*Reader).fill(0xc8205f4360)\n\t/usr/local/go/src/bufio/bufio.go:97 +0x1e9\nbufio.(*Reader).Read(0xc8205f4360, 0xc820708038, 0x9, 0x9, 0x9, 0x0, 0x0)\n\t/usr/local/go/src/bufio/bufio.go:207 +0x260\nio.ReadAtLeast(0x7f70b225b1d0, 0xc8205f4360, 0xc820708038, 0x9, 0x9, 0x9, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/io/io.go:297 +0xe6\nio.ReadFull(0x7f70b225b1d0, 0xc8205f4360, 0xc820708038, 0x9, 0x9, 0x7cc13079dbcfbd01, 0x0, 0x0)\n\t/usr/local/go/src/io/io.go:315 +0x62\ngolang.org/x/net/http2.readFrameHeader(0xc820708038, 0x9, 0x9, 0x7f70b225b1d0, 0xc8205f4360, 0xb2000000, 0xc800000000, 0x0, 0x0)\n\t/go/src/golang.org/x/net/http2/frame.go:236 +0xa5\ngolang.org/x/net/http2.(*Framer).ReadFrame(0xc820708000, 0x0, 0x0, 0x0, 0x0)\n\t/go/src/golang.org/x/net/http2/frame.go:463 +0x106\ngoogle.golang.org/grpc/transport.(*framer).readFrame(0xc82065e4b0, 0x0, 0x0, 0x0, 0x0)\n\t/go/src/google.golang.org/grpc/transport/http_util.go:406 +0x3d\ngoogle.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc820f18000, 0xc82065e540)\n\t/go/src/google.golang.org/grpc/transport/http2_server.go:247 +0x646\ngoogle.golang.org/grpc.(*Server).serveStreams(0xc8206ef300, 0x7f70b225b220, 0xc820f18000)\n\t/go/src/google.golang.org/grpc/server.go:325 +0x159\ngoogle.golang.org/grpc.(*Server).serveNewHTTP2Transport(0xc8206ef300, 0x7f70b225b0f8, 0xc8207e4f00, 0x7f70b225b158, 0xc8202671e0)\n\t/go/src/google.golang.org/grpc/server.go:312 +0x49d\ngoogle.golang.org/grpc.(*Server).handleRawConn(0xc8206ef300, 0x7f70b22e6fb8, 0xc8204cebb0)\n\t/go/src/google.golang.org/grpc/server.go:289 +0x4ee\ncreated by google.golang.org/grpc.(*Server).Serve\n\t/go/src/google.golang.org/grpc/server.go:261 +0x372\n\ngoroutine 1449 [select]:\ngithub.com/cockroachdb/cockroach/server.(*Node).startGossip.func1()\n\t/go/src/github.com/cockroachdb/cockroach/server/node.go:545 +0x406\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc820522d20)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1432 [chan receive]:\ngithub.com/cockroachdb/cmux.(*muxListener).Accept(0xc8204fc640, 0x0, 0x0, 0x0, 0x0)\n\t<autogenerated>:9 +0x125\ngithub.com/cockroachdb/cockroach/util.ServeHandler.func3(0x7f70b229f4f8, 0xc8204fc640, 0xc82067e020, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/util/net.go:105 +0x66\ngithub.com/cockroachdb/cockroach/server.(*Server).Start.func6()\n\t/go/src/github.com/cockroachdb/cockroach/server/server.go:319 +0x96\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc820511830)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1408 [select]:\ngithub.com/cockroachdb/cockroach/storage.(*Store).startGossip.func1()\n\t/go/src/github.com/cockroachdb/cockroach/storage/store.go:882 +0x3cf\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc820522540)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1409 [select]:\ngithub.com/cockroachdb/cockroach/storage.(*Store).startGossip.func2()\n\t/go/src/github.com/cockroachdb/cockroach/storage/store.go:901 +0x3cf\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc820522560)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1426 [chan receive]:\ngithub.com/cockroachdb/cockroach/server.(*Server).Start.func1()\n\t/go/src/github.com/cockroachdb/cockroach/server/server.go:264 +0x62\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc8204fc620)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1453 [select]:\ngithub.com/cockroachdb/cockroach/server.(*Node).startWriteSummaries.func1()\n\t/go/src/github.com/cockroachdb/cockroach/server/node.go:612 +0x35c\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc820522e00)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1474 [runnable]:\ngithub.com/cockroachdb/cockroach/storage.(*idAllocator).start.func1()\n\t/go/src/github.com/cockroachdb/cockroach/storage/id_alloc.go:131 +0x910\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc82065d050)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1328 [select]:\ngithub.com/cockroachdb/cockroach/sql.NewExecutor.func1()\n\t/go/src/github.com/cockroachdb/cockroach/sql/executor.go:235 +0x15c\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc820598f00)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1430 [chan receive]:\ngithub.com/cockroachdb/cmux.(*muxListener).Accept(0xc8204fc8a0, 0x0, 0x0, 0x0, 0x0)\n\t<autogenerated>:9 +0x125\ncrypto/tls.(*listener).Accept(0xc8204fc9c0, 0x0, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/crypto/tls/tls.go:52 +0x60\nnet/http.(*Server).Serve(0xc8206ef480, 0x7f70b229f530, 0xc8204fc9c0, 0x0, 0x0)\n\t/usr/local/go/src/net/http/server.go:2117 +0x129\ngithub.com/cockroachdb/cockroach/util.ServeHandler.func2()\n\t/go/src/github.com/cockroachdb/cockroach/util/net.go:90 +0x66\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc82061bb80)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1429 [chan receive]:\ngithub.com/cockroachdb/cmux.(*muxListener).Accept(0xc8204fc860, 0x0, 0x0, 0x0, 0x0)\n\t<autogenerated>:9 +0x125\nnet/http.(*Server).Serve(0xc8206ef400, 0x7f70b229f4f8, 0xc8204fc860, 0x0, 0x0)\n\t/usr/local/go/src/net/http/server.go:2117 +0x129\ngithub.com/cockroachdb/cockroach/util.ServeHandler.func2()\n\t/go/src/github.com/cockroachdb/cockroach/util/net.go:90 +0x66\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc82061bb00)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1522 [runnable]:\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunAsyncTask.func1(0xc8207a2070, 0x20cddb5, 0x13, 0x11e, 0xc820363bf0)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:179\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunAsyncTask\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:182 +0xc9\n\ngoroutine 1343 [select]:\ngoogle.golang.org/grpc/transport.(*http2Server).controller(0xc820f18000)\n\t/go/src/google.golang.org/grpc/transport/http2_server.go:620 +0x5da\ncreated by google.golang.org/grpc/transport.newHTTP2Server\n\t/go/src/google.golang.org/grpc/transport/http2_server.go:134 +0x853\n\ngoroutine 1345 [IO wait]:\nnet.runtime_pollWait(0x7f70b229ef88, 0x72, 0xc82011f800)\n\t/usr/local/go/src/runtime/netpoll.go:160 +0x60\nnet.(*pollDesc).Wait(0xc8207de140, 0x72, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\nnet.(*pollDesc).WaitRead(0xc8207de140, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36\nnet.(*netFD).Read(0xc8207de0e0, 0xc82011f800, 0x400, 0x400, 0x0, 0x7f70b22d9050, 0xc82000e068)\n\t/usr/local/go/src/net/fd_unix.go:250 +0x23a\nnet.(*conn).Read(0xc820862008, 0xc82011f800, 0x400, 0x400, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:172 +0xe4\ncrypto/tls.(*block).readFromUntil(0xc820d8e390, 0x7f70b22a1670, 0xc820862008, 0x5, 0x0, 0x0)\n\t/usr/local/go/src/crypto/tls/conn.go:460 +0xcc\ncrypto/tls.(*Conn).readRecord(0xc820d84300, 0x1c3d217, 0x0, 0x0)\n\t/usr/local/go/src/crypto/tls/conn.go:562 +0x2d1\ncrypto/tls.(*Conn).Read(0xc820d84300, 0xc820e92000, 0x8000, 0x8000, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/crypto/tls/conn.go:939 +0x167\nbufio.(*Reader).fill(0xc8205f4600)\n\t/usr/local/go/src/bufio/bufio.go:97 +0x1e9\nbufio.(*Reader).Read(0xc8205f4600, 0xc8207080f8, 0x9, 0x9, 0xc81ff938c8, 0x0, 0x0)\n\t/usr/local/go/src/bufio/bufio.go:207 +0x260\nio.ReadAtLeast(0x7f70b225b1d0, 0xc8205f4600, 0xc8207080f8, 0x9, 0x9, 0x9, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/io/io.go:297 +0xe6\nio.ReadFull(0x7f70b225b1d0, 0xc8205f4600, 0xc8207080f8, 0x9, 0x9, 0xc82065e728, 0x0, 0x0)\n\t/usr/local/go/src/io/io.go:315 +0x62\ngolang.org/x/net/http2.readFrameHeader(0xc8207080f8, 0x9, 0x9, 0x7f70b225b1d0, 0xc8205f4600, 0x0, 0xc800000000, 0x0, 0x0)\n\t/go/src/golang.org/x/net/http2/frame.go:236 +0xa5\ngolang.org/x/net/http2.(*Framer).ReadFrame(0xc8207080c0, 0x0, 0x0, 0x0, 0x0)\n\t/go/src/golang.org/x/net/http2/frame.go:463 +0x106\ngoogle.golang.org/grpc/transport.(*framer).readFrame(0xc82065e600, 0x0, 0x0, 0x0, 0x0)\n\t/go/src/google.golang.org/grpc/transport/http_util.go:406 +0x3d\ngoogle.golang.org/grpc/transport.(*http2Client).reader(0xc82025a000)\n\t/go/src/google.golang.org/grpc/transport/http2_client.go:757 +0x109\ncreated by google.golang.org/grpc/transport.newHTTP2Client\n\t/go/src/google.golang.org/grpc/transport/http2_client.go:200 +0x159a\n\ngoroutine 1406 [runnable]:\ngithub.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1()\n\t/go/src/github.com/cockroachdb/cockroach/storage/store.go:1909 +0xbf1\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc820876a20)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1452 [select]:\ngithub.com/cockroachdb/cockroach/ts.(*poller).start.func1()\n\t/go/src/github.com/cockroachdb/cockroach/ts/db.go:78 +0x150\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc8208774e0)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1460 [select]:\ngithub.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/storage/queue.go:356 +0x303\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc820522fa0)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1427 [chan receive]:\ngithub.com/cockroachdb/cockroach/server.(*Server).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/server/server.go:285 +0x59\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc8204fc6e0)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1411 [select]:\ngoogle.golang.org/grpc.(*Conn).transportMonitor(0xc8201720e0)\n\t/go/src/google.golang.org/grpc/clientconn.go:510 +0x1d3\ngoogle.golang.org/grpc.NewConn.func1(0xc8201720e0)\n\t/go/src/google.golang.org/grpc/clientconn.go:321 +0x1b5\ncreated by google.golang.org/grpc.NewConn\n\t/go/src/google.golang.org/grpc/clientconn.go:322 +0x4dd\n\ngoroutine 1457 [select]:\ngithub.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/storage/queue.go:356 +0x303\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc820522f40)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1324 [chan receive]:\ngithub.com/cockroachdb/cockroach/rpc.NewContext.func1()\n\t/go/src/github.com/cockroachdb/cockroach/rpc/context.go:105 +0x57\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc8204fc060)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1428 [IO wait]:\nnet.runtime_pollWait(0x7f70b229eec8, 0x72, 0x0)\n\t/usr/local/go/src/runtime/netpoll.go:160 +0x60\nnet.(*pollDesc).Wait(0xc820682840, 0x72, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\nnet.(*pollDesc).WaitRead(0xc820682840, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36\nnet.(*netFD).accept(0xc8206827e0, 0x0, 0x7f70b222ce48, 0xc820522000)\n\t/usr/local/go/src/net/fd_unix.go:426 +0x27c\nnet.(*TCPListener).AcceptTCP(0xc820816120, 0x621210, 0x0, 0x0)\n\t/usr/local/go/src/net/tcpsock_posix.go:254 +0x4d\nnet.(*TCPListener).Accept(0xc820816120, 0x0, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/tcpsock_posix.go:264 +0x3d\ngithub.com/cockroachdb/cmux.(*cMux).Serve(0xc82061bac0, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cmux/cmux.go:110 +0x9d\ngithub.com/cockroachdb/cockroach/server.(*Server).Start.func3()\n\t/go/src/github.com/cockroachdb/cockroach/server/server.go:297 +0x32\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc8204fc8c0)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1311 [chan receive]:\ngithub.com/cockroachdb/cockroach/server.RegisterAdminHandlerFromEndpoint.func1.1(0x7f70b229e130, 0xc82061b900, 0xc82026ad20, 0xc8208edd10, 0xf)\n\t/go/src/github.com/cockroachdb/cockroach/server/admin.pb.gw.go:186 +0x51\ncreated by github.com/cockroachdb/cockroach/server.RegisterAdminHandlerFromEndpoint.func1\n\t/go/src/github.com/cockroachdb/cockroach/server/admin.pb.gw.go:190 +0x205\n\ngoroutine 1433 [chan receive]:\ngithub.com/cockroachdb/cockroach/gossip.(*server).start.func3()\n\t/go/src/github.com/cockroachdb/cockroach/gossip/server.go:298 +0x5c\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc8204fcb40)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1326 [select]:\ngithub.com/cockroachdb/cockroach/kv.(*TxnCoordSender).startStats(0xc8207fb440)\n\t/go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:243 +0xd21\ngithub.com/cockroachdb/cockroach/kv.(*TxnCoordSender).(github.com/cockroachdb/cockroach/kv.startStats)-fm()\n\t/go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:228 +0x20\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc8208ed600)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1307 [select]:\ngoogle.golang.org/grpc/transport.(*http2Server).controller(0xc820090090)\n\t/go/src/google.golang.org/grpc/transport/http2_server.go:620 +0x5da\ncreated by google.golang.org/grpc/transport.newHTTP2Server\n\t/go/src/google.golang.org/grpc/transport/http2_server.go:134 +0x853\n\ngoroutine 1310 [select]:\ngoogle.golang.org/grpc.(*Conn).transportMonitor(0xc8208002a0)\n\t/go/src/google.golang.org/grpc/clientconn.go:510 +0x1d3\ncreated by google.golang.org/grpc.NewConn\n\t/go/src/google.golang.org/grpc/clientconn.go:312 +0x499\n\ngoroutine 1425 [runnable]:\ngithub.com/cockroachdb/cockroach/kv.(*TxnCoordSender).heartbeatLoop(0xc8207fb440, 0x7f70b22a05e8, 0xc82077f350, 0xc44cf13850055185, 0xdb24858e64e0e79f)\n\t/go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:630 +0x5ec\ngithub.com/cockroachdb/cockroach/kv.(*TxnCoordSender).updateState.func2()\n\t/go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:894 +0x6e\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunAsyncTask.func1(0xc8207a2070, 0x20c8dea, 0x16, 0x37f, 0xc82077eea0)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:181 +0x62\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunAsyncTask\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:182 +0xc9\n\ngoroutine 1451 [select]:\ngithub.com/cockroachdb/cockroach/server.(*Server).startSampleEnvironment.func1()\n\t/go/src/github.com/cockroachdb/cockroach/server/server.go:398 +0x15d\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc820522de0)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1459 [select]:\ngithub.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/storage/queue.go:356 +0x303\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc820522f80)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1523 [runnable]:\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunLimitedAsyncTask.func1(0xc8207a2070, 0x20c88d5, 0x1a, 0x14b, 0xc820226180, 0xc8204d8dc0)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:215\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunLimitedAsyncTask\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:219 +0x1e2\n\ngoroutine 1306 [IO wait]:\nnet.runtime_pollWait(0x7f70b229f048, 0x72, 0xc8205fcc00)\n\t/usr/local/go/src/runtime/netpoll.go:160 +0x60\nnet.(*pollDesc).Wait(0xc8202713a0, 0x72, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\nnet.(*pollDesc).WaitRead(0xc8202713a0, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36\nnet.(*netFD).Read(0xc820271340, 0xc8205fcc00, 0x400, 0x400, 0x0, 0x7f70b22d9050, 0xc82000e068)\n\t/usr/local/go/src/net/fd_unix.go:250 +0x23a\nnet.(*conn).Read(0xc82014e698, 0xc8205fcc00, 0x400, 0x400, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:172 +0xe4\ngithub.com/cockroachdb/cmux.(*MuxConn).Read(0xc8200d2370, 0xc8205fcc00, 0x400, 0x400, 0xd, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cmux/cmux.go:204 +0x117\ncrypto/tls.(*block).readFromUntil(0xc82065e1b0, 0x7f70b22e7018, 0xc8200d2370, 0x5, 0x0, 0x0)\n\t/usr/local/go/src/crypto/tls/conn.go:460 +0xcc\ncrypto/tls.(*Conn).readRecord(0xc820322300, 0x1c3d217, 0x0, 0x0)\n\t/usr/local/go/src/crypto/tls/conn.go:562 +0x2d1\ncrypto/tls.(*Conn).Read(0xc820322300, 0xc820fb2000, 0x8000, 0x8000, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/crypto/tls/conn.go:939 +0x167\nbufio.(*Reader).fill(0xc8206863c0)\n\t/usr/local/go/src/bufio/bufio.go:97 +0x1e9\nbufio.(*Reader).Read(0xc8206863c0, 0xc820096038, 0x9, 0x9, 0x9, 0x0, 0x0)\n\t/usr/local/go/src/bufio/bufio.go:207 +0x260\nio.ReadAtLeast(0x7f70b225b1d0, 0xc8206863c0, 0xc820096038, 0x9, 0x9, 0x9, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/io/io.go:297 +0xe6\nio.ReadFull(0x7f70b225b1d0, 0xc8206863c0, 0xc820096038, 0x9, 0x9, 0x7cc13079dbcfbd01, 0x0, 0x0)\n\t/usr/local/go/src/io/io.go:315 +0x62\ngolang.org/x/net/http2.readFrameHeader(0xc820096038, 0x9, 0x9, 0x7f70b225b1d0, 0xc8206863c0, 0xb2000000, 0xc800000000, 0x0, 0x0)\n\t/go/src/golang.org/x/net/http2/frame.go:236 +0xa5\ngolang.org/x/net/http2.(*Framer).ReadFrame(0xc820096000, 0x0, 0x0, 0x0, 0x0)\n\t/go/src/golang.org/x/net/http2/frame.go:463 +0x106\ngoogle.golang.org/grpc/transport.(*framer).readFrame(0xc820da02a0, 0x0, 0x0, 0x0, 0x0)\n\t/go/src/google.golang.org/grpc/transport/http_util.go:406 +0x3d\ngoogle.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc820090090, 0xc820da0330)\n\t/go/src/google.golang.org/grpc/transport/http2_server.go:247 +0x646\ngoogle.golang.org/grpc.(*Server).serveStreams(0xc8206ef300, 0x7f70b225b220, 0xc820090090)\n\t/go/src/google.golang.org/grpc/server.go:325 +0x159\ngoogle.golang.org/grpc.(*Server).serveNewHTTP2Transport(0xc8206ef300, 0x7f70b225b0f8, 0xc820322300, 0x7f70b225b158, 0xc8202aa2c0)\n\t/go/src/google.golang.org/grpc/server.go:312 +0x49d\ngoogle.golang.org/grpc.(*Server).handleRawConn(0xc8206ef300, 0x7f70b22e6fb8, 0xc8200d2370)\n\t/go/src/google.golang.org/grpc/server.go:289 +0x4ee\ncreated by google.golang.org/grpc.(*Server).Serve\n\t/go/src/google.golang.org/grpc/server.go:261 +0x372\n\ngoroutine 1412 [select]:\ngithub.com/cockroachdb/cockroach/rpc.(*Context).runHeartbeat(0xc8208adb90, 0xc8201ac3c0, 0xc820876ce0, 0xf, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/rpc/context.go:225 +0x5e5\ngithub.com/cockroachdb/cockroach/rpc.(*Context).GRPCDial.func1()\n\t/go/src/github.com/cockroachdb/cockroach/rpc/context.go:173 +0x66\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc82077e300)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1325 [select]:\ngithub.com/cockroachdb/cockroach/storage.(*StorePool).start.func1()\n\t/go/src/github.com/cockroachdb/cockroach/storage/store_pool.go:220 +0x23f\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc8204fc1e0)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\ngoroutine 1308 [select]:\ngoogle.golang.org/grpc/transport.(*http2Client).controller(0xc8206401e0)\n\t/go/src/google.golang.org/grpc/transport/http2_client.go:835 +0x5da\ncreated by google.golang.org/grpc/transport.newHTTP2Client\n\t/go/src/google.golang.org/grpc/transport/http2_client.go:194 +0x153b\n\ngoroutine 1455 [IO wait]:\nnet.runtime_pollWait(0x7f70b229f108, 0x72, 0x0)\n\t/usr/local/go/src/runtime/netpoll.go:160 +0x60\nnet.(*pollDesc).Wait(0xc8206827d0, 0x72, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\nnet.(*pollDesc).WaitRead(0xc8206827d0, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36\nnet.(*netFD).accept(0xc820682770, 0x0, 0x7f70b222ce48, 0xc82084ffc0)\n\t/usr/local/go/src/net/fd_unix.go:426 +0x27c\nnet.(*TCPListener).AcceptTCP(0xc820816108, 0xc820e37ed8, 0x0, 0x0)\n\t/usr/local/go/src/net/tcpsock_posix.go:254 +0x4d\nnet.(*TCPListener).Accept(0xc820816108, 0x0, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/tcpsock_posix.go:264 +0x3d\ngithub.com/cockroachdb/cmux.(*cMux).Serve(0xc82061ba80, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cmux/cmux.go:110 +0x9d\ngithub.com/cockroachdb/cockroach/server.(*Server).Start.func9()\n\t/go/src/github.com/cockroachdb/cockroach/server/server.go:375 +0x32\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc8207a2070, 0xc820522e20)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:139 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:140 +0x62\n\n\nERROR: exit status 2\n```\n\nRun Details:\n\n```\n0 runs so far, 0 failures, over 5s\n0 runs so far, 0 failures, over 10s\n0 runs so far, 0 failures, over 15s\n0 runs so far, 0 failures, over 20s\n0 runs so far, 0 failures, over 25s\n0 runs so far, 0 failures, over 30s\n3 runs so far, 0 failures, over 35s\n9 runs so far, 0 failures, over 40s\n15 runs so far, 0 failures, over 45s\n16 runs so far, 0 failures, over 50s\n17 runs completed, 1 failures, over 55s\nFAIL\n```\nPlease assign, take a look and update the issue accordingly."}}