{"action":"opened","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/5126","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/5126/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/5126/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/5126/events","html_url":"https://github.com/cockroachdb/cockroach/issues/5126","id":140064925,"number":5126,"title":"circleci: failed tests (14340): TestPut","user":{"login":"tschottdorf","id":5076964,"avatar_url":"https://avatars.githubusercontent.com/u/5076964?v=3","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/cockroachdb/cockroach/labels/test-failure","name":"test-failure","color":"e11d21"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-03-11T02:08:33Z","updated_at":"2016-03-11T02:08:33Z","closed_at":null,"body":"The following test appears to have failed:\n\n[#14340](https://circleci.com/gh/cockroachdb/cockroach/14340):\n\n```\n  pprof:     docker exec -it 5ddac /bin/bash -c 'go tool pprof /cockroach <(wget --no-check-certificate -qO- https://$(hostname):8080/debug/pprof/heap)'\n  cockroach: [start --ca-cert=/certs/ca.crt --cert=/certs/node.crt --key=/certs/node.key --host=roach2 --alsologtostderr=INFO --store=/data3.0 --join=roach0:26257 --log-dir=/logs/roach2 --logtostderr=false --alsologtostderr=INFO]\nI160311 02:08:27.933030 acceptance/replication_test.go:50  waiting for first range to have 3 replicas\nI160311 02:08:28.934923 acceptance/replication_test.go:66  found 3 replicas\nI160311 02:08:28.934951 acceptance/replication_test.go:74  found 3 replicas\npanic: test timed out after 5m0s\n\ngoroutine 1399 [running]:\npanic(0xaba3a0, 0xc82037d2e0)\n\t/usr/local/go/src/runtime/panic.go:464 +0x3e6\ntesting.startAlarm.func1()\n\t/usr/local/go/src/testing/testing.go:725 +0x14b\ncreated by time.goFunc\n\t/usr/local/go/src/time/sleep.go:129 +0x3a\n\ngoroutine 1 [chan receive]:\ntesting.RunTests(0xe9a220, 0x133fe60, 0xf, 0xf, 0x29445cd439773001)\n\t/usr/local/go/src/testing/testing.go:583 +0x8d2\ntesting.(*M).Run(0xc820345f08, 0xe99170)\n\t/usr/local/go/src/testing/testing.go:515 +0x81\ngithub.com/cockroachdb/cockroach/acceptance.TestMain(0xc820345f08)\n\t/go/src/github.com/cockroachdb/cockroach/acceptance/main_test.go:48 +0x3e\nmain.main()\n\tgithub.com/cockroachdb/cockroach/acceptance/_test/_testmain.go:80 +0x114\n\ngoroutine 17 [syscall, 4 minutes, locked to thread]:\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1998 +0x1\n\ngoroutine 19 [chan receive]:\ngithub.com/cockroachdb/cockroach/util/log.(*loggingT).flushDaemon(0x135c6e0)\n\t/go/src/github.com/cockroachdb/cockroach/util/log/clog.go:1010 +0x64\ncreated by github.com/cockroachdb/cockroach/util/log.init.1\n\t/go/src/github.com/cockroachdb/cockroach/util/log/clog.go:604 +0x8a\n\ngoroutine 5 [chan receive, 4 minutes]:\ngithub.com/cockroachdb/cockroach/acceptance.TestMain.func1()\n\t/go/src/github.com/cockroachdb/cockroach/acceptance/main_test.go:39 +0xd8\ncreated by github.com/cockroachdb/cockroach/acceptance.TestMain\n\t/go/src/github.com/cockroachdb/cockroach/acceptance/main_test.go:47 +0x30\n\ngoroutine 4 [syscall, 4 minutes]:\nos/signal.signal_recv(0x6)\n\t/usr/local/go/src/runtime/sigqueue.go:116 +0x132\nos/signal.loop()\n\t/usr/local/go/src/os/signal/signal_unix.go:22 +0x18\ncreated by os/signal.init.1\n\t/usr/local/go/src/os/signal/signal_unix.go:28 +0x37\n\ngoroutine 1340 [select]:\ngithub.com/cockroachdb/cockroach/acceptance.testPutInner(0xc820198630, 0x7f172663dca0, 0xc820184680, 0xc8202f6170, 0x9, 0xc8202d58c0, 0x1, 0x1, 0x12a05f200, 0x1bf08eb000)\n\t/go/src/github.com/cockroachdb/cockroach/acceptance/put_test.go:63 +0x72b\ngithub.com/cockroachdb/cockroach/acceptance.runTestOnConfigs.func1(0xc820198630, 0xc82027dec8, 0xe991f8)\n\t/go/src/github.com/cockroachdb/cockroach/acceptance/util_test.go:127 +0xb1\ngithub.com/cockroachdb/cockroach/acceptance.runTestOnConfigs(0xc820198630, 0xe991f8)\n\t/go/src/github.com/cockroachdb/cockroach/acceptance/util_test.go:128 +0x123\ngithub.com/cockroachdb/cockroach/acceptance.TestPut(0xc820198630)\n\t/go/src/github.com/cockroachdb/cockroach/acceptance/put_test.go:33 +0x2d\ntesting.tRunner(0xc820198630, 0x133ff20)\n\t/usr/local/go/src/testing/testing.go:473 +0x98\ncreated by testing.RunTests\n\t/usr/local/go/src/testing/testing.go:582 +0x892\n\ngoroutine 24 [select, 4 minutes, locked to thread]:\nruntime.gopark(0xe9aa20, 0xc820024f28, 0xd2a118, 0x6, 0x18, 0x2)\n\t/usr/local/go/src/runtime/proc.go:262 +0x163\nruntime.selectgoImpl(0xc820024f28, 0x0, 0x18)\n\t/usr/local/go/src/runtime/select.go:392 +0xa67\nruntime.selectgo(0xc820024f28)\n\t/usr/local/go/src/runtime/select.go:215 +0x12\nruntime.ensureSigM.func1()\n\t/usr/local/go/src/runtime/signal1_unix.go:279 +0x358\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1998 +0x1\n\ngoroutine 9 [IO wait, 4 minutes]:\nnet.runtime_pollWait(0x7f17266325d0, 0x72, 0xc8201f5000)\n\t/usr/local/go/src/runtime/netpoll.go:160 +0x60\nnet.(*pollDesc).Wait(0xc8201d5fe0, 0x72, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\nnet.(*pollDesc).WaitRead(0xc8201d5fe0, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36\nnet.(*netFD).Read(0xc8201d5f80, 0xc8201f5000, 0x1000, 0x1000, 0x0, 0x7f17265eb028, 0xc8200840b0)\n\t/usr/local/go/src/net/fd_unix.go:250 +0x23a\nnet.(*conn).Read(0xc82002a2f0, 0xc8201f5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:172 +0xe4\n--\nnet/http.(*persistConn).readLoop(0xc820185790)\n\t/usr/local/go/src/net/http/transport.go:1069 +0x177\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:853 +0x10a6\n\ngoroutine 10 [select, 4 minutes]:\nnet/http.(*persistConn).writeLoop(0xc820185790)\n\t/usr/local/go/src/net/http/transport.go:1273 +0x472\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:854 +0x10cb\n\ngoroutine 453 [select, 1 minutes]:\nnet/http.(*persistConn).writeLoop(0xc82015b520)\n\t/usr/local/go/src/net/http/transport.go:1273 +0x472\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:854 +0x10cb\n\ngoroutine 137 [IO wait, 3 minutes]:\nnet.runtime_pollWait(0x7f17266322d0, 0x72, 0xc820277000)\n\t/usr/local/go/src/runtime/netpoll.go:160 +0x60\nnet.(*pollDesc).Wait(0xc820382370, 0x72, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\nnet.(*pollDesc).WaitRead(0xc820382370, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36\nnet.(*netFD).Read(0xc820382310, 0xc820277000, 0x1000, 0x1000, 0x0, 0x7f17265eb028, 0xc8200840b0)\n\t/usr/local/go/src/net/fd_unix.go:250 +0x23a\nnet.(*conn).Read(0xc82002a1a8, 0xc820277000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:172 +0xe4\n--\nnet/http.(*persistConn).readLoop(0xc82015aa90)\n\t/usr/local/go/src/net/http/transport.go:1069 +0x177\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:853 +0x10a6\n\ngoroutine 379 [chan receive, 3 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc8203e5a20)\n\t/usr/local/go/src/database/sql/sql.go:727 +0x45\ncreated by database/sql.Open\n\t/usr/local/go/src/database/sql/sql.go:493 +0x33f\n\ngoroutine 529 [select, 1 minutes]:\nnet/http.(*persistConn).writeLoop(0xc820184410)\n\t/usr/local/go/src/net/http/transport.go:1273 +0x472\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:854 +0x10cb\n\ngoroutine 375 [chan receive, 3 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc8203e58c0)\n\t/usr/local/go/src/database/sql/sql.go:727 +0x45\ncreated by database/sql.Open\n\t/usr/local/go/src/database/sql/sql.go:493 +0x33f\n\ngoroutine 1053 [IO wait]:\nnet.runtime_pollWait(0x7f1726631b50, 0x72, 0xc8202db000)\n\t/usr/local/go/src/runtime/netpoll.go:160 +0x60\nnet.(*pollDesc).Wait(0xc820398370, 0x72, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\nnet.(*pollDesc).WaitRead(0xc820398370, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36\nnet.(*netFD).Read(0xc820398310, 0xc8202db000, 0x1000, 0x1000, 0x0, 0x7f17265eb028, 0xc8200840b0)\n\t/usr/local/go/src/net/fd_unix.go:250 +0x23a\nnet.(*conn).Read(0xc82002a270, 0xc8202db000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:172 +0xe4\n--\nnet/http.(*persistConn).readLoop(0xc8201852b0)\n\t/usr/local/go/src/net/http/transport.go:1069 +0x177\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:853 +0x10a6\n\ngoroutine 942 [select]:\nnet/http.(*persistConn).writeLoop(0xc820184ea0)\n\t/usr/local/go/src/net/http/transport.go:1273 +0x472\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:854 +0x10cb\n\ngoroutine 452 [IO wait, 1 minutes]:\nnet.runtime_pollWait(0x7f1726632210, 0x72, 0xc8203f7000)\n\t/usr/local/go/src/runtime/netpoll.go:160 +0x60\nnet.(*pollDesc).Wait(0xc8204044c0, 0x72, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\nnet.(*pollDesc).WaitRead(0xc8204044c0, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36\nnet.(*netFD).Read(0xc820404460, 0xc8203f7000, 0x1000, 0x1000, 0x0, 0x7f17265eb028, 0xc8200840b0)\n\t/usr/local/go/src/net/fd_unix.go:250 +0x23a\nnet.(*conn).Read(0xc82008a7b8, 0xc8203f7000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:172 +0xe4\n--\nnet/http.(*persistConn).readLoop(0xc82015b520)\n\t/usr/local/go/src/net/http/transport.go:1069 +0x177\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:853 +0x10a6\n\ngoroutine 732 [select]:\nnet/http.(*persistConn).writeLoop(0xc82015b6c0)\n\t/usr/local/go/src/net/http/transport.go:1273 +0x472\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:854 +0x10cb\n\ngoroutine 234 [select, 3 minutes]:\nnet/http.(*persistConn).writeLoop(0xc82015add0)\n\t/usr/local/go/src/net/http/transport.go:1273 +0x472\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:854 +0x10cb\n\ngoroutine 383 [chan receive, 3 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc8203e5c30)\n\t/usr/local/go/src/database/sql/sql.go:727 +0x45\ncreated by database/sql.Open\n\t/usr/local/go/src/database/sql/sql.go:493 +0x33f\n\ngoroutine 138 [select, 3 minutes]:\nnet/http.(*persistConn).writeLoop(0xc82015aa90)\n\t/usr/local/go/src/net/http/transport.go:1273 +0x472\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:854 +0x10cb\n\ngoroutine 478 [chan receive, 1 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc8201556b0)\n\t/usr/local/go/src/database/sql/sql.go:727 +0x45\ncreated by database/sql.Open\n\t/usr/local/go/src/database/sql/sql.go:493 +0x33f\n\ngoroutine 1054 [select]:\nnet/http.(*persistConn).writeLoop(0xc8201852b0)\n\t/usr/local/go/src/net/http/transport.go:1273 +0x472\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:854 +0x10cb\n\ngoroutine 233 [IO wait, 3 minutes]:\nnet.runtime_pollWait(0x7f1726632450, 0x72, 0xc820302000)\n\t/usr/local/go/src/runtime/netpoll.go:160 +0x60\nnet.(*pollDesc).Wait(0xc8203823e0, 0x72, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\nnet.(*pollDesc).WaitRead(0xc8203823e0, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36\nnet.(*netFD).Read(0xc820382380, 0xc820302000, 0x1000, 0x1000, 0x0, 0x7f17265eb028, 0xc8200840b0)\n\t/usr/local/go/src/net/fd_unix.go:250 +0x23a\nnet.(*conn).Read(0xc82008a038, 0xc820302000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:172 +0xe4\n--\nnet/http.(*persistConn).readLoop(0xc82015add0)\n\t/usr/local/go/src/net/http/transport.go:1069 +0x177\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:853 +0x10a6\n\ngoroutine 319 [select, 1 minutes]:\nnet/http.(*persistConn).writeLoop(0xc82006a0d0)\n\t/usr/local/go/src/net/http/transport.go:1273 +0x472\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:854 +0x10cb\n\ngoroutine 402 [chan receive, 3 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc82013c8f0)\n\t/usr/local/go/src/database/sql/sql.go:727 +0x45\ncreated by database/sql.Open\n\t/usr/local/go/src/database/sql/sql.go:493 +0x33f\n\ngoroutine 318 [IO wait, 1 minutes]:\nnet.runtime_pollWait(0x7f1726632510, 0x72, 0xc820468000)\n\t/usr/local/go/src/runtime/netpoll.go:160 +0x60\nnet.(*pollDesc).Wait(0xc820396060, 0x72, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\nnet.(*pollDesc).WaitRead(0xc820396060, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36\nnet.(*netFD).Read(0xc820396000, 0xc820468000, 0x1000, 0x1000, 0x0, 0x7f17265eb028, 0xc8200840b0)\n\t/usr/local/go/src/net/fd_unix.go:250 +0x23a\nnet.(*conn).Read(0xc82002a000, 0xc820468000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:172 +0xe4\n--\nnet/http.(*persistConn).readLoop(0xc82006a0d0)\n\t/usr/local/go/src/net/http/transport.go:1069 +0x177\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:853 +0x10a6\n\ngoroutine 148 [IO wait, 4 minutes]:\nnet.runtime_pollWait(0x7f1726632390, 0x72, 0xc820288000)\n\t/usr/local/go/src/runtime/netpoll.go:160 +0x60\nnet.(*pollDesc).Wait(0xc8203961b0, 0x72, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\nnet.(*pollDesc).WaitRead(0xc8203961b0, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36\nnet.(*netFD).Read(0xc820396150, 0xc820288000, 0x1000, 0x1000, 0x0, 0x7f17265eb028, 0xc8200840b0)\n\t/usr/local/go/src/net/fd_unix.go:250 +0x23a\nnet.(*conn).Read(0xc82008a040, 0xc820288000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:172 +0xe4\n--\nnet/http.(*persistConn).readLoop(0xc82006a4e0)\n\t/usr/local/go/src/net/http/transport.go:1069 +0x177\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:853 +0x10a6\n\ngoroutine 149 [select, 4 minutes]:\nnet/http.(*persistConn).writeLoop(0xc82006a4e0)\n\t/usr/local/go/src/net/http/transport.go:1273 +0x472\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:854 +0x10cb\n\ngoroutine 528 [IO wait, 1 minutes]:\nnet.runtime_pollWait(0x7f17265fb490, 0x72, 0xc820454000)\n\t/usr/local/go/src/runtime/netpoll.go:160 +0x60\nnet.(*pollDesc).Wait(0xc820383170, 0x72, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\nnet.(*pollDesc).WaitRead(0xc820383170, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36\nnet.(*netFD).Read(0xc820383110, 0xc820454000, 0x1000, 0x1000, 0x0, 0x7f17265eb028, 0xc8200840b0)\n\t/usr/local/go/src/net/fd_unix.go:250 +0x23a\nnet.(*conn).Read(0xc82008a2f0, 0xc820454000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:172 +0xe4\n--\nnet/http.(*persistConn).readLoop(0xc820184410)\n\t/usr/local/go/src/net/http/transport.go:1069 +0x177\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:853 +0x10a6\n\ngoroutine 363 [chan receive, 3 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc8200dfb80)\n\t/usr/local/go/src/database/sql/sql.go:727 +0x45\ncreated by database/sql.Open\n\t/usr/local/go/src/database/sql/sql.go:493 +0x33f\n\ngoroutine 1369 [select]:\nnet/http.(*persistConn).writeLoop(0xc82015bee0)\n\t/usr/local/go/src/net/http/transport.go:1273 +0x472\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:854 +0x10cb\n\ngoroutine 407 [chan receive, 3 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc82013ca50)\n\t/usr/local/go/src/database/sql/sql.go:727 +0x45\ncreated by database/sql.Open\n\t/usr/local/go/src/database/sql/sql.go:493 +0x33f\n\ngoroutine 1349 [select]:\ngithub.com/docker/engine-api/client/transport/cancellable.Do.func3(0x7f17265f6ca8, 0xc8204e4340, 0xc82046c6a0, 0xc820334000)\n\t/go/src/github.com/docker/engine-api/client/transport/cancellable/cancellable.go:78 +0xef\ncreated by github.com/docker/engine-api/client/transport/cancellable.Do\n\t/go/src/github.com/docker/engine-api/client/transport/cancellable/cancellable.go:84 +0x336\n\ngoroutine 597 [select, 1 minutes]:\nnet/http.(*persistConn).writeLoop(0xc8201845b0)\n\t/usr/local/go/src/net/http/transport.go:1273 +0x472\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:854 +0x10cb\n\ngoroutine 496 [chan receive, 1 minutes]:\ndatabase/sql.(*DB).connectionOpener(0xc8204dc9a0)\n\t/usr/local/go/src/database/sql/sql.go:727 +0x45\ncreated by database/sql.Open\n\t/usr/local/go/src/database/sql/sql.go:493 +0x33f\n\ngoroutine 1388 [select]:\ngoogle.golang.org/grpc/transport.(*Stream).Header(0xc82035cee0, 0xc82035cee0, 0x0, 0x0)\n\t/go/src/google.golang.org/grpc/transport/transport.go:221 +0x242\ngoogle.golang.org/grpc.recvResponse(0x7f17265f6e48, 0x137b820, 0x0, 0x0, 0x0, 0x0, 0x7f17265f6e80, 0xc82022ae20, 0x0, 0x0, ...)\n\t/go/src/google.golang.org/grpc/call.go:54 +0x4e\ngoogle.golang.org/grpc.Invoke(0x7f1726631420, 0xc8200856a0, 0xde00e0, 0x21, 0xcfa6a0, 0xc820338660, 0xcdacc0, 0xc820490540, 0xc8202ba000, 0x0, ...)\n\t/go/src/google.golang.org/grpc/call.go:177 +0xae6\ngithub.com/cockroachdb/cockroach/roachpb.(*externalClient).Batch(0xc82002a3f0, 0x7f1726631420, 0xc8200856a0, 0xc820338660, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/roachpb/api.pb.go:1243 +0xec\ngithub.com/cockroachdb/cockroach/client.(*sender).Send(0xc82022ae40, 0x7f1726631420, 0xc8200856a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/go/src/github.com/cockroachdb/cockroach/client/rpc_sender.go:65 +0x32f\n--\ngithub.com/cockroachdb/cockroach/acceptance.testPutInner.func1(0xece741fa1, 0xc837d9a9c0, 0x135c3c0, 0xc8203c2fe0, 0xc8202eda90, 0xc8203d3b60)\n\t/go/src/github.com/cockroachdb/cockroach/acceptance/put_test.go:52 +0x339\ncreated by github.com/cockroachdb/cockroach/acceptance.testPutInner\n\t/go/src/github.com/cockroachdb/cockroach/acceptance/put_test.go:58 +0x27f\n\ngoroutine 731 [IO wait]:\nnet.runtime_pollWait(0x7f1726631c10, 0x72, 0xc820442000)\n\t/usr/local/go/src/runtime/netpoll.go:160 +0x60\nnet.(*pollDesc).Wait(0xc820404300, 0x72, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\nnet.(*pollDesc).WaitRead(0xc820404300, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36\nnet.(*netFD).Read(0xc8204042a0, 0xc820442000, 0x1000, 0x1000, 0x0, 0x7f17265eb028, 0xc8200840b0)\n\t/usr/local/go/src/net/fd_unix.go:250 +0x23a\nnet.(*conn).Read(0xc82002a1f8, 0xc820442000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:172 +0xe4\n--\nnet/http.(*persistConn).readLoop(0xc82015b6c0)\n\t/usr/local/go/src/net/http/transport.go:1069 +0x177\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:853 +0x10a6\n\ngoroutine 596 [IO wait, 1 minutes]:\nnet.runtime_pollWait(0x7f1726631790, 0x72, 0xc820371000)\n\t/usr/local/go/src/runtime/netpoll.go:160 +0x60\nnet.(*pollDesc).Wait(0xc820386300, 0x72, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\nnet.(*pollDesc).WaitRead(0xc820386300, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36\nnet.(*netFD).Read(0xc8203862a0, 0xc820371000, 0x1000, 0x1000, 0x0, 0x7f17265eb028, 0xc8200840b0)\n\t/usr/local/go/src/net/fd_unix.go:250 +0x23a\nnet.(*conn).Read(0xc82008a078, 0xc820371000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:172 +0xe4\n--\nnet/http.(*persistConn).readLoop(0xc8201845b0)\n\t/usr/local/go/src/net/http/transport.go:1069 +0x177\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:853 +0x10a6\n\ngoroutine 1344 [select]:\nnet/http.(*persistConn).writeLoop(0xc820185ba0)\n\t/usr/local/go/src/net/http/transport.go:1273 +0x472\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:854 +0x10cb\n\ngoroutine 1397 [select]:\ngoogle.golang.org/grpc/transport.(*http2Client).controller(0xc8202d6780)\n\t/go/src/google.golang.org/grpc/transport/http2_client.go:831 +0x5da\ncreated by google.golang.org/grpc/transport.newHTTP2Client\n\t/go/src/google.golang.org/grpc/transport/http2_client.go:194 +0x153b\n\ngoroutine 1385 [select]:\ngoogle.golang.org/grpc.(*Conn).transportMonitor(0xc8200ebb20)\n\t/go/src/google.golang.org/grpc/clientconn.go:511 +0x1d3\ngoogle.golang.org/grpc.NewConn.func1(0xc8200ebb20)\n\t/go/src/google.golang.org/grpc/clientconn.go:322 +0x1b5\ncreated by google.golang.org/grpc.NewConn\n\t/go/src/google.golang.org/grpc/clientconn.go:323 +0x509\n\ngoroutine 1240 [select]:\nnet/http.(*persistConn).writeLoop(0xc820185a00)\n\t/usr/local/go/src/net/http/transport.go:1273 +0x472\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:854 +0x10cb\n\ngoroutine 1364 [IO wait]:\nnet.runtime_pollWait(0x7f1726631910, 0x72, 0xc8204d5000)\n\t/usr/local/go/src/runtime/netpoll.go:160 +0x60\nnet.(*pollDesc).Wait(0xc8200e53a0, 0x72, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\nnet.(*pollDesc).WaitRead(0xc8200e53a0, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36\nnet.(*netFD).Read(0xc8200e5340, 0xc8204d5000, 0x1000, 0x1000, 0x0, 0x7f17265eb028, 0xc8200840b0)\n\t/usr/local/go/src/net/fd_unix.go:250 +0x23a\nnet.(*conn).Read(0xc82002a290, 0xc8204d5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:172 +0xe4\n--\ngithub.com/cockroachdb/cockroach/acceptance/cluster.(*LocalCluster).monitor(0xc820184680)\n\t/go/src/github.com/cockroachdb/cockroach/acceptance/cluster/localcluster.go:500 +0x218\ncreated by github.com/cockroachdb/cockroach/acceptance/cluster.(*LocalCluster).Start\n\t/go/src/github.com/cockroachdb/cockroach/acceptance/cluster/localcluster.go:539 +0x731\n\ngoroutine 1368 [select]:\nnet/http.(*persistConn).readLoop(0xc82015bee0)\n\t/usr/local/go/src/net/http/transport.go:1178 +0xd52\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:853 +0x10a6\n\ngoroutine 941 [IO wait]:\nnet.runtime_pollWait(0x7f1726631e50, 0x72, 0xc82016c000)\n\t/usr/local/go/src/runtime/netpoll.go:160 +0x60\nnet.(*pollDesc).Wait(0xc820404610, 0x72, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\nnet.(*pollDesc).WaitRead(0xc820404610, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36\nnet.(*netFD).Read(0xc8204045b0, 0xc82016c000, 0x1000, 0x1000, 0x0, 0x7f17265eb028, 0xc8200840b0)\n\t/usr/local/go/src/net/fd_unix.go:250 +0x23a\nnet.(*conn).Read(0xc82008a3e8, 0xc82016c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:172 +0xe4\n--\nnet/http.(*persistConn).readLoop(0xc820184ea0)\n\t/usr/local/go/src/net/http/transport.go:1069 +0x177\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:853 +0x10a6\n\ngoroutine 1361 [chan receive]:\ngithub.com/cockroachdb/cockroach/rpc.NewContext.func1()\n\t/go/src/github.com/cockroachdb/cockroach/rpc/context.go:99 +0x57\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820057570, 0xc8203ab480)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:98 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:99 +0x62\n\ngoroutine 1389 [select]:\ngoogle.golang.org/grpc/transport.(*Stream).Header(0xc820234b60, 0xc820234b60, 0x0, 0x0)\n\t/go/src/google.golang.org/grpc/transport/transport.go:221 +0x242\ngoogle.golang.org/grpc.recvResponse(0x7f17265f6e48, 0x137b820, 0x0, 0x0, 0x0, 0x0, 0x7f17265f6e80, 0xc82022ae20, 0x0, 0x0, ...)\n\t/go/src/google.golang.org/grpc/call.go:54 +0x4e\ngoogle.golang.org/grpc.Invoke(0x7f1726631420, 0xc8200856a0, 0xde00e0, 0x21, 0xcfa6a0, 0xc82036ed80, 0xcdacc0, 0xc820216d40, 0xc8202ba000, 0x0, ...)\n\t/go/src/google.golang.org/grpc/call.go:177 +0xae6\ngithub.com/cockroachdb/cockroach/roachpb.(*externalClient).Batch(0xc82002a3f0, 0x7f1726631420, 0xc8200856a0, 0xc82036ed80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/roachpb/api.pb.go:1243 +0xec\ngithub.com/cockroachdb/cockroach/client.(*sender).Send(0xc82022ae40, 0x7f1726631420, 0xc8200856a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/go/src/github.com/cockroachdb/cockroach/client/rpc_sender.go:65 +0x32f\n--\ngithub.com/cockroachdb/cockroach/acceptance.testPutInner.func1(0xece741fa1, 0xc837d9a9c0, 0x135c3c0, 0xc8203c2fe0, 0xc8202eda90, 0xc8203d3b60)\n\t/go/src/github.com/cockroachdb/cockroach/acceptance/put_test.go:52 +0x339\ncreated by github.com/cockroachdb/cockroach/acceptance.testPutInner\n\t/go/src/github.com/cockroachdb/cockroach/acceptance/put_test.go:58 +0x27f\n\ngoroutine 1239 [IO wait]:\nnet.runtime_pollWait(0x7f1726631cd0, 0x72, 0xc820791000)\n\t/usr/local/go/src/runtime/netpoll.go:160 +0x60\nnet.(*pollDesc).Wait(0xc820398450, 0x72, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\nnet.(*pollDesc).WaitRead(0xc820398450, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36\nnet.(*netFD).Read(0xc8203983f0, 0xc820791000, 0x1000, 0x1000, 0x0, 0x7f17265eb028, 0xc8200840b0)\n\t/usr/local/go/src/net/fd_unix.go:250 +0x23a\nnet.(*conn).Read(0xc82008a430, 0xc820791000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:172 +0xe4\n--\nnet/http.(*persistConn).readLoop(0xc820185a00)\n\t/usr/local/go/src/net/http/transport.go:1069 +0x177\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:853 +0x10a6\n\ngoroutine 1398 [IO wait]:\nnet.runtime_pollWait(0x7f1726631850, 0x72, 0xc8204b8800)\n\t/usr/local/go/src/runtime/netpoll.go:160 +0x60\nnet.(*pollDesc).Wait(0xc820057790, 0x72, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\nnet.(*pollDesc).WaitRead(0xc820057790, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36\nnet.(*netFD).Read(0xc820057730, 0xc8204b8800, 0x400, 0x400, 0x0, 0x7f17265eb028, 0xc8200840b0)\n\t/usr/local/go/src/net/fd_unix.go:250 +0x23a\nnet.(*conn).Read(0xc82008a498, 0xc8204b8800, 0x400, 0x400, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:172 +0xe4\n--\ngoogle.golang.org/grpc/transport.(*http2Client).reader(0xc8202d6780)\n\t/go/src/google.golang.org/grpc/transport/http2_client.go:754 +0xef\ncreated by google.golang.org/grpc/transport.newHTTP2Client\n\t/go/src/google.golang.org/grpc/transport/http2_client.go:200 +0x159a\n\ngoroutine 1387 [select]:\ngoogle.golang.org/grpc/transport.(*Stream).Header(0xc82035ce00, 0xc82035ce00, 0x0, 0x0)\n\t/go/src/google.golang.org/grpc/transport/transport.go:221 +0x242\ngoogle.golang.org/grpc.recvResponse(0x7f17265f6e48, 0x137b820, 0x0, 0x0, 0x0, 0x0, 0x7f17265f6e80, 0xc82022ae20, 0x0, 0x0, ...)\n\t/go/src/google.golang.org/grpc/call.go:54 +0x4e\ngoogle.golang.org/grpc.Invoke(0x7f1726631420, 0xc8200856a0, 0xde00e0, 0x21, 0xcfa6a0, 0xc8203383c0, 0xcdacc0, 0xc820490340, 0xc8202ba000, 0x0, ...)\n\t/go/src/google.golang.org/grpc/call.go:177 +0xae6\ngithub.com/cockroachdb/cockroach/roachpb.(*externalClient).Batch(0xc82002a3f0, 0x7f1726631420, 0xc8200856a0, 0xc8203383c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/roachpb/api.pb.go:1243 +0xec\ngithub.com/cockroachdb/cockroach/client.(*sender).Send(0xc82022ae40, 0x7f1726631420, 0xc8200856a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/go/src/github.com/cockroachdb/cockroach/client/rpc_sender.go:65 +0x32f\n--\ngithub.com/cockroachdb/cockroach/acceptance.testPutInner.func1(0xece741fa1, 0xc837d9a9c0, 0x135c3c0, 0xc8203c2fe0, 0xc8202eda90, 0xc8203d3b60)\n\t/go/src/github.com/cockroachdb/cockroach/acceptance/put_test.go:52 +0x339\ncreated by github.com/cockroachdb/cockroach/acceptance.testPutInner\n\t/go/src/github.com/cockroachdb/cockroach/acceptance/put_test.go:58 +0x27f\n\ngoroutine 1343 [IO wait]:\nnet.runtime_pollWait(0x7f17266319d0, 0x72, 0xc8203ef000)\n\t/usr/local/go/src/runtime/netpoll.go:160 +0x60\nnet.(*pollDesc).Wait(0xc8203988b0, 0x72, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\nnet.(*pollDesc).WaitRead(0xc8203988b0, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36\nnet.(*netFD).Read(0xc820398850, 0xc8203ef000, 0x1000, 0x1000, 0x0, 0x7f17265eb028, 0xc8200840b0)\n\t/usr/local/go/src/net/fd_unix.go:250 +0x23a\nnet.(*conn).Read(0xc82008a460, 0xc8203ef000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:172 +0xe4\n--\nnet/http.(*persistConn).readLoop(0xc820185ba0)\n\t/usr/local/go/src/net/http/transport.go:1069 +0x177\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:853 +0x10a6\n\ngoroutine 1155 [IO wait]:\nnet.runtime_pollWait(0x7f1726631a90, 0x72, 0xc8204a4000)\n\t/usr/local/go/src/runtime/netpoll.go:160 +0x60\nnet.(*pollDesc).Wait(0xc8203983e0, 0x72, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\nnet.(*pollDesc).WaitRead(0xc8203983e0, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36\nnet.(*netFD).Read(0xc820398380, 0xc8204a4000, 0x1000, 0x1000, 0x0, 0x7f17265eb028, 0xc8200840b0)\n\t/usr/local/go/src/net/fd_unix.go:250 +0x23a\nnet.(*conn).Read(0xc82008a400, 0xc8204a4000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:172 +0xe4\n--\nnet/http.(*persistConn).readLoop(0xc820185520)\n\t/usr/local/go/src/net/http/transport.go:1069 +0x177\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:853 +0x10a6\n\ngoroutine 1156 [select]:\nnet/http.(*persistConn).writeLoop(0xc820185520)\n\t/usr/local/go/src/net/http/transport.go:1273 +0x472\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:854 +0x10cb\n\ngoroutine 1386 [select]:\ngithub.com/cockroachdb/cockroach/rpc.(*Context).runHeartbeat(0xc8204138c0, 0xc8202ba000, 0xc8203755a0, 0xf, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/rpc/context.go:201 +0x411\ngithub.com/cockroachdb/cockroach/rpc.(*Context).GRPCDial.func1()\n\t/go/src/github.com/cockroachdb/cockroach/rpc/context.go:154 +0x66\ngithub.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820057570, 0xc8203ce270)\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:98 +0x52\ncreated by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:99 +0x62\nok github.com/cockroachdb/cockroach/acceptance 1337s\n\n```\nPlease assign, take a look and update the issue accordingly."}}