{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/10430","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/10430/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/10430/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/10430/events","html_url":"https://github.com/cockroachdb/cockroach/issues/10430","id":187007499,"number":10430,"title":"teamcity: failed tests on master: acceptance/TestNodeRestart","user":{"login":"cockroach-teamcity","id":20559710,"avatar_url":"https://avatars.githubusercontent.com/u/20559710?v=3","gravatar_id":"","url":"https://api.github.com/users/cockroach-teamcity","html_url":"https://github.com/cockroach-teamcity","followers_url":"https://api.github.com/users/cockroach-teamcity/followers","following_url":"https://api.github.com/users/cockroach-teamcity/following{/other_user}","gists_url":"https://api.github.com/users/cockroach-teamcity/gists{/gist_id}","starred_url":"https://api.github.com/users/cockroach-teamcity/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cockroach-teamcity/subscriptions","organizations_url":"https://api.github.com/users/cockroach-teamcity/orgs","repos_url":"https://api.github.com/users/cockroach-teamcity/repos","events_url":"https://api.github.com/users/cockroach-teamcity/events{/privacy}","received_events_url":"https://api.github.com/users/cockroach-teamcity/received_events","type":"User","site_admin":false},"labels":[{"id":335217860,"url":"https://api.github.com/repos/cockroachdb/cockroach/labels/Robot","name":"Robot","color":"5319e7","default":false},{"id":191587409,"url":"https://api.github.com/repos/cockroachdb/cockroach/labels/test-failure","name":"test-failure","color":"e11d21","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/cockroachdb/cockroach/milestones/4","html_url":"https://github.com/cockroachdb/cockroach/milestones/Q2","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/milestones/4/labels","id":1690471,"number":4,"title":"Q2","description":"","creator":{"login":"petermattis","id":6201335,"avatar_url":"https://avatars.githubusercontent.com/u/6201335?v=3","gravatar_id":"","url":"https://api.github.com/users/petermattis","html_url":"https://github.com/petermattis","followers_url":"https://api.github.com/users/petermattis/followers","following_url":"https://api.github.com/users/petermattis/following{/other_user}","gists_url":"https://api.github.com/users/petermattis/gists{/gist_id}","starred_url":"https://api.github.com/users/petermattis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/petermattis/subscriptions","organizations_url":"https://api.github.com/users/petermattis/orgs","repos_url":"https://api.github.com/users/petermattis/repos","events_url":"https://api.github.com/users/petermattis/events{/privacy}","received_events_url":"https://api.github.com/users/petermattis/received_events","type":"User","site_admin":false},"open_issues":15,"closed_issues":468,"state":"closed","created_at":"2016-04-07T18:25:48Z","updated_at":"2016-11-03T16:13:38Z","due_on":"2016-06-30T04:00:00Z","closed_at":"2016-07-11T13:26:35Z"},"comments":1,"created_at":"2016-11-03T09:14:26Z","updated_at":"2016-11-03T16:13:38Z","closed_at":"2016-11-03T16:13:38Z","body":"The following tests appear to have failed:\n\n[#41144](https://teamcity.cockroachdb.com/viewLog.html?buildId=41144):\n\n```\n--- FAIL: acceptance/TestNodeRestart (578.281s)\nTest ended in panic.\nI161103 09:04:17.403173 195 acceptance/cluster/localcluster.go:291  creating docker network with name: cockroachdb_acceptance-66a01551\nI161103 09:04:17.509330 195 acceptance/cluster/localcluster.go:323  Initializing Cluster AdHoc 3x1:\n{\"name\":\"AdHoc 3x1\",\"nodes\":[{\"count\":3,\"stores\":[{\"count\":1,\"max_ranges\":0}]}],\"duration\":5000000000}\nI161103 09:04:17.514186 195 acceptance/cluster/docker.go:107  ImagePull cockroachdb/builder:20161024-143446 already exists\nI161103 09:04:18.551487 195 acceptance/cluster/localcluster.go:642  creating certs (1024bit) in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/acceptance/.localcluster.certs.191494601\nI161103 09:04:18.623430 195 acceptance/cluster/localcluster.go:426  creating docker container with name: roach-66a01551-0\nI161103 09:04:19.707385 195 acceptance/cluster/localcluster.go:537  *** started roach-66a01551-0 ***\n  ui:        https://127.0.0.1:33085\n  trace:     https://127.0.0.1:33085/debug/requests\n  logs:      /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/TestNodeRestart-66a01551/roach-66a01551-0/cockroach.INFO\n  pprof:     docker exec -it c5866 /bin/bash -c 'go tool pprof /cockroach <(wget --no-check-certificate -qO- https://$(hostname):8080/debug/pprof/heap)'\n  cockroach: [start --ca-cert=/certs/ca.crt --cert=/certs/node.crt --key=/certs/node.key --host=roach-66a01551-0 --alsologtostderr=INFO --verbosity=1 --store=path=/data1.0 --log-dir=/logs/roach-66a01551-0 --logtostderr=false]\n\n  cli-env:   COCKROACH_INSECURE=false COCKROACH_CA_CERT=/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/acceptance/.localcluster.certs.191494601/ca.crt COCKROACH_CERT=/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/acceptance/.localcluster.certs.191494601/node.crt COCKROACH_KEY=/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/acceptance/.localcluster.certs.191494601/node.key COCKROACH_HOST=127.0.0.1 COCKROACH_PORT=33085\nI161103 09:04:19.707467 195 acceptance/cluster/localcluster.go:426  creating docker container with name: roach-66a01551-1\nI161103 09:04:21.273242 195 acceptance/cluster/localcluster.go:537  *** started roach-66a01551-1 ***\n  ui:        https://127.0.0.1:33087\n  trace:     https://127.0.0.1:33087/debug/requests\n  logs:      /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/TestNodeRestart-66a01551/roach-66a01551-1/cockroach.INFO\n  pprof:     docker exec -it 8e5c6 /bin/bash -c 'go tool pprof /cockroach <(wget --no-check-certificate -qO- https://$(hostname):8080/debug/pprof/heap)'\n  cockroach: [start --ca-cert=/certs/ca.crt --cert=/certs/node.crt --key=/certs/node.key --host=roach-66a01551-1 --alsologtostderr=INFO --verbosity=1 --store=path=/data2.0 --join=roach-66a01551-0:26257 --log-dir=/logs/roach-66a01551-1 --logtostderr=false]\n\n  cli-env:   COCKROACH_INSECURE=false COCKROACH_CA_CERT=/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/acceptance/.localcluster.certs.191494601/ca.crt COCKROACH_CERT=/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/acceptance/.localcluster.certs.191494601/node.crt COCKROACH_KEY=/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/acceptance/.localcluster.certs.191494601/node.key COCKROACH_HOST=127.0.0.1 COCKROACH_PORT=33087\nI161103 09:04:21.273314 195 acceptance/cluster/localcluster.go:426  creating docker container with name: roach-66a01551-2\nI161103 09:04:21.917369 195 acceptance/cluster/localcluster.go:537  *** started roach-66a01551-2 ***\n  ui:        https://127.0.0.1:33089\n  trace:     https://127.0.0.1:33089/debug/requests\n  logs:      /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/TestNodeRestart-66a01551/roach-66a01551-2/cockroach.INFO\n  pprof:     docker exec -it 3ab25 /bin/bash -c 'go tool pprof /cockroach <(wget --no-check-certificate -qO- https://$(hostname):8080/debug/pprof/heap)'\n  cockroach: [start --ca-cert=/certs/ca.crt --cert=/certs/node.crt --key=/certs/node.key --host=roach-66a01551-2 --alsologtostderr=INFO --verbosity=1 --store=path=/data3.0 --join=roach-66a01551-0:26257 --log-dir=/logs/roach-66a01551-2 --logtostderr=false]\n\n  cli-env:   COCKROACH_INSECURE=false COCKROACH_CA_CERT=/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/acceptance/.localcluster.certs.191494601/ca.crt COCKROACH_CERT=/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/acceptance/.localcluster.certs.191494601/node.crt COCKROACH_KEY=/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/acceptance/.localcluster.certs.191494601/node.key COCKROACH_HOST=127.0.0.1 COCKROACH_PORT=33089\nI161103 09:04:21.917387 195 acceptance/replication_test.go:55  waiting for first range to have 3 replicas\nI161103 09:04:22.919844 195 util/stop/stopper.go:396  stop has been called, stopping or quiescing all running tasks\nI161103 09:04:23.923006 195 util/stop/stopper.go:396  stop has been called, stopping or quiescing all running tasks\nI161103 09:04:24.926122 195 util/stop/stopper.go:396  stop has been called, stopping or quiescing all running tasks\nI161103 09:04:25.930157 195 util/stop/stopper.go:396  stop has been called, stopping or quiescing all running tasks\nI161103 09:04:26.932650 195 util/stop/stopper.go:396  stop has been called, stopping or quiescing all running tasks\nI161103 09:04:27.935739 195 util/stop/stopper.go:396  stop has been called, stopping or quiescing all running tasks\nI161103 09:04:28.938946 195 util/stop/stopper.go:396  stop has been called, stopping or quiescing all running tasks\npanic: test timed out after 10m0s\n\ngoroutine 289 [running]:\npanic(0x13d3260, 0xc4202082d0)\n\t/usr/local/go/src/runtime/panic.go:500 +0x1a1\ntesting.startAlarm.func1()\n\t/usr/local/go/src/testing/testing.go:918 +0x10b\ncreated by time.goFunc\n\t/usr/local/go/src/time/sleep.go:154 +0x44\n\ngoroutine 1 [chan receive, 9 minutes]:\ntesting.(*T).Run(0xc420154480, 0x15652de, 0xf, 0x16ac2e8, 0xc420295c01)\n\t/usr/local/go/src/testing/testing.go:647 +0x316\ntesting.RunTests.func1(0xc420154480)\n\t/usr/local/go/src/testing/testing.go:793 +0x6d\ntesting.tRunner(0xc420154480, 0xc420041da8)\n\t/usr/local/go/src/testing/testing.go:610 +0x81\ntesting.RunTests(0x16aec08, 0x1e8e2e0, 0x24, 0x24, 0xc420295df0)\n\t/usr/local/go/src/testing/testing.go:799 +0x2f5\ntesting.(*M).Run(0xc420041ef8, 0x16ac380)\n\t/usr/local/go/src/testing/testing.go:743 +0x85\ngithub.com/cockroachdb/cockroach/pkg/acceptance.runTests(0xc420041ef8)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/acceptance/util_test.go:138 +0x48\ngithub.com/cockroachdb/cockroach/pkg/acceptance.TestMain(0xc420041ef8)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/acceptance/main_test.go:38 +0x45\nmain.main()\n\tgithub.com/cockroachdb/cockroach/pkg/acceptance/_test/_testmain.go:122 +0xc6\n\ngoroutine 17 [syscall, 9 minutes, locked to thread]:\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1\n\ngoroutine 20 [chan receive]:\ngithub.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).flushDaemon(0x21f4e80)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1001 +0x77\ncreated by github.com/cockroachdb/cockroach/pkg/util/log.init.1\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:579 +0x95\n\ngoroutine 21 [syscall, 9 minutes]:\nos/signal.signal_recv(0x0)\n\t/usr/local/go/src/runtime/sigqueue.go:116 +0x157\nos/signal.loop()\n\t/usr/local/go/src/os/signal/signal_unix.go:22 +0x22\ncreated by os/signal.init.1\n\t/usr/local/go/src/os/signal/signal_unix.go:28 +0x41\n\ngoroutine 23 [chan receive, 9 minutes]:\ngithub.com/cockroachdb/cockroach/pkg/acceptance.runTests.func1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/acceptance/util_test.go:129 +0xb6\ncreated by github.com/cockroachdb/cockroach/pkg/acceptance.runTests\n\t/go/src/github.com/cockroachdb/cockroach/pkg/acceptance/util_test.go:137 +0x3a\n\ngoroutine 195 [select, 9 minutes]:\ngoogle.golang.org/grpc/transport.(*Stream).Header(0xc42036c3c0, 0x16aea40, 0xc4202cf1e8, 0x1eb03c0)\n\t/go/src/google.golang.org/grpc/transport/transport.go:239 +0x299\ngoogle.golang.org/grpc.recvResponse(0x1eaa340, 0x222df30, 0x0, 0x0, 0x0, 0x0, 0x1e9de80, 0xc42052eee0, 0x0, 0x0, ...)\n\t/go/src/google.golang.org/grpc/call.go:62 +0xad\ngoogle.golang.org/grpc.Invoke(0x7fd67fe8c8c0, 0xc420014558, 0x157686c, 0x21, 0x15413e0, 0xc420366230, 0x14e1780, 0xc4203307e0, 0xc420415f00, 0x0, ...)\n\t/go/src/google.golang.org/grpc/call.go:202 +0x8a1\ngithub.com/cockroachdb/cockroach/pkg/roachpb.(*externalClient).Batch(0xc4203380d0, 0x7fd67fe8c8c0, 0xc420014558, 0xc420366230, 0x0, 0x0, 0x0, 0xc400000000, 0x5d3659, 0x222dec0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:1562 +0xd2\ngithub.com/cockroachdb/cockroach/pkg/internal/client.sender.Send(0x1e97c80, 0xc4203380d0, 0x7fd67fe8c8c0, 0xc420014558, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/rpc_sender.go:44 +0xbf\ngithub.com/cockroachdb/cockroach/pkg/internal/client.(*sender).Send(0xc4202a0c80, 0x7fd67fe8c8c0, 0xc420014558, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t<autogenerated>:4 +0xb0\ngithub.com/cockroachdb/cockroach/pkg/internal/client.(*DB).send(0xc42001d9a0, 0x7fd67fe8c8c0, 0xc420014558, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:511 +0xbf\ngithub.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run.func1(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:445 +0x6e\ngithub.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0xc4202bc260, 0xc4203f4600, 0x0, 0x1eb0e40)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:416 +0x168\ngithub.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run(0xc42001d9a0, 0x7fd67fe8c8c0, 0xc420014558, 0xc4203f4600, 0x8, 0x18)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:447 +0xac\ngithub.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Get(0xc42001d9a0, 0x7fd67fe8c8c0, 0xc420014558, 0x14b9140, 0xc4202bc240, 0xc4202bc240, 0x18, 0x18, 0x1, 0xc4202bc240, ...)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:214 +0x9a\ngithub.com/cockroachdb/cockroach/pkg/internal/client.(*DB).GetProto(0xc42001d9a0, 0x7fd67fe8c8c0, 0xc420014558, 0x14b9140, 0xc4202bc240, 0x1ea43c0, 0xc420330780, 0xc4202cf9e0, 0x60033c)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:222 +0x69\ngithub.com/cockroachdb/cockroach/pkg/acceptance.countRangeReplicas(0xc42001d9a0, 0xc420455b60, 0x0, 0xc42001d901)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/acceptance/replication_test.go:36 +0x157\ngithub.com/cockroachdb/cockroach/pkg/acceptance.checkRangeReplication.func1(0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/acceptance/replication_test.go:71 +0x1f4\ngithub.com/cockroachdb/cockroach/pkg/util.RetryForDuration(0xa7a358200, 0xc4202cfce0, 0x2, 0x158343f)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/testing.go:129 +0xcd\ngithub.com/cockroachdb/cockroach/pkg/util.SucceedsSoonDepth(0x1, 0x1ead780, 0xc420154300, 0xc4202cfce0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/testing.go:115 +0x41\ngithub.com/cockroachdb/cockroach/pkg/util.SucceedsSoon(0x1ead780, 0xc420154300, 0xc4202cfce0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/testing.go:109 +0x48\ngithub.com/cockroachdb/cockroach/pkg/acceptance.checkRangeReplication(0xc420154300, 0x1eb1ee0, 0xc420415800, 0x4a817c800)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/acceptance/replication_test.go:83 +0x2d3\ngithub.com/cockroachdb/cockroach/pkg/acceptance.StartCluster(0xc420154300, 0xc420015760, 0x9, 0xc4201b5de0, 0x1, 0x1, 0x12a05f200, 0x1eb1ee0, 0xc420415800)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/acceptance/util_test.go:345 +0x188\ngithub.com/cockroachdb/cockroach/pkg/acceptance.runTestOnConfigs.func1(0xc420154300, 0xc4202cfec8, 0x16ac3f8)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/acceptance/util_test.go:312 +0x4e\ngithub.com/cockroachdb/cockroach/pkg/acceptance.runTestOnConfigs(0xc420154300, 0x16ac3f8)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/acceptance/util_test.go:315 +0xe2\ngithub.com/cockroachdb/cockroach/pkg/acceptance.TestNodeRestart(0xc420154300)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/acceptance/chaos_test.go:370 +0x37\ntesting.tRunner(0xc420154300, 0x16ac2e8)\n\t/usr/local/go/src/testing/testing.go:610 +0x81\ncreated by testing.(*T).Run\n\t/usr/local/go/src/testing/testing.go:646 +0x2ec\n\ngoroutine 6 [select, 9 minutes, locked to thread]:\nruntime.gopark(0x16af4c0, 0x0, 0x155ba25, 0x6, 0x18, 0x2)\n\t/usr/local/go/src/runtime/proc.go:259 +0x13a\nruntime.selectgoImpl(0xc420030730, 0x0, 0x18)\n\t/usr/local/go/src/runtime/select.go:423 +0x11d9\nruntime.selectgo(0xc420030730)\n\t/usr/local/go/src/runtime/select.go:238 +0x1c\nruntime.ensureSigM.func1()\n\t/usr/local/go/src/runtime/signal1_unix.go:304 +0x2f3\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1\n\ngoroutine 26 [IO wait, 9 minutes]:\nnet.runtime_pollWait(0x7fd67fe8db58, 0x72, 0x4)\n\t/usr/local/go/src/runtime/netpoll.go:160 +0x59\nnet.(*pollDesc).wait(0xc420430370, 0x72, 0xc42003e9d0, 0xc420014068)\n\t/usr/local/go/src/net/fd_poll_runtime.go:73 +0x38\nnet.(*pollDesc).waitRead(0xc420430370, 0x1e9ec80, 0xc420014068)\n\t/usr/local/go/src/net/fd_poll_runtime.go:78 +0x34\nnet.(*netFD).Read(0xc420430310, 0xc420456000, 0x1000, 0x1000, 0x0, 0x1e9ec80, 0xc420014068)\n\t/usr/local/go/src/net/fd_unix.go:243 +0x1a1\nnet.(*conn).Read(0xc420128a00, 0xc420456000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:173 +0x70\nnet/http.(*persistConn).Read(0xc420415900, 0xc420456000, 0x1000, 0x1000, 0x30, 0xc42003eb58, 0x60033c)\n\t/usr/local/go/src/net/http/transport.go:1261 +0x154\nbufio.(*Reader).fill(0xc4202d9b00)\n\t/usr/local/go/src/bufio/bufio.go:97 +0x10c\nbufio.(*Reader).Peek(0xc4202d9b00, 0x1, 0x0, 0x1, 0x0, 0xc420074540, 0x0)\n\t/usr/local/go/src/bufio/bufio.go:129 +0x62\nnet/http.(*persistConn).readLoop(0xc420415900)\n\t/usr/local/go/src/net/http/transport.go:1418 +0x1a1\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:1062 +0x4e9\n\ngoroutine 27 [select, 9 minutes]:\nnet/http.(*persistConn).writeLoop(0xc420415900)\n\t/usr/local/go/src/net/http/transport.go:1646 +0x3bd\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:1063 +0x50e\n\ngoroutine 172 [select, 9 minutes]:\nnet/http.(*persistConn).writeLoop(0xc42034c300)\n\t/usr/local/go/src/net/http/transport.go:1646 +0x3bd\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:1063 +0x50e\n\ngoroutine 65 [select, 9 minutes]:\nnet/http.(*persistConn).writeLoop(0xc4204a8200)\n\t/usr/local/go/src/net/http/transport.go:1646 +0x3bd\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:1063 +0x50e\n\ngoroutine 212 [IO wait, 9 minutes]:\nnet.runtime_pollWait(0x7fd67fe8da98, 0x72, 0xa)\n\t/usr/local/go/src/runtime/netpoll.go:160 +0x59\nnet.(*pollDesc).wait(0xc4202c01b0, 0x72, 0xc420581818, 0xc420014068)\n\t/usr/local/go/src/net/fd_poll_runtime.go:73 +0x38\nnet.(*pollDesc).waitRead(0xc4202c01b0, 0x1e9ec80, 0xc420014068)\n\t/usr/local/go/src/net/fd_poll_runtime.go:78 +0x34\nnet.(*netFD).Read(0xc4202c0150, 0xc42033a000, 0x1000, 0x1000, 0x0, 0x1e9ec80, 0xc420014068)\n\t/usr/local/go/src/net/fd_unix.go:243 +0x1a1\nnet.(*conn).Read(0xc420338020, 0xc42033a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:173 +0x70\nnet/http.(*persistConn).Read(0xc42034c000, 0xc42033a000, 0x1000, 0x1000, 0x9ccb48, 0xc4203b3568, 0x7d)\n\t/usr/local/go/src/net/http/transport.go:1261 +0x154\nbufio.(*Reader).fill(0xc4200dcb40)\n\t/usr/local/go/src/bufio/bufio.go:97 +0x10c\nbufio.(*Reader).ReadSlice(0xc4200dcb40, 0x10a, 0x2, 0x0, 0x199, 0x186, 0x186)\n\t/usr/local/go/src/bufio/bufio.go:330 +0xb5\nnet/http/internal.readChunkLine(0xc4200dcb40, 0x5d3659, 0xc4200dc118, 0x0, 0x1, 0xc420581a88)\n\t/usr/local/go/src/net/http/internal/chunked.go:110 +0x34\nnet/http/internal.(*chunkedReader).beginChunk(0xc4202c6ba0)\n\t/usr/local/go/src/net/http/internal/chunked.go:47 +0x32\nnet/http/internal.(*chunkedReader).Read(0xc4202c6ba0, 0xc42049e001, 0x5ff, 0x5ff, 0x619db0, 0xc420581b70, 0xc420581b78)\n\t/usr/local/go/src/net/http/internal/chunked.go:77 +0x7b\nnet/http.(*body).readLocked(0xc4205017c0, 0xc42049e001, 0x5ff, 0x5ff, 0x2, 0x2, 0x0)\n\t/usr/local/go/src/net/http/transfer.go:651 +0x61\nnet/http.(*body).Read(0xc4205017c0, 0xc42049e001, 0x5ff, 0x5ff, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/http/transfer.go:643 +0xf6\nnet/http.(*bodyEOFSignal).Read(0xc420501800, 0xc42049e001, 0x5ff, 0x5ff, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/http/transport.go:1974 +0x116\nencoding/json.(*Decoder).refill(0xc4203b3520, 0xa, 0x9)\n\t/usr/local/go/src/encoding/json/stream.go:152 +0xfa\nencoding/json.(*Decoder).readValue(0xc4203b3520, 0x0, 0x0, 0x80)\n\t/usr/local/go/src/encoding/json/stream.go:128 +0x198\nencoding/json.(*Decoder).Decode(0xc4203b3520, 0x13a1f60, 0xc42000cc00, 0x0, 0x0)\n\t/usr/local/go/src/encoding/json/stream.go:57 +0x8e\ngithub.com/docker/docker/client.(*Client).Events.func1(0xc4200dc120, 0xc4203e50a0, 0x0, 0x0, 0x0, 0x0, 0xc42023b8f0, 0x7fd67fec72a0, 0xc420501640, 0xc4200dc0c0)\n\t/go/src/github.com/docker/docker/client/events.go:50 +0x2a7\ncreated by github.com/docker/docker/client.(*Client).Events\n\t/go/src/github.com/docker/docker/client/events.go:63 +0xe2\n\ngoroutine 64 [IO wait, 9 minutes]:\nnet.runtime_pollWait(0x7fd67fe8d9d8, 0x72, 0x6)\n\t/usr/local/go/src/runtime/netpoll.go:160 +0x59\nnet.(*pollDesc).wait(0xc4204647d0, 0x72, 0xc4202f89d0, 0xc420014068)\n\t/usr/local/go/src/net/fd_poll_runtime.go:73 +0x38\nnet.(*pollDesc).waitRead(0xc4204647d0, 0x1e9ec80, 0xc420014068)\n\t/usr/local/go/src/net/fd_poll_runtime.go:78 +0x34\nnet.(*netFD).Read(0xc420464770, 0xc420254000, 0x1000, 0x1000, 0x0, 0x1e9ec80, 0xc420014068)\n\t/usr/local/go/src/net/fd_unix.go:243 +0x1a1\nnet.(*conn).Read(0xc420128408, 0xc420254000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:173 +0x70\nnet/http.(*persistConn).Read(0xc4204a8200, 0xc420254000, 0x1000, 0x1000, 0x30, 0xc4202f8b58, 0x60033c)\n\t/usr/local/go/src/net/http/transport.go:1261 +0x154\nbufio.(*Reader).fill(0xc42016afc0)\n\t/usr/local/go/src/bufio/bufio.go:97 +0x10c\nbufio.(*Reader).Peek(0xc42016afc0, 0x1, 0x0, 0x1, 0x0, 0xc4202d8540, 0x0)\n\t/usr/local/go/src/bufio/bufio.go:129 +0x62\nnet/http.(*persistConn).readLoop(0xc4204a8200)\n\t/usr/local/go/src/net/http/transport.go:1418 +0x1a1\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:1062 +0x4e9\n\ngoroutine 130 [select, 9 minutes]:\nnet/http.(*persistConn).writeLoop(0xc42034c100)\n\t/usr/local/go/src/net/http/transport.go:1646 +0x3bd\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:1063 +0x50e\n\ngoroutine 286 [select]:\ngithub.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc4203b4880, 0xc420415f00, 0xc42032d7a0, 0xf, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:291 +0x617\ngithub.com/cockroachdb/cockroach/pkg/rpc.(*Context).GRPCDial.func1.1.1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:226 +0x69\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42017f440, 0xc420125aa0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 171 [IO wait, 9 minutes]:\nnet.runtime_pollWait(0x7fd67fe8d918, 0x72, 0x8)\n\t/usr/local/go/src/runtime/netpoll.go:160 +0x59\nnet.(*pollDesc).wait(0xc420464530, 0x72, 0xc4202fa9d0, 0xc420014068)\n\t/usr/local/go/src/net/fd_poll_runtime.go:73 +0x38\nnet.(*pollDesc).waitRead(0xc420464530, 0x1e9ec80, 0xc420014068)\n\t/usr/local/go/src/net/fd_poll_runtime.go:78 +0x34\nnet.(*netFD).Read(0xc4204644d0, 0xc420550000, 0x1000, 0x1000, 0x0, 0x1e9ec80, 0xc420014068)\n\t/usr/local/go/src/net/fd_unix.go:243 +0x1a1\nnet.(*conn).Read(0xc420032068, 0xc420550000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:173 +0x70\nnet/http.(*persistConn).Read(0xc42034c300, 0xc420550000, 0x1000, 0x1000, 0x30, 0xc4202fab58, 0x60033c)\n\t/usr/local/go/src/net/http/transport.go:1261 +0x154\nbufio.(*Reader).fill(0xc420454900)\n\t/usr/local/go/src/bufio/bufio.go:97 +0x10c\nbufio.(*Reader).Peek(0xc420454900, 0x1, 0x0, 0x1, 0x0, 0xc42016a060, 0x0)\n\t/usr/local/go/src/bufio/bufio.go:129 +0x62\nnet/http.(*persistConn).readLoop(0xc42034c300)\n\t/usr/local/go/src/net/http/transport.go:1418 +0x1a1\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:1062 +0x4e9\n\ngoroutine 113 [IO wait, 9 minutes]:\nnet.runtime_pollWait(0x7fd67fe8d798, 0x72, 0x7)\n\t/usr/local/go/src/runtime/netpoll.go:160 +0x59\nnet.(*pollDesc).wait(0xc4203e53a0, 0x72, 0xc4202f79d0, 0xc420014068)\n\t/usr/local/go/src/net/fd_poll_runtime.go:73 +0x38\nnet.(*pollDesc).waitRead(0xc4203e53a0, 0x1e9ec80, 0xc420014068)\n\t/usr/local/go/src/net/fd_poll_runtime.go:78 +0x34\nnet.(*netFD).Read(0xc4203e5340, 0xc4202fc000, 0x1000, 0x1000, 0x0, 0x1e9ec80, 0xc420014068)\n\t/usr/local/go/src/net/fd_unix.go:243 +0x1a1\nnet.(*conn).Read(0xc420032058, 0xc4202fc000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:173 +0x70\nnet/http.(*persistConn).Read(0xc42034c100, 0xc4202fc000, 0x1000, 0x1000, 0x30, 0xc4202f7b58, 0x60033c)\n\t/usr/local/go/src/net/http/transport.go:1261 +0x154\nbufio.(*Reader).fill(0xc4202d8d20)\n\t/usr/local/go/src/bufio/bufio.go:97 +0x10c\nbufio.(*Reader).Peek(0xc4202d8d20, 0x1, 0x0, 0x1, 0x0, 0xc420454360, 0x0)\n\t/usr/local/go/src/bufio/bufio.go:129 +0x62\nnet/http.(*persistConn).readLoop(0xc42034c100)\n\t/usr/local/go/src/net/http/transport.go:1418 +0x1a1\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:1062 +0x4e9\n\ngoroutine 198 [select, 9 minutes]:\nnet/http.(*persistConn).writeLoop(0xc420415a00)\n\t/usr/local/go/src/net/http/transport.go:1646 +0x3bd\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:1063 +0x50e\n\ngoroutine 296 [chan receive, 9 minutes]:\ngithub.com/cockroachdb/cockroach/pkg/rpc.NewContext.func1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:137 +0x76\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42017f440, 0xc420231d40)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:197 +0x66\n\ngoroutine 214 [select, 9 minutes]:\nnet/http.(*persistConn).readLoop(0xc42034c000)\n\t/usr/local/go/src/net/http/transport.go:1541 +0x9c9\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:1062 +0x4e9\n\ngoroutine 197 [IO wait, 9 minutes]:\nnet.runtime_pollWait(0x7fd67fe8d858, 0x72, 0x9)\n\t/usr/local/go/src/runtime/netpoll.go:160 +0x59\nnet.(*pollDesc).wait(0xc4203e52c0, 0x72, 0xc4205e09d0, 0xc420014068)\n\t/usr/local/go/src/net/fd_poll_runtime.go:73 +0x38\nnet.(*pollDesc).waitRead(0xc4203e52c0, 0x1e9ec80, 0xc420014068)\n\t/usr/local/go/src/net/fd_poll_runtime.go:78 +0x34\nnet.(*netFD).Read(0xc4203e5260, 0xc42028c000, 0x1000, 0x1000, 0x0, 0x1e9ec80, 0xc420014068)\n\t/usr/local/go/src/net/fd_unix.go:243 +0x1a1\nnet.(*conn).Read(0xc420032070, 0xc42028c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:173 +0x70\nnet/http.(*persistConn).Read(0xc420415a00, 0xc42028c000, 0x1000, 0x1000, 0x8a0ec0, 0xc4205e0b58, 0x5c90ad)\n\t/usr/local/go/src/net/http/transport.go:1261 +0x154\nbufio.(*Reader).fill(0xc4200dcba0)\n\t/usr/local/go/src/bufio/bufio.go:97 +0x10c\nbufio.(*Reader).Peek(0xc4200dcba0, 0x1, 0xc4205e0bbd, 0x1, 0x0, 0xc42013bbc0, 0x0)\n\t/usr/local/go/src/bufio/bufio.go:129 +0x62\nnet/http.(*persistConn).readLoop(0xc420415a00)\n\t/usr/local/go/src/net/http/transport.go:1418 +0x1a1\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:1062 +0x4e9\n\ngoroutine 203 [select, 9 minutes]:\ngithub.com/cockroachdb/cockroach/pkg/acceptance/cluster.(*LocalCluster).monitor.func1(0x1)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/acceptance/cluster/localcluster.go:609 +0x4aa\ngithub.com/cockroachdb/cockroach/pkg/acceptance/cluster.(*LocalCluster).monitor(0xc420415800)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/acceptance/cluster/localcluster.go:629 +0x61\ncreated by github.com/cockroachdb/cockroach/pkg/acceptance/cluster.(*LocalCluster).Start\n\t/go/src/github.com/cockroachdb/cockroach/pkg/acceptance/cluster/localcluster.go:653 +0x57e\n\ngoroutine 215 [select, 9 minutes]:\nnet/http.(*persistConn).writeLoop(0xc42034c000)\n\t/usr/local/go/src/net/http/transport.go:1646 +0x3bd\ncreated by net/http.(*Transport).dialConn\n\t/usr/local/go/src/net/http/transport.go:1063 +0x50e\n\ngoroutine 287 [IO wait]:\nnet.runtime_pollWait(0x7fd67fe8d6d8, 0x72, 0xb)\n\t/usr/local/go/src/runtime/netpoll.go:160 +0x59\nnet.(*pollDesc).wait(0xc420370680, 0x72, 0xc4205e1828, 0xc420014068)\n\t/usr/local/go/src/net/fd_poll_runtime.go:73 +0x38\nnet.(*pollDesc).waitRead(0xc420370680, 0x1e9ec80, 0xc420014068)\n\t/usr/local/go/src/net/fd_poll_runtime.go:78 +0x34\nnet.(*netFD).Read(0xc420370620, 0xc420067000, 0x400, 0x400, 0x0, 0x1e9ec80, 0xc420014068)\n\t/usr/local/go/src/net/fd_unix.go:243 +0x1a1\nnet.(*conn).Read(0xc4201284b0, 0xc420067000, 0x400, 0x400, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:173 +0x70\ncrypto/tls.(*block).readFromUntil(0xc420125b90, 0x7fd67fe3d940, 0xc4201284b0, 0x5, 0xc4201284b0, 0xc4204047a0)\n\t/usr/local/go/src/crypto/tls/conn.go:476 +0x91\ncrypto/tls.(*Conn).readRecord(0xc4201c5500, 0x16af517, 0xc4201c5608, 0xc420208280)\n\t/usr/local/go/src/crypto/tls/conn.go:578 +0xc4\ncrypto/tls.(*Conn).Read(0xc4201c5500, 0xc420502000, 0x8000, 0x8000, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/crypto/tls/conn.go:1113 +0x116\nbufio.(*Reader).fill(0xc420075320)\n\t/usr/local/go/src/bufio/bufio.go:97 +0x10c\nbufio.(*Reader).Read(0xc420075320, 0xc4205446f8, 0x9, 0x9, 0xc4205e1ca8, 0x619eb0, 0xc420489640)\n\t/usr/local/go/src/bufio/bufio.go:209 +0x1bc\nio.ReadAtLeast(0x1e97300, 0xc420075320, 0xc4205446f8, 0x9, 0x9, 0x9, 0xc4201db0f0, 0xc4201db0e8, 0x1e9b0c0)\n\t/usr/local/go/src/io/io.go:307 +0xa4\nio.ReadFull(0x1e97300, 0xc420075320, 0xc4205446f8, 0x9, 0x9, 0xc4205e1eb0, 0xee1f00, 0xc42036cff0)\n\t/usr/local/go/src/io/io.go:325 +0x58\ngolang.org/x/net/http2.readFrameHeader(0xc4205446f8, 0x9, 0x9, 0x1e97300, 0xc420075320, 0x0, 0xc400000000, 0xc4204047a0, 0xc)\n\t/go/src/golang.org/x/net/http2/frame.go:237 +0x7b\ngolang.org/x/net/http2.(*Framer).ReadFrame(0xc4205446c0, 0x0, 0x0, 0x0, 0x0)\n\t/go/src/golang.org/x/net/http2/frame.go:464 +0xa4\ngoogle.golang.org/grpc/transport.(*framer).readFrame(0xc4201e1230, 0xc4201b7d40, 0xc4201b7d40, 0x0, 0x0)\n\t/go/src/google.golang.org/grpc/transport/http_util.go:505 +0x2f\ngoogle.golang.org/grpc/transport.(*http2Client).reader(0xc420072800)\n\t/go/src/google.golang.org/grpc/transport/http2_client.go:921 +0xb6\ncreated by google.golang.org/grpc/transport.newHTTP2Client\n\t/go/src/google.golang.org/grpc/transport/http2_client.go:203 +0x8b1\n\ngoroutine 285 [select, 9 minutes]:\ngoogle.golang.org/grpc.(*addrConn).transportMonitor(0xc4202f3040)\n\t/go/src/google.golang.org/grpc/clientconn.go:720 +0x63e\ngoogle.golang.org/grpc.(*ClientConn).resetAddrConn.func1(0xc4202f3040)\n\t/go/src/google.golang.org/grpc/clientconn.go:479 +0x1de\ncreated by google.golang.org/grpc.(*ClientConn).resetAddrConn\n\t/go/src/google.golang.org/grpc/clientconn.go:480 +0x3c7\n\ngoroutine 288 [select, 9 minutes]:\ngoogle.golang.org/grpc/transport.(*http2Client).controller(0xc420072800)\n\t/go/src/google.golang.org/grpc/transport/http2_client.go:999 +0x59d\ncreated by google.golang.org/grpc/transport.newHTTP2Client\n\t/go/src/google.golang.org/grpc/transport/http2_client.go:233 +0xe78\n\n```\nPlease assign, take a look and update the issue accordingly."},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/258191039","html_url":"https://github.com/cockroachdb/cockroach/issues/10430#issuecomment-258191039","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/10430","id":258191039,"user":{"login":"tamird","id":1535036,"avatar_url":"https://avatars.githubusercontent.com/u/1535036?v=3","gravatar_id":"","url":"https://api.github.com/users/tamird","html_url":"https://github.com/tamird","followers_url":"https://api.github.com/users/tamird/followers","following_url":"https://api.github.com/users/tamird/following{/other_user}","gists_url":"https://api.github.com/users/tamird/gists{/gist_id}","starred_url":"https://api.github.com/users/tamird/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tamird/subscriptions","organizations_url":"https://api.github.com/users/tamird/orgs","repos_url":"https://api.github.com/users/tamird/repos","events_url":"https://api.github.com/users/tamird/events{/privacy}","received_events_url":"https://api.github.com/users/tamird/received_events","type":"User","site_admin":false},"created_at":"2016-11-03T16:13:38Z","updated_at":"2016-11-03T16:13:38Z","body":"#10409."}}
{"actor":{"display_login":"tamird"}}