{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/57061","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/57061/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/57061/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/57061/events","html_url":"https://github.com/kubernetes/kubernetes/issues/57061","id":281218387,"number":57061,"title":"Kube 1.7 against etcd 3.2.9 can result in hung etcd connections at large scales","user":{"login":"smarterclayton","id":1163175,"avatar_url":"https://avatars0.githubusercontent.com/u/1163175?v=4","gravatar_id":"","url":"https://api.github.com/users/smarterclayton","html_url":"https://github.com/smarterclayton","followers_url":"https://api.github.com/users/smarterclayton/followers","following_url":"https://api.github.com/users/smarterclayton/following{/other_user}","gists_url":"https://api.github.com/users/smarterclayton/gists{/gist_id}","starred_url":"https://api.github.com/users/smarterclayton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smarterclayton/subscriptions","organizations_url":"https://api.github.com/users/smarterclayton/orgs","repos_url":"https://api.github.com/users/smarterclayton/repos","events_url":"https://api.github.com/users/smarterclayton/events{/privacy}","received_events_url":"https://api.github.com/users/smarterclayton/received_events","type":"User","site_admin":false},"labels":[{"id":105146071,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21","default":false},{"id":173493835,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/api-machinery","name":"sig/api-machinery","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":11,"created_at":"2017-12-12T00:47:10Z","updated_at":"2018-03-19T04:51:33Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"On a very large 1.7 cluster with etcd 3.2.9, a large number of goroutines in the kube-apiserver are waiting for a new http2 stream for gRPC against etcd.  This results in service account token requests hanging with a timeout (504) because they are unable to acquire a gRPC stream within the default 30s timeout, which results in the service account token authenticator failing.  This also causes almost 100% 429 requests, because when `request rate * duration / max-in-flight > 1` we run out of open connections.\r\n\r\nEach storage has its own etcd connection (usually) which does not result in a transport being shared across backend types.  Only retrieving secrets was affected - not pods or nodes.\r\n\r\n80 node cluster, performing ~60-120 secret requests a second.\r\n\r\nUpstream gRPC has a few known hangs fixed, but only when Keepalive or server side InTapHandle are on (which we don't turn on in Kube).  I don't see an existing etcd issues.\r\n\r\n```\r\ngoroutine profile: total 66730\r\n26058 @ 0x430bea 0x440054 0x43ecbc 0x2286c54 0x22777f5 0x2289c85 0x228a72a 0x228a11c 0x22bdbc2 0x2335ed2 0x232ee2a 0x232ec87 0x2326586 0x231ed18 0x231e5e5 0x231e112 0x2338535 0x3a6c9f4 0x3a1ae74 0x3a1a991 0x3a45837 0x38cc377 0x8740ed 0x8734b7 0x3a51c37 0x3a56e06 0x4184222 0x38d382d 0x38d3412 0x3a51851 0x3a56e06 0x472da22\r\n#\t0x2286c53\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc/transport.wait+0x2e3\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc/transport/transport.go:577\r\n#\t0x22777f4\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc/transport.(*http2Client).NewStream+0x21a4\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc/transport/http2_client.go:319\r\n#\t0x2289c84\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc.sendRequest+0x94\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc/call.go:80\r\n#\t0x228a729\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc.invoke+0x5d9\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc/call.go:191\r\n#\t0x228a11b\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc.Invoke+0x19b\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc/call.go:118\r\n#\t0x22bdbc1\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb.(*kVClient).Range+0xd1\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:2239\r\n#\t0x2335ed1\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3.(*retryWriteKVClient).Range+0x91\t\t\t\t\t<autogenerated>:177\r\n#\t0x232ee29\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3.(*retryKVClient).Range.func1+0x89\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3/retry.go:92\r\n#\t0x232ec86\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3.(*Client).newAuthRetryWrapper.func1+0x46\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3/retry.go:61\r\n#\t0x2326585\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3.(*retryKVClient).Range+0x155\t\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3/retry.go:94\r\n#\t0x231ed17\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3.(*kv).do+0x4f7\t\t\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3/kv.go:140\r\n#\t0x231e5e4\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3.(*kv).Do+0x84\t\t\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3/kv.go:119\r\n#\t0x231e111\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3.(*kv).Get+0xe1\t\t\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3/kv.go:93\r\n#\t0x2338534\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/storage/etcd3.(*store).Get+0x134\t\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/storage/etcd3/store.go:125\r\n#\t0x3a6c9f3\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/registry/generic/registry.(*Store).Get+0x183\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/registry/generic/registry/store.go:617\r\n#\t0x3a1ae73\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers.GetResource.func1+0x193\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:175\r\n#\t0x3a1a990\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers.getResourceHandler.func1+0x200\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:129\r\n#\t0x3a45836\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints.restfulGetResource.func1+0xb6\t\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1097\r\n#\t0x38cc376\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1+0x206\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:144\r\n#\t0x8740ec\tgithub.com/openshift/origin/vendor/github.com/emicklei/go-restful.(*Container).dispatch+0xb8c\t\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/emicklei/go-restful/container.go:277\r\n#\t0x8734b6\tgithub.com/openshift/origin/vendor/github.com/emicklei/go-restful.(*Container).Dispatch+0x56\t\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/emicklei/go-restful/container.go:199\r\n#\t0x3a51c36\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP+0x6e6\t\t\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:153\r\n#\t0x3a56e05\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP+0x85\t\t\t\t\t\t<autogenerated>:72\r\n#\t0x4184221\tgithub.com/openshift/origin/vendor/k8s.io/kube-aggregator/pkg/apiserver.(*proxyHandler).ServeHTTP+0x121\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:91\r\n#\t0x38d382c\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP+0x3dc\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248\r\n#\t0x38d3411\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP+0x71\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234\r\n#\t0x3a51850\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP+0x300\t\t\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:161\r\n#\t0x3a56e05\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP+0x85\t\t\t\t\t\t<autogenerated>:72\r\n#\t0x472da21\tgithub.com/openshift/origin/pkg/cmd/server/origin.namespacingFilter.func1+0xd1\t\t\t\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/origin/handlers.go:52\r\n\r\n\r\n7542 @ 0x430bea 0x440054 0x43ecbc 0x38ce16d 0x38c6739 0x71cfa4 0x170d10f 0x71cfa4 0x3d77138 0x71cfa4 0x472d932 0x71cfa4 0x45980b7 0x71cfa4 0x472e8b1 0x71cfa4 0x472e8b1 0x71cfa4 0x3a52471 0x71f8b2 0x7212b3 0x746494 0x73da1d 0x6f6199 0x45e211\r\n#\t0x38ce16c\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP+0x2ac\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:109\r\n#\t0x38c6738\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithRequestInfo.func1+0x218\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:45\r\n#\t0x71cfa3\tnet/http.HandlerFunc.ServeHTTP+0x43\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1942\r\n#\t0x170d10e\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request.WithRequestContext.func1+0xee\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110\r\n#\t0x71cfa3\tnet/http.HandlerFunc.ServeHTTP+0x43\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1942\r\n#\t0x3d77137\tgithub.com/openshift/origin/pkg/cmd/server/handlers.TranslateLegacyScopeImpersonation.func1+0x227\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/handlers/impersonation_scopes.go:20\r\n#\t0x71cfa3\tnet/http.HandlerFunc.ServeHTTP+0x43\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1942\r\n#\t0x472d931\tgithub.com/openshift/origin/pkg/cmd/server/origin.cacheControlFilter.func1+0xc1\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/origin/handlers.go:20\r\n#\t0x71cfa3\tnet/http.HandlerFunc.ServeHTTP+0x43\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1942\r\n#\t0x45980b6\tgithub.com/openshift/origin/pkg/assets/apiserver.WithAssetServerRedirect.func1+0x86\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/pkg/assets/apiserver/asset_apiserver.go:321\r\n#\t0x71cfa3\tnet/http.HandlerFunc.ServeHTTP+0x43\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1942\r\n#\t0x472e8b0\tgithub.com/openshift/origin/pkg/cmd/server/origin.WithPatternPrefixHandler.func1+0xd0\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/origin/master.go:364\r\n#\t0x71cfa3\tnet/http.HandlerFunc.ServeHTTP+0x43\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1942\r\n#\t0x472e8b0\tgithub.com/openshift/origin/pkg/cmd/server/origin.WithPatternPrefixHandler.func1+0xd0\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/origin/master.go:364\r\n#\t0x71cfa3\tnet/http.HandlerFunc.ServeHTTP+0x43\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1942\r\n#\t0x3a52470\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.(*APIServerHandler).ServeHTTP+0x50\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:198\r\n#\t0x71f8b1\tnet/http.serverHandler.ServeHTTP+0x91\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:2568\r\n#\t0x7212b2\tnet/http.initNPNRequest.ServeHTTP+0x92\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:3088\r\n#\t0x746493\tnet/http.(*initNPNRequest).ServeHTTP+0x73\t\t\t\t\t\t\t\t\t<autogenerated>:312\r\n#\t0x73da1c\tnet/http.(Handler).ServeHTTP-fm+0x4c\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/h2_bundle.go:4331\r\n#\t0x6f6198\tnet/http.(*http2serverConn).runHandler+0x88\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/h2_bundle.go:4611\r\n\r\n6393 @ 0x430bea 0x440054 0x43ecbc 0x922fe4 0x91f073 0x91eba0 0x9035e9 0x728979 0x100586c 0x6d29d2 0x6d2658 0x6d3cf4 0x1011f0d 0x10127f0 0x14d8bf1 0x3b6a1f5 0x3b72a6e 0x16be369 0x3db59ce 0x3db502a 0x25ef382 0x25f13a1 0x25f13a1 0x25ee695 0x38c28d4 0x71cfa4 0x170d10f 0x71cfa4 0x38cefe9 0x71cfa4 0x38d08bb 0x71cfa4\r\n#\t0x922fe3\tgithub.com/openshift/origin/vendor/golang.org/x/net/http2.(*ClientConn).RoundTrip+0xb83\t\t\t\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/golang.org/x/net/http2/transport.go:835\r\n#\t0x91f072\tgithub.com/openshift/origin/vendor/golang.org/x/net/http2.(*Transport).RoundTripOpt+0x172\t\t\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/golang.org/x/net/http2/transport.go:339\r\n#\t0x91eb9f\tgithub.com/openshift/origin/vendor/golang.org/x/net/http2.(*Transport).RoundTrip+0x3f\t\t\t\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/golang.org/x/net/http2/transport.go:301\r\n#\t0x9035e8\tgithub.com/openshift/origin/vendor/golang.org/x/net/http2.noDialH2RoundTripper.RoundTrip+0x38\t\t\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/golang.org/x/net/http2/configure_transport.go:75\r\n#\t0x728978\tnet/http.(*Transport).RoundTrip+0xaf8\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/transport.go:349\r\n#\t0x100586b\tgithub.com/openshift/origin/vendor/k8s.io/client-go/transport.(*userAgentRoundTripper).RoundTrip+0x10b\t\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/transport/round_trippers.go:162\r\n#\t0x6d29d1\tnet/http.send+0x161\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/client.go:249\r\n#\t0x6d2657\tnet/http.(*Client).send+0x107\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/client.go:173\r\n#\t0x6d3cf3\tnet/http.(*Client).Do+0x253\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/client.go:595\r\n#\t0x1011f0c\tgithub.com/openshift/origin/vendor/k8s.io/client-go/rest.(*Request).request+0x2cc\t\t\t\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/rest/request.go:824\r\n#\t0x10127ef\tgithub.com/openshift/origin/vendor/k8s.io/client-go/rest.(*Request).Do+0xbf\t\t\t\t\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/rest/request.go:898\r\n#\t0x14d8bf0\tgithub.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/clientset_generated/clientset/typed/core/v1.(*secrets).Get+0x170\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/clientset_generated/clientset/typed/core/v1/secret.go:69\r\n#\t0x3b6a1f4\tgithub.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/serviceaccount.clientGetter.GetSecret+0xc4\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/serviceaccount/tokengetter.go:49\r\n#\t0x3b72a6d\tgithub.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/serviceaccount.(*clientGetter).GetSecret+0x7d\t\t\t\t<autogenerated>:2\r\n#\t0x16be368\tgithub.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/serviceaccount.(*jwtTokenAuthenticator).AuthenticateToken+0xf18\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/serviceaccount/jwt.go:288\r\n#\t0x3db59cd\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authentication/token/union.(*unionAuthTokenHandler).AuthenticateToken+0x9d\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authentication/token/union/union.go:55\r\n#\t0x3db5029\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authentication/token/cache.(*cachedTokenAuthenticator).AuthenticateToken+0x99\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authentication/token/cache/cached_token_authenticator.go:71\r\n#\t0x25ef381\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authentication/request/bearertoken.(*Authenticator).AuthenticateRequest+0x191\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authentication/request/bearertoken/bearertoken.go:55\r\n#\t0x25f13a0\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authentication/request/union.(*unionAuthRequestHandler).AuthenticateRequest+0x90\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authentication/request/union/union.go:57\r\n#\t0x25f13a0\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authentication/request/union.(*unionAuthRequestHandler).AuthenticateRequest+0x90\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authentication/request/union/union.go:57\r\n#\t0x25ee694\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authentication/group.(*AuthenticatedGroupAdder).AuthenticateRequest+0x54\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authentication/group/authenticated_group_adder.go:40\r\n#\t0x38c28d3\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1+0x83\t\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:55\r\n#\t0x71cfa3\tnet/http.HandlerFunc.ServeHTTP+0x43\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1942\r\n#\t0x170d10e\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request.WithRequestContext.func1+0xee\t\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110\r\n#\t0x71cfa3\tnet/http.HandlerFunc.ServeHTTP+0x43\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1942\r\n#\t0x38cefe8\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithCORS.func1+0x188\t\t\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/cors.go:75\r\n#\t0x71cfa3\tnet/http.HandlerFunc.ServeHTTP+0x43\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1942\r\n#\t0x38d08ba\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.WithPanicRecovery.func1+0x11a\t\t\t\t\t\t/builddir/build/BUILD/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:41\r\n#\t0x71cfa3\tnet/http.HandlerFunc.ServeHTTP+0x43\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1942\r\n```\r\n\r\nThis is very similar to https://github.com/kubernetes/kubernetes/issues/53361, although that was watches, and this has a different symptoms."},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/374100788","html_url":"https://github.com/kubernetes/kubernetes/issues/57061#issuecomment-374100788","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/57061","id":374100788,"user":{"login":"smarterclayton","id":1163175,"avatar_url":"https://avatars0.githubusercontent.com/u/1163175?v=4","gravatar_id":"","url":"https://api.github.com/users/smarterclayton","html_url":"https://github.com/smarterclayton","followers_url":"https://api.github.com/users/smarterclayton/followers","following_url":"https://api.github.com/users/smarterclayton/following{/other_user}","gists_url":"https://api.github.com/users/smarterclayton/gists{/gist_id}","starred_url":"https://api.github.com/users/smarterclayton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smarterclayton/subscriptions","organizations_url":"https://api.github.com/users/smarterclayton/orgs","repos_url":"https://api.github.com/users/smarterclayton/repos","events_url":"https://api.github.com/users/smarterclayton/events{/privacy}","received_events_url":"https://api.github.com/users/smarterclayton/received_events","type":"User","site_admin":false},"created_at":"2018-03-19T04:51:33Z","updated_at":"2018-03-19T04:51:33Z","author_association":"CONTRIBUTOR","body":"The problem here requires client side fixes"}}
{"actor":{"display_login":"smarterclayton"}}