{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/13382","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/13382/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/13382/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/13382/events","html_url":"https://github.com/kubernetes/kubernetes/issues/13382","id":104046753,"number":13382,"title":"Suddenly getting TLS Handshake timeout on most requests to the api server","user":{"login":"joshm1","id":182629,"avatar_url":"https://avatars.githubusercontent.com/u/182629?v=3","gravatar_id":"","url":"https://api.github.com/users/joshm1","html_url":"https://github.com/joshm1","followers_url":"https://api.github.com/users/joshm1/followers","following_url":"https://api.github.com/users/joshm1/following{/other_user}","gists_url":"https://api.github.com/users/joshm1/gists{/gist_id}","starred_url":"https://api.github.com/users/joshm1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/joshm1/subscriptions","organizations_url":"https://api.github.com/users/joshm1/orgs","repos_url":"https://api.github.com/users/joshm1/repos","events_url":"https://api.github.com/users/joshm1/events{/privacy}","received_events_url":"https://api.github.com/users/joshm1/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P1","name":"priority/P1","color":"eb6420"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/CSI-API%20Machinery%20SIG","name":"team/CSI-API Machinery SIG","color":"d2b48c"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":40,"created_at":"2015-08-31T12:43:59Z","updated_at":"2016-03-15T00:33:01Z","closed_at":null,"body":"I'm unable to use `kubectl` because of TLS handshake timeout.\r\n\r\n```\r\nkubectl get pods\r\nerror: couldn't read version from server: Get https://master-ip/api: net/http: TLS handshake timeout\r\n```\r\n\r\nedit: After several tries I got one to go through, so it's not happening 100% of the time.  But the error has been recorded over 4k times in the logs.\r\n\r\nI've also noticed the error is showing up often in `/var/log/kube-apisever.log` from requests from minions.\r\n\r\n```\r\nI0831 12:35:19.950945       8 logs.go:41] http: TLS handshake error from 172.20.0.65:58143: EOF\r\nI0831 12:35:20.599641       8 logs.go:41] http: TLS handshake error from 172.20.0.148:53774: EOF\r\nI0831 12:35:20.601809       8 logs.go:41] http: TLS handshake error from 172.20.0.240:41027: EOF\r\nI0831 12:35:22.272985       8 logs.go:41] http: TLS handshake error from 172.20.0.25:42939: EOF\r\nI0831 12:35:23.210921       8 logs.go:41] http: TLS handshake error from 172.20.0.240:41034: EOF\r\nI0831 12:35:24.520112       8 logs.go:41] http: TLS handshake error from 172.20.0.65:58172: EOF\r\n```\r\n\r\nI also noticed a lot of `dial tcp 127.0.0.1:8080: connection refused` errors in the logs on various endpoints.\r\n\r\n```\r\nE0831 12:36:54.588879       5 reflector.go:136] Failed to list *api.ResourceQuota: Get http://127.0.0.1:8080/api/v1/resourcequotas: dial tcp 127.0.0.1:8080: connection refused\r\nE0831 12:36:54.589161       5 reflector.go:136] Failed to list *api.Secret: Get http://127.0.0.1:8080/api/v1/secrets?fieldSelector=type%3Dkubernetes.io%2Fservice-account-token: dial tcp 127.0.0.1:8080: connection refused\r\nE0831 12:36:54.628359       5 reflector.go:136] Failed to list *api.ServiceAccount: Get http://127.0.0.1:8080/api/v1/serviceaccounts: dial tcp 127.0.0.1:8080: connection refused\r\nE0831 12:36:54.628471       5 reflector.go:136] Failed to list *api.LimitRange: Get http://127.0.0.1:8080/api/v1/limitranges: dial tcp 127.0.0.1:8080: connection refused\r\nE0831 12:36:54.628608       5 reflector.go:136] Failed to list *api.Namespace: Get http://127.0.0.1:8080/api/v1/namespaces: dial tcp 127.0.0.1:8080: connection refused\r\nE0831 12:36:54.628669       5 reflector.go:136] Failed to list *api.Namespace: Get http://127.0.0.1:8080/api/v1/namespaces: dial tcp 127.0.0.1:8080: connection refused\r\n```\r\n\r\nIf it's relevant, the ephemeral filesystem on /mnt/ephemeral/kubernetes is 99% on one of my minions.  Most of my kube-system pods (kube-ui, kube-dns, elasticsearch, etc) are running that minion. It's full because of elasticsearch and heapster empity-dir volumes, and the mount is only 3.75GB. This filesystem being full caused other problems this weekend, including containers from the kube-dns pod shutting down which in turn brought down all of my other production pods over the weekend.  "},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/196586064","html_url":"https://github.com/kubernetes/kubernetes/issues/13382#issuecomment-196586064","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/13382","id":196586064,"user":{"login":"nikhiljindal","id":10199099,"avatar_url":"https://avatars.githubusercontent.com/u/10199099?v=3","gravatar_id":"","url":"https://api.github.com/users/nikhiljindal","html_url":"https://github.com/nikhiljindal","followers_url":"https://api.github.com/users/nikhiljindal/followers","following_url":"https://api.github.com/users/nikhiljindal/following{/other_user}","gists_url":"https://api.github.com/users/nikhiljindal/gists{/gist_id}","starred_url":"https://api.github.com/users/nikhiljindal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nikhiljindal/subscriptions","organizations_url":"https://api.github.com/users/nikhiljindal/orgs","repos_url":"https://api.github.com/users/nikhiljindal/repos","events_url":"https://api.github.com/users/nikhiljindal/events{/privacy}","received_events_url":"https://api.github.com/users/nikhiljindal/received_events","type":"User","site_admin":false},"created_at":"2016-03-15T00:33:01Z","updated_at":"2016-03-15T00:33:01Z","body":"We had increased the number of file descriptors for apiserver before: https://github.com/kubernetes/kubernetes/pull/6353, but the error at that time was specifically: `too many open files`.\r\nAlso, in the e2e failure: subsequent requests worked fine. There was just one command that failed.\r\n\r\ne2e logs are at http://kubekins.dls.corp.google.com:8081/job/kubernetes-pull-build-test-e2e-gce/32142/"}}