{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/30975","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/30975/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/30975/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/30975/events","html_url":"https://github.com/kubernetes/kubernetes/issues/30975","id":172069712,"number":30975,"title":"Investigate whether we reuse connections after \"429\" error","user":{"login":"wojtek-t","id":10743879,"avatar_url":"https://avatars.githubusercontent.com/u/10743879?v=3","gravatar_id":"","url":"https://api.github.com/users/wojtek-t","html_url":"https://github.com/wojtek-t","followers_url":"https://api.github.com/users/wojtek-t/followers","following_url":"https://api.github.com/users/wojtek-t/following{/other_user}","gists_url":"https://api.github.com/users/wojtek-t/gists{/gist_id}","starred_url":"https://api.github.com/users/wojtek-t/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wojtek-t/subscriptions","organizations_url":"https://api.github.com/users/wojtek-t/orgs","repos_url":"https://api.github.com/users/wojtek-t/repos","events_url":"https://api.github.com/users/wojtek-t/events{/privacy}","received_events_url":"https://api.github.com/users/wojtek-t/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/area/performance","name":"area/performance","color":"d4c5f9"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P1","name":"priority/P1","color":"eb6420"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/CSI-API%20Machinery%20SIG","name":"team/CSI-API Machinery SIG","color":"d2b48c"}],"state":"open","locked":false,"assignee":{"login":"krousey","id":157083,"avatar_url":"https://avatars.githubusercontent.com/u/157083?v=3","gravatar_id":"","url":"https://api.github.com/users/krousey","html_url":"https://github.com/krousey","followers_url":"https://api.github.com/users/krousey/followers","following_url":"https://api.github.com/users/krousey/following{/other_user}","gists_url":"https://api.github.com/users/krousey/gists{/gist_id}","starred_url":"https://api.github.com/users/krousey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/krousey/subscriptions","organizations_url":"https://api.github.com/users/krousey/orgs","repos_url":"https://api.github.com/users/krousey/repos","events_url":"https://api.github.com/users/krousey/events{/privacy}","received_events_url":"https://api.github.com/users/krousey/received_events","type":"User","site_admin":false},"assignees":[{"login":"krousey","id":157083,"avatar_url":"https://avatars.githubusercontent.com/u/157083?v=3","gravatar_id":"","url":"https://api.github.com/users/krousey","html_url":"https://github.com/krousey","followers_url":"https://api.github.com/users/krousey/followers","following_url":"https://api.github.com/users/krousey/following{/other_user}","gists_url":"https://api.github.com/users/krousey/gists{/gist_id}","starred_url":"https://api.github.com/users/krousey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/krousey/subscriptions","organizations_url":"https://api.github.com/users/krousey/orgs","repos_url":"https://api.github.com/users/krousey/repos","events_url":"https://api.github.com/users/krousey/events{/privacy}","received_events_url":"https://api.github.com/users/krousey/received_events","type":"User","site_admin":false},{"login":"lavalamp","id":647318,"avatar_url":"https://avatars.githubusercontent.com/u/647318?v=3","gravatar_id":"","url":"https://api.github.com/users/lavalamp","html_url":"https://github.com/lavalamp","followers_url":"https://api.github.com/users/lavalamp/followers","following_url":"https://api.github.com/users/lavalamp/following{/other_user}","gists_url":"https://api.github.com/users/lavalamp/gists{/gist_id}","starred_url":"https://api.github.com/users/lavalamp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lavalamp/subscriptions","organizations_url":"https://api.github.com/users/lavalamp/orgs","repos_url":"https://api.github.com/users/lavalamp/repos","events_url":"https://api.github.com/users/lavalamp/events{/privacy}","received_events_url":"https://api.github.com/users/lavalamp/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/22","html_url":"https://github.com/kubernetes/kubernetes/milestones/v1.4","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/22/labels","id":1777384,"number":22,"title":"v1.4","description":"https://github.com/kubernetes/features/blob/master/release-1.4/Release-1.4.md","creator":{"login":"bgrant0607","id":7725777,"avatar_url":"https://avatars.githubusercontent.com/u/7725777?v=3","gravatar_id":"","url":"https://api.github.com/users/bgrant0607","html_url":"https://github.com/bgrant0607","followers_url":"https://api.github.com/users/bgrant0607/followers","following_url":"https://api.github.com/users/bgrant0607/following{/other_user}","gists_url":"https://api.github.com/users/bgrant0607/gists{/gist_id}","starred_url":"https://api.github.com/users/bgrant0607/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bgrant0607/subscriptions","organizations_url":"https://api.github.com/users/bgrant0607/orgs","repos_url":"https://api.github.com/users/bgrant0607/repos","events_url":"https://api.github.com/users/bgrant0607/events{/privacy}","received_events_url":"https://api.github.com/users/bgrant0607/received_events","type":"User","site_admin":false},"open_issues":237,"closed_issues":241,"state":"open","created_at":"2016-05-19T18:50:19Z","updated_at":"2016-08-19T16:29:11Z","due_on":"2016-09-15T00:00:00Z","closed_at":null},"comments":1,"created_at":"2016-08-19T07:19:08Z","updated_at":"2016-08-19T16:59:09Z","closed_at":null,"body":"From the recent runs of different scalability tests, I have a hypothesis that we are not reusing connections after \"429\" error is returned to the client.\r\n\r\nHere is description of one tests that I've seen failed:\r\n1. basically in 2000-node kubemark all watches were dropped at some point (that's a separate issue #30873 that is a bit unrelated here).\r\n2. what happened next is that basically all watchers decided to do list + watch, and obvious a bunch of lists failed with 429 error (this is still expected). And we started retrying them.\r\n And this is still ok.\r\n3. Since that point, apiserver started consuming ~30 cores (which also isn't completely unexpected).\r\n4. But what IS unexpected to me, is that looking into profile ~70% of the CPU (which was ~20 cores) were spend in \"doFullHandshake\".\r\n And this is a bit strange to me.\r\n\r\n Do we do something specific on 429 errors (e.g. close the connection) so that we won't be able to reuse it in the future?\r\n Or do you have any other hypothesis, why we were mostly doing \"full handshakes\" at that time?\r\n\r\n@krousey @lavalamp @caesarxuchao @kubernetes/sig-api-machinery "},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/241073792","html_url":"https://github.com/kubernetes/kubernetes/issues/30975#issuecomment-241073792","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/30975","id":241073792,"user":{"login":"smarterclayton","id":1163175,"avatar_url":"https://avatars.githubusercontent.com/u/1163175?v=3","gravatar_id":"","url":"https://api.github.com/users/smarterclayton","html_url":"https://github.com/smarterclayton","followers_url":"https://api.github.com/users/smarterclayton/followers","following_url":"https://api.github.com/users/smarterclayton/following{/other_user}","gists_url":"https://api.github.com/users/smarterclayton/gists{/gist_id}","starred_url":"https://api.github.com/users/smarterclayton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smarterclayton/subscriptions","organizations_url":"https://api.github.com/users/smarterclayton/orgs","repos_url":"https://api.github.com/users/smarterclayton/repos","events_url":"https://api.github.com/users/smarterclayton/events{/privacy}","received_events_url":"https://api.github.com/users/smarterclayton/received_events","type":"User","site_admin":false},"created_at":"2016-08-19T16:59:09Z","updated_at":"2016-08-19T16:59:09Z","body":"On the server side, we can use\r\n\r\n```\r\n      // ErrorLog specifies an optional logger for errors accepting\r\n        // connections and unexpected behavior from handlers.\r\n        // If nil, logging goes to os.Stderr via the log package's\r\n        // standard logger.\r\n        ErrorLog *log.Logger\r\n```\r\n\r\non http server to get more info.  Can we do the same thing on the client?  I don't see anything obvious, but knowing when a connection from a client is closed (vs. terminated by the server, vs gracefully terminated via Connection: Close) might be useful in debugging these cases.\r\n\r\nThe only thing I can think of is that the kubelet has multiple connections to the master, and so instead of reusing its pooled connections it's asking for new connections all at once (and blowing out the pool size in the client).\r\n\r\nCan we verify that the kubelet TCP connection pool size is greater than the default (which is 2?).  Kubelet should definitely have a number of idle connections proportional to number of watches."}}
{"actor":{"display_login":"smarterclayton"}}