{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/20975","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/20975/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/20975/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/20975/events","html_url":"https://github.com/kubernetes/kubernetes/pull/20975","id":132727076,"number":20975,"title":"Introduce Controller to Check API Server Health","user":{"login":"DirectXMan12","id":119737,"avatar_url":"https://avatars.githubusercontent.com/u/119737?v=3","gravatar_id":"","url":"https://api.github.com/users/DirectXMan12","html_url":"https://github.com/DirectXMan12","followers_url":"https://api.github.com/users/DirectXMan12/followers","following_url":"https://api.github.com/users/DirectXMan12/following{/other_user}","gists_url":"https://api.github.com/users/DirectXMan12/gists{/gist_id}","starred_url":"https://api.github.com/users/DirectXMan12/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DirectXMan12/subscriptions","organizations_url":"https://api.github.com/users/DirectXMan12/orgs","repos_url":"https://api.github.com/users/DirectXMan12/repos","events_url":"https://api.github.com/users/DirectXMan12/events{/privacy}","received_events_url":"https://api.github.com/users/DirectXMan12/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/cla:%20yes","name":"cla: yes","color":"bfe5bf"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/api-change","name":"kind/api-change","color":"c7def8"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/old-docs","name":"kind/old-docs","color":"c7def8"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/release-note-label-needed","name":"release-note-label-needed","color":"db5a64"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/size/XL","name":"size/XL","color":"ee5500"}],"state":"open","locked":false,"assignee":{"login":"smarterclayton","id":1163175,"avatar_url":"https://avatars.githubusercontent.com/u/1163175?v=3","gravatar_id":"","url":"https://api.github.com/users/smarterclayton","html_url":"https://github.com/smarterclayton","followers_url":"https://api.github.com/users/smarterclayton/followers","following_url":"https://api.github.com/users/smarterclayton/following{/other_user}","gists_url":"https://api.github.com/users/smarterclayton/gists{/gist_id}","starred_url":"https://api.github.com/users/smarterclayton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smarterclayton/subscriptions","organizations_url":"https://api.github.com/users/smarterclayton/orgs","repos_url":"https://api.github.com/users/smarterclayton/repos","events_url":"https://api.github.com/users/smarterclayton/events{/privacy}","received_events_url":"https://api.github.com/users/smarterclayton/received_events","type":"User","site_admin":false},"milestone":null,"comments":26,"created_at":"2016-02-10T15:17:31Z","updated_at":"2016-04-26T20:40:38Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/20975","html_url":"https://github.com/kubernetes/kubernetes/pull/20975","diff_url":"https://github.com/kubernetes/kubernetes/pull/20975.diff","patch_url":"https://github.com/kubernetes/kubernetes/pull/20975.patch"},"body":"The \"kubernetes\" service (in the \"default\" namespace) has endpoints which\r\npoint to any API servers active in the cluster.  Each API server is in charge\r\nof maintaining the list of endpoints -- it ensures that its own endpoint\r\nis present, and trims the list down to a fixed number of endpoints,\r\nthereby maintaining that the list will eventually contain the correct\r\nnumber of active API servers\r\n\r\nHowever, conditions may occur in which an API server crashes or\r\notherwise becomes unreachable, and a new server is not brought online\r\n(or is not brought online in a timely fashion).  In this case, one or\r\nmore addresses in the \"kubernetes\" endpoints may exist which point to\r\nunreachable API servers.  This, in turn, can cause issues (such as\r\ndelays) as clients attempt to use the \"kubernetes\" service, and requests\r\ntime out or are not answered.\r\n\r\nTo remedy this, a new controller is introduced which periodically checks\r\nthe health of the API servers using by checking the '/healthz' URL\r\nserved by the API servers.  If it finds an unreachable API server, it\r\nremoves the corresponding address from the \"kubernetes\" endpoints\r\nlist (if the API server comes back up, it will re-add itself to the\r\nlist).  The controller is off by default, but may be turned on by\r\nsetting the `--health-check-controller-sync-peroid` to a non-zero\r\ntime period."},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/214879129","html_url":"https://github.com/kubernetes/kubernetes/pull/20975#issuecomment-214879129","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/20975","id":214879129,"user":{"login":"smarterclayton","id":1163175,"avatar_url":"https://avatars.githubusercontent.com/u/1163175?v=3","gravatar_id":"","url":"https://api.github.com/users/smarterclayton","html_url":"https://github.com/smarterclayton","followers_url":"https://api.github.com/users/smarterclayton/followers","following_url":"https://api.github.com/users/smarterclayton/following{/other_user}","gists_url":"https://api.github.com/users/smarterclayton/gists{/gist_id}","starred_url":"https://api.github.com/users/smarterclayton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smarterclayton/subscriptions","organizations_url":"https://api.github.com/users/smarterclayton/orgs","repos_url":"https://api.github.com/users/smarterclayton/repos","events_url":"https://api.github.com/users/smarterclayton/events{/privacy}","received_events_url":"https://api.github.com/users/smarterclayton/received_events","type":"User","site_admin":false},"created_at":"2016-04-26T20:40:38Z","updated_at":"2016-04-26T20:40:38Z","body":"To refresh the key you'll need to deal with retries, timing windows, and ensuring the create / fail errors are properly handled.  However, how do they know which entries to remove from the list?"}}