{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/coreos-kubernetes/issues/641","repository_url":"https://api.github.com/repos/coreos/coreos-kubernetes","labels_url":"https://api.github.com/repos/coreos/coreos-kubernetes/issues/641/labels{/name}","comments_url":"https://api.github.com/repos/coreos/coreos-kubernetes/issues/641/comments","events_url":"https://api.github.com/repos/coreos/coreos-kubernetes/issues/641/events","html_url":"https://github.com/coreos/coreos-kubernetes/issues/641","id":173250927,"number":641,"title":"kube-aws: single point of failure in kube-dns","user":{"login":"danielfm","id":29534,"avatar_url":"https://avatars.githubusercontent.com/u/29534?v=3","gravatar_id":"","url":"https://api.github.com/users/danielfm","html_url":"https://github.com/danielfm","followers_url":"https://api.github.com/users/danielfm/followers","following_url":"https://api.github.com/users/danielfm/following{/other_user}","gists_url":"https://api.github.com/users/danielfm/gists{/gist_id}","starred_url":"https://api.github.com/users/danielfm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielfm/subscriptions","organizations_url":"https://api.github.com/users/danielfm/orgs","repos_url":"https://api.github.com/users/danielfm/repos","events_url":"https://api.github.com/users/danielfm/events{/privacy}","received_events_url":"https://api.github.com/users/danielfm/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/coreos/coreos-kubernetes/labels/kind/question","name":"kind/question","color":"c7def8"},{"url":"https://api.github.com/repos/coreos/coreos-kubernetes/labels/priority/Pmaybe","name":"priority/Pmaybe","color":"fef2c0"}],"state":"open","locked":false,"assignee":{"login":"aaronlevy","id":374924,"avatar_url":"https://avatars.githubusercontent.com/u/374924?v=3","gravatar_id":"","url":"https://api.github.com/users/aaronlevy","html_url":"https://github.com/aaronlevy","followers_url":"https://api.github.com/users/aaronlevy/followers","following_url":"https://api.github.com/users/aaronlevy/following{/other_user}","gists_url":"https://api.github.com/users/aaronlevy/gists{/gist_id}","starred_url":"https://api.github.com/users/aaronlevy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aaronlevy/subscriptions","organizations_url":"https://api.github.com/users/aaronlevy/orgs","repos_url":"https://api.github.com/users/aaronlevy/repos","events_url":"https://api.github.com/users/aaronlevy/events{/privacy}","received_events_url":"https://api.github.com/users/aaronlevy/received_events","type":"User","site_admin":false},"assignees":[{"login":"aaronlevy","id":374924,"avatar_url":"https://avatars.githubusercontent.com/u/374924?v=3","gravatar_id":"","url":"https://api.github.com/users/aaronlevy","html_url":"https://github.com/aaronlevy","followers_url":"https://api.github.com/users/aaronlevy/followers","following_url":"https://api.github.com/users/aaronlevy/following{/other_user}","gists_url":"https://api.github.com/users/aaronlevy/gists{/gist_id}","starred_url":"https://api.github.com/users/aaronlevy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aaronlevy/subscriptions","organizations_url":"https://api.github.com/users/aaronlevy/orgs","repos_url":"https://api.github.com/users/aaronlevy/repos","events_url":"https://api.github.com/users/aaronlevy/events{/privacy}","received_events_url":"https://api.github.com/users/aaronlevy/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2016-08-25T16:31:25Z","updated_at":"2016-08-29T18:47:32Z","closed_at":null,"body":"As of now, we are currently using just a [single kube-dns replica](https://github.com/coreos/coreos-kubernetes/blob/64e44562fd4da459d631d28f6269c22779372cca/multi-node/aws/pkg/config/templates/cloud-config-controller#L464).\r\n\r\nIsn't this a problem? If the kube-dns pod fails for some reason, all DNS resolution requests would fail until that pod is restarted (or worse, it might take even more time to get the kube-dns pod up if it gets rescheduled in a different node due to resource starvation or something like that).\r\n\r\nThat being said, wouldn't be best to start _at least_ two replicas there?"},"comment":{"url":"https://api.github.com/repos/coreos/coreos-kubernetes/issues/comments/243217236","html_url":"https://github.com/coreos/coreos-kubernetes/issues/641#issuecomment-243217236","issue_url":"https://api.github.com/repos/coreos/coreos-kubernetes/issues/641","id":243217236,"user":{"login":"danielfm","id":29534,"avatar_url":"https://avatars.githubusercontent.com/u/29534?v=3","gravatar_id":"","url":"https://api.github.com/users/danielfm","html_url":"https://github.com/danielfm","followers_url":"https://api.github.com/users/danielfm/followers","following_url":"https://api.github.com/users/danielfm/following{/other_user}","gists_url":"https://api.github.com/users/danielfm/gists{/gist_id}","starred_url":"https://api.github.com/users/danielfm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielfm/subscriptions","organizations_url":"https://api.github.com/users/danielfm/orgs","repos_url":"https://api.github.com/users/danielfm/repos","events_url":"https://api.github.com/users/danielfm/events{/privacy}","received_events_url":"https://api.github.com/users/danielfm/received_events","type":"User","site_admin":false},"created_at":"2016-08-29T18:47:31Z","updated_at":"2016-08-29T18:47:31Z","body":"> So far we've left this to an operator to decide, and it should be as simple as:\r\n> ```\r\n> kubectl scale rc kube-dns --replicas=X\r\n> ```\r\n\r\nThat's what I did in my cluster aside from changing `userdata/cloud-config-controller` so that I don't forget about this in further updates, of course. 😄 \r\n\r\n> I feel like this is still important for an operator to be aware of, as \"having X copies\" still doesn't mean you're necessarily HA.\r\n\r\nThat's exactly why I opened this issue. As someone who hangs in #kubernetes-users a lot, it's rather common to see people running into problems because of not using enough replicas there.\r\n\r\nSince `kube-aws` \"just works\", people don't seem to be aware that the DNS is a hugely important component of a Kubernetes cluster, and that things _will_ get ugly if they forget about it as their workload increase.\r\n\r\n> Consider we bootstrap a single node cluster and launch 2 DNS pods, it will end up scheduling them to the same machine. If we add machines to the cluster, it will not reschedule those pods to spread them across the new nodes (although re-scheduling is something which is being discussed upstream).\r\n\r\nThat is a nasty corner case indeed, although in most real-world use cases people will create multi-node multi-AZ clusters, and the scheduler will most likely spread those pods in a way that this does not happen.\r\n\r\nSince having multiple DNS replicas for single-node clusters does not make much sense, we might specify a sane default depending on the number of workers, like:\r\n\r\n- 1 replica for single-node clusters\r\n- 2 replicas multi-node clusters\r\n\r\n(I thought about coming up with a way to calculate the optimal number of replicas based on the cluster size, but this not possible because it very much depends on a number of factors beyond the number of nodes.)\r\n\r\n> Maybe this would be best covered in some form of \"operators\" documentation?\r\n\r\nRegardless of the solution we choose to implement (if any), explaining the importance of the cluster DNS service for a healthy cluster operation is a must."}}
{"actor":{"display_login":"danielfm"}}