{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/56993","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/56993/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/56993/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/56993/events","html_url":"https://github.com/kubernetes/kubernetes/issues/56993","id":280774834,"number":56993,"title":"Support sole tenancy","user":{"login":"davidopp","id":8264113,"avatar_url":"https://avatars2.githubusercontent.com/u/8264113?v=4","gravatar_id":"","url":"https://api.github.com/users/davidopp","html_url":"https://github.com/davidopp","followers_url":"https://api.github.com/users/davidopp/followers","following_url":"https://api.github.com/users/davidopp/following{/other_user}","gists_url":"https://api.github.com/users/davidopp/gists{/gist_id}","starred_url":"https://api.github.com/users/davidopp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidopp/subscriptions","organizations_url":"https://api.github.com/users/davidopp/orgs","repos_url":"https://api.github.com/users/davidopp/repos","events_url":"https://api.github.com/users/davidopp/events{/privacy}","received_events_url":"https://api.github.com/users/davidopp/received_events","type":"User","site_admin":false},"labels":[{"id":267761362,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/feature","name":"kind/feature","color":"c7def8","default":false},{"id":778118404,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/lifecycle/stale","name":"lifecycle/stale","color":"795548","default":false},{"id":357119284,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/auth","name":"sig/auth","color":"d2b48c","default":false},{"id":125550211,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/scheduling","name":"sig/scheduling","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2017-12-10T02:14:42Z","updated_at":"2018-03-28T19:51:41Z","closed_at":null,"author_association":"OWNER","body":"tl;dr: We need to modify  [PodAffinityTerm.Namespaces](https://github.com/kubernetes/api/blob/5cb170de1f060ebf2dde62cc4b8241fdb3c01203/core/v1/types.go#L2584) to allow a way to say \"all namespaces.\" We also need an optional admission controller to add a \"tenant\" label to all non-system pods.\r\n\r\nFull details:\r\n\r\n[Taints and tolerations](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) enable \"dedicated nodes\" i.e. statically space-partition your cluster among users/workloads, although this isn't yet secure (see https://github.com/kubernetes/kubernetes/issues/48041#issuecomment-323266418 and https://github.com/kubernetes/community/pull/911). \r\n\r\nA more efficient way of ensuring that untrusting/conflicting tenants don't share a node, is \"sole tenancy.\" In this model a pod can only use a node if it is empty (ignoring system pods) or is only running pods from the same tenant (ignoring system pods). This dynamically space-partitions the cluster among tenants and doesn't require the management overhead of dedicated nodes. (Of course in both approaches, some nodes can be shared, by tenants that have pods that don't need exclusive access, if that's desired.) Sole tenancy is what this issue is about.\r\n\r\nUnfortunately there's no way to do this right now.\r\n\r\nTo understand the rest of this issue, you need to understand [PodAffinityTerm](https://github.com/kubernetes/api/blob/5cb170de1f060ebf2dde62cc4b8241fdb3c01203/core/v1/types.go#L2584) which is the way a pod that is requesting pod affinity or anti-affinity specifies which pods it wants to affinitize/anti-affinitize to/from. The important part is that it takes a list of namespaces, and a label selector that is applied against the pods in that/those namespace(s). If the list of namespaces is empty, it means \"my namespace.\" There is no way to specify \"all namespaces except mine\" (you'd have to list them individually, and keep them up-to-date). Label selector is described [here](https://github.com/kubernetes/apimachinery/blob/a04e753f5223cf882db01ac64212682ea28767b4/pkg/apis/meta/v1/types.go#L899); empty label selector matches all objects. Lastly, note that you can specify more than one pod affinity and/or pod anti-affinity expression for a pod, and the resulting sets of nodes are intersected. (There is no way to do OR.) \r\n\r\nFirst, note that hard pod affinity (e.g. the tenant labels all of its pods with some label tenant=X, and uses hard pod affinity to say \"only run me on nodes that are already running pods with tenant=X\") doesn't work because (1) the scheduler may choose a node that is already running pods without tenant=X, and (2) even if the scheduler picks an initially empty node, once it fills up that node with tenant=X pods, the rest of the tenant's pods will go pending. (If you're wondering how the first pod from the tenant gets scheduled, see the special case code [here](https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/algorithm/predicates/predicates.go#L1305).) An alternative way of using hard pod affinity has the same problem: don't bother with the labels, and say \"only run me on nodes that have pods that match the 'return all objects' label selector in my namespace\" (i.e. on nodes that have pods from my namespace). Note that there is no way to say \"OR empty nodes\" for either of these approaches, but such a feature wouldn't help anyway because it would only solve problem (2) not problem (1).\r\n\r\nSo, we need to use hard pod anti-affinity. What we want to say is \"do NOT run me on nodes that are already running a pod from any namespace such that the pod has label tenant != X\" (note that \"do NOT run me on nodes that are already running pods from another namespace\" sounds appealing but doesn't work because nodes invariably will have some pods from the kube-system namespace running). Unfortunately you can't express this today because PodAffinityTerm has no way to say \"all namespaces\" -- you either have to give an explicit list of namespaces you care about or say \"my namespace.\" So we need to change `PodAffinityTerm.Namespaces` to let you say \"all namespaces.\" (I believe it just requires changing [this function](https://github.com/kubernetes/kubernetes/blob/078a3b653b9c1bd74adb4265f3bc8a3dbc386153/plugin/pkg/scheduler/algorithm/priorities/util/topologies.go#L28).) Then we use an admission controller to label all pods with a \"tenant\" label, and a pod from tenant X that wants sole tenancy can put in its PodSpec a PodAntiAffinity that looks like (pseudo-config here!)\r\n```\r\nPodAntiAffinity.RequiredDuringSchedulingIgnoredDuringExecution = {\r\n    [LabelSelector: tenant NOT IN {\"X\"}\r\n    Namespaces: all namespaces\r\n    TopologyKey: node]\r\n}\r\n```\r\n\r\nPer-node system pods (e.g. fluentd) would not have a tenant label, otherwise no sole-tenant pods would ever schedule. Cluster-level system pods (e.g. heapster) could or could not have a tenant label, depending on whether the cluster admin wants sole tenant pods to be able to co-schedule with cluster-level system pods. Note that the reason this works (i.e. a pod with with such an anti-affinity rule can schedule onto a node with pods that have no \"tenant\" label) is that `tenant NOT IN {\"X\"}` will not match pods that lack the tenant label (due to [this code](https://github.com/kubernetes/apimachinery/blob/18a564baac720819100827c16fdebcadb05b2d0d/pkg/labels/selector.go#L187)).\r\n\r\nTo make this secure, the admission controller that adds a \"tenant\" label to all non-system pods should also deny creation of pods that already have the label if the value is not set properly (or change it to the proper one), and deny updating a pod to change or remove the \"tenant\" label.\r\n\r\nBTW I'm 90% sure we originally let you say \"all namespaces\" but removed that feature for some reason. But I don't see how to do sole tenancy without it. It would be good to re-discover why we removed it, though (probably performance issues?). There is hopefully some way we can make this not have terrible performance (e.g. do the label comparison once per pod instead of once per pod per namespace in the system).\r\n\r\nThe idea of having the system automatically add a \"namespace\" label to every pod has been discussed in the past. I believe it was in the discussion about how to select pods for NetworkPolicy.\r\n\r\n/kind feature\r\n/sig scheduling\r\n/sig auth\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/377014019","html_url":"https://github.com/kubernetes/kubernetes/issues/56993#issuecomment-377014019","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/56993","id":377014019,"user":{"login":"bsalamat","id":14308438,"avatar_url":"https://avatars3.githubusercontent.com/u/14308438?v=4","gravatar_id":"","url":"https://api.github.com/users/bsalamat","html_url":"https://github.com/bsalamat","followers_url":"https://api.github.com/users/bsalamat/followers","following_url":"https://api.github.com/users/bsalamat/following{/other_user}","gists_url":"https://api.github.com/users/bsalamat/gists{/gist_id}","starred_url":"https://api.github.com/users/bsalamat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bsalamat/subscriptions","organizations_url":"https://api.github.com/users/bsalamat/orgs","repos_url":"https://api.github.com/users/bsalamat/repos","events_url":"https://api.github.com/users/bsalamat/events{/privacy}","received_events_url":"https://api.github.com/users/bsalamat/received_events","type":"User","site_admin":false},"created_at":"2018-03-28T19:51:41Z","updated_at":"2018-03-28T19:51:41Z","author_association":"CONTRIBUTOR","body":"/remove-lifecycle stale"}}
{"actor":{"display_login":"bsalamat"}}