<p>In Kubernetes, we have multiple environments, separated by different namespace. I want to ensure that one set of nodes are only used by a specific namespace/environment.</p>

<ol>
<li>Nodes of specific labels should reject all pods, which doesn't belong to a namespace</li>
<li>Pods from a specific namespace should always choose nodes with the labels configured.</li>
</ol>

<p>What is the way to achieve it? Heard about mutating webhook admission controllers, Anybody has a sample, to see how it works.</p>

Ysak
<p>You can use mutating webhook to mutate an incoming pod request from a specific namespace to add <a href="https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity" rel="noreferrer">node affinity</a> or <a href="https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector" rel="noreferrer">node selector</a> in the pod spec.</p>

<p>An example of <code>nodeSelector</code> admission controller <a href="https://pkg.go.dev/k8s.io/kubernetes/plugin/pkg/admission/podnodeselector?tab=doc" rel="noreferrer">here</a>. Full <a href="https://www.mgasch.com/post/podnodesel/" rel="noreferrer">guide</a> on how to use it.</p>

<p><strong>Prepare API Server</strong> </p>

<p>Assuming your cluster has been deployed with kubeadm, the kube-apiserver.yaml file is the configuration manifest for the Kubernetes API server. It is located in /etc/kubernetes/manifests. Add the PodNodeSelector admission controller in the --admission-control= flag</p>

<p>Then add a label to a node</p>

<pre><code>kubectl label node kubeprod01 env=production
</code></pre>

<p>Then use that label in the annotation of the namespace.</p>

<pre><code>apiVersion: v1
kind: Namespace
metadata:
  annotations:
    scheduler.alpha.kubernetes.io/node-selector: env=production
</code></pre>

<p>An example of adding node affinity via mutating webhook.</p>

<pre><code>func mutatePods(ar v1beta1.AdmissionReview, o *options) *v1beta1.AdmissionResponse {
    var reviewResponse = &amp;v1beta1.AdmissionResponse{
        Allowed: true,
    }

    podResource := metav1.GroupVersionResource{Group: "", Version: "v1", Resource: "pods"}
    if ar.Request.Resource != podResource {
        glog.Errorf("expect resource to be %s", podResource)
        return nil
    }

    raw := ar.Request.Object.Raw
    pod := v1.Pod{}
    // glog.V(2).Infof("Object: %v", string(raw))
    if err := json.Unmarshal(raw, &amp;pod); err != nil {
        glog.Error(err)
        return nil
    }

    addPodAffinityPatch := fmt.Sprintf(`[
         {"op":"add","path":"/spec/affinity","value":{"nodeAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"preference":{"matchExpressions":[{"key":"%s","operator":"NotIn","values":["%s"]}]},"weight":1}]}}}
    ]`, o.PodAffinityKey, o.PodAffinityValue)

    glog.V(2).Infof("patching pod")
    reviewResponse.Patch = []byte(addPodAffinityPatch)
    pt := v1beta1.PatchTypeJSONPatch
    reviewResponse.PatchType = &amp;pt

    return reviewResponse
}
</code></pre>
