<p>I want to work out a <a href="https://kubernetes.io/docs/concepts/services-networking/network-policies/" rel="nofollow noreferrer">network policies</a> in Kubernetes cluster to have a fine grained generalized access control between modules.</p>

<p>I've prepared a Kubernetes setup with 2 manifests:</p>

<ol>
<li>2-container nginx pod, with 2 ports listening and some generic data returned, one is port 80, another - 81</li>
<li>I have 3 console pods with 2 on/off labels: "allow80" and "allow80". So, if "allow80"present, console pod can access dual-nginx via service entry point, port 80. Same is applied for port 81</li>
</ol>

<p>I have 3 console pods:</p>

<ol>
<li>console-full - access port 80 and 81, [allow80, allow81]</li>
<li>console-partial - port 80 - on, 81 - off, [allow80]</li>
<li>console-no-access - both, 80 and 81 - restricted []</li>
</ol>

<p>Test setup. It will create all necessary components in the "net-policy-test" namespace.</p>

<p>To create:</p>

<p><code>kubectl apply -f net_policy_test.yaml</code></p>

<p>To cleanup:</p>

<p><code>kubectl delete -f net_policy_test.yaml</code></p>

<pre><code>apiVersion: v1
kind: Namespace
metadata:
  name: net-policy-test
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx1
  namespace: net-policy-test
data:
  index.html: |
    &lt;!DOCTYPE html&gt;
    &lt;html&gt;
    &lt;head&gt;
    &lt;title&gt;nginx, instance1&lt;/title&gt;
    &lt;/head&gt;
    &lt;body&gt;
      &lt;h1&gt;nginx, instance 1, port 80&lt;/h1&gt;
    &lt;/body&gt;
    &lt;/html&gt;
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx2
  namespace: net-policy-test
data:
  index.html: |
    &lt;!DOCTYPE html&gt;
    &lt;html&gt;
    &lt;head&gt;
    &lt;title&gt;nginx, instance2&lt;/title&gt;
    &lt;/head&gt;
    &lt;body&gt;
      &lt;h1&gt;nginx, instance 2, port 81&lt;/h1&gt;
    &lt;/body&gt;
    &lt;/html&gt;
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-conf1
  namespace: net-policy-test
data:
  default.conf: |
    server {
        listen       80;
        server_name  localhost;


        location / {
            root   /usr/share/nginx/html;
            index  index.html index.htm;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/share/nginx/html;
        }
    }
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-conf2
  namespace: net-policy-test
data:
  default.conf: |
    server {
        listen       81;
        server_name  localhost;


        location / {
            root   /usr/share/nginx/html;
            index  index.html index.htm;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/share/nginx/html;
        }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dual-nginx
  namespace: net-policy-test
  labels:
    app: dual-nginx
    environment: test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dual-nginx
  template:
    metadata:
      labels:
        app: dual-nginx
        name: dual-nginx
    spec:
      containers:
      - image: nginx
        name: nginx1
        ports:
        - name: http1
          containerPort: 80
        volumeMounts:
          - name: html1
            mountPath: /usr/share/nginx/html
          - name: config1
            mountPath: /etc/nginx/conf.d
      - image: nginx
        name: nginx2
        ports:
        - name: http2
          containerPort: 81
        volumeMounts:
          - name: html2
            mountPath: /usr/share/nginx/html
          - name: config2
            mountPath: /etc/nginx/conf.d
      volumes:
        - name: html1
          configMap:
            name: nginx1
        - name: html2
          configMap:
            name: nginx2
        - name: config1
          configMap:
            name: nginx-conf1
        - name: config2
          configMap:
            name: nginx-conf2

---
apiVersion: v1
kind: Service
metadata:
  name: dual-nginx
  namespace: net-policy-test
spec:
  selector:
    app: dual-nginx
  ports:
  - name: web1
    port: 80
    targetPort: http1
  - name: web2
    port: 81
    targetPort: http2
---
# this console deployment will have full access to nginx
apiVersion: apps/v1
kind: Deployment
metadata:
  name: console-full
  namespace: net-policy-test
  labels:
    app: console-full
    environment: test
    nginx-access: full
spec:
  replicas: 1
  selector:
    matchLabels:
      app: console-full
  template:
    metadata:
      labels:
        app: console-full
        name: console-full
        allow80: "true"
        allow81: "true"
    spec:
      containers:
      - image: alpine:3.9
        name: main
        command: ["sh", "-c", "apk update &amp;&amp; apk add curl &amp;&amp; sleep 10000"]

---
# this console deployment will have partial access to nginx
apiVersion: apps/v1
kind: Deployment
metadata:
  name: console-partial
  namespace: net-policy-test
  labels:
    app: console-partial
    environment: test
    nginx-access: partial
spec:
  replicas: 1
  selector:
    matchLabels:
      app: console-partial
  template:
    metadata:
      labels:
        app: console-partial
        name: console-partial
        allow80: "true"

    spec:
      containers:
      - image: alpine:3.9
        name: main
        command: ["sh", "-c", "apk update &amp;&amp; apk add curl &amp;&amp; sleep 10000"]
---
# this console deployment will have no access to nginx
apiVersion: apps/v1
kind: Deployment
metadata:
  name: console-no-access
  namespace: net-policy-test
  labels:
    app: console-no-access
    environment: test
    nginx-access: none
spec:
  replicas: 1
  selector:
    matchLabels:
      app: console-no-access
  template:
    metadata:
      labels:
        app: console-no-access
        name: console-no-access
    spec:
      containers:
      - image: alpine:3.9
        name: main
        command: ["sh", "-c", "apk update &amp;&amp; apk add curl &amp;&amp; sleep 10000"]
</code></pre>

<p>And policies, 
again, to apply:</p>

<p><code>kubectl apply -f policies.yaml</code></p>

<p>To cleanup:</p>

<p><code>kubectl delete -f policies.yaml</code></p>

<pre><code>
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: nginx-restrict80
spec:
  podSelector:
    matchLabels:
      app: "dual-nginx"
  policyTypes:
    - Ingress
  ingress:
  - from:
      - podSelector:
          matchLabels:
            allow80: "true"
    ports:
      - protocol: TCP
        port: 80
  - from:
      - podSelector:
          matchLabels:
            allow81: "true"
    ports:
      - protocol: TCP
        port: 81
</code></pre>

<p>If I leave just one condition with "from", for one port, it works as expected. I have or not have a port access, depending on corresponding label presence, allow80 or allow81</p>

<p>If 2 conditions are present, partial pod has access to both, port 80 and 81:</p>

<ol>
<li>Switch to right namespace:</li>
</ol>

<pre class="lang-sh prettyprint-override"><code>kubectl config set-context --current --namespace=net-policy-test
</code></pre>

<ol start="2">
<li>Check the labels:</li>
</ol>

<pre class="lang-sh prettyprint-override"><code>kubectl get pods -l allow80
NAME                               READY   STATUS    RESTARTS   AGE
console-full-78d5499959-p5kbb      1/1     Running   1          4h14m
console-partial-6679745d79-kbs5w   1/1     Running   1          4h14m

kubectl get pods -l allow81
NAME                            READY   STATUS    RESTARTS   AGE
console-full-78d5499959-p5kbb   1/1     Running   1          4h14m
</code></pre>

<ol start="3">
<li>Check the access from pod "console-partial-..." which should access port 80 and not 81:</li>
</ol>

<pre class="lang-sh prettyprint-override"><code>kubectl exec -ti console-partial-6679745d79-kbs5w curl http://dual-nginx:80
&lt;!DOCTYPE html&gt;
&lt;html&gt;
&lt;head&gt;
&lt;title&gt;nginx, instance1&lt;/title&gt;
&lt;/head&gt;
&lt;body&gt;
  &lt;h1&gt;nginx, instance 1, port 80&lt;/h1&gt;
&lt;/body&gt;
&lt;/html&gt;

kubectl exec -ti console-partial-6679745d79-kbs5w curl http://dual-nginx:81
&lt;!DOCTYPE html&gt;
&lt;html&gt;
&lt;head&gt;
&lt;title&gt;nginx, instance2&lt;/title&gt;
&lt;/head&gt;
&lt;body&gt;
  &lt;h1&gt;nginx, instance 2, port 81&lt;/h1&gt;
&lt;/body&gt;
&lt;/html&gt;

</code></pre>

<p>Partial-access pod has access to ports 80 and 81.</p>

<p>Pod, having no labels (console-no-access-), having no access to either port, which is expected</p>

<p>It resembles to what is described in this presentation: <a href="https://www.youtube.com/watch?v=3gGpMmYeEO8&amp;feature=youtu.be&amp;t=620" rel="nofollow noreferrer">Youtube, Securing Cluster Networking with Network Policies - Ahmet Balkan, Google</a>. So, at least one flag, "allow80" or "allow81" gives access to everything. How come?</p>

<p>Now, questions:</p>

<ol>
<li>Is it expected behaviour?</li>
<li>How to make simple flag based access control, for purpose automating it or passing to admins, who can easily produce these in large numbers?</li>
</ol>

Vetal
<p><strong>TLDR</strong>: It's working on my cluster exactly how you wanted it to work.</p>

<p>Now a bit of explanations and examples.</p>

<p>I've created a cluster at <a href="https://cloud.google.com/kubernetes-engine/" rel="nofollow noreferrer">GKE</a> and enabled <a href="https://kubernetes.io/docs/concepts/services-networking/network-policies/" rel="nofollow noreferrer">Network Policies</a> using following command:</p>

<p><code>gcloud beta container clusters create test1 --enable-network-policy --zone us-cental1-a</code></p>

<p>Then copied your exact deployment yaml and Network Policy yaml with no changes and deployed them.</p>

<pre><code>$ kubectl apply -f policy-test.yaml
namespace/net-policy-test created
configmap/nginx1 created
configmap/nginx2 created
configmap/nginx-conf1 created
configmap/nginx-conf2 created
deployment.apps/dual-nginx created
service/dual-nginx created
deployment.apps/console-full created
deployment.apps/console-partial created
deployment.apps/console-no-access created

$ kubectl apply -f policy.yaml
networkpolicy.networking.k8s.io/nginx-restrict80 configured
</code></pre>

<p>The Network Policy that you wrote worked exactly as you wanted.</p>

<p><em>console-partial was only able to access nginx on port 80.
console-no-access had no access to any nginx.</em></p>

<p>I think this is because GKE uses Calico as CNI</p>

<blockquote>
  <p>Google Container Engine (GKE) also provides beta support for <a href="https://cloud.google.com/container-engine/docs/network-policy" rel="nofollow noreferrer">Network Policies</a> using the Calico networking plugin</p>
</blockquote>

<p>While you used <code>--network-policy azure</code>, which is <a href="https://github.com/Azure/azure-container-networking/blob/master/docs/cni.md" rel="nofollow noreferrer">Azure-CNI</a>.
I'm unable to test this on AKS but you might try changing that to calico. This is explained here, <a href="https://learn.microsoft.com/bs-latn-ba/azure/aks/use-network-policies#create-an-aks-cluster-and-enable-network-policy" rel="nofollow noreferrer">Create an AKS cluster and enable network policy</a></p>

<blockquote>
  <ul>
  <li>Creates an AKS cluster in the defined virtual network and enables network policy.
  
  <ul>
  <li>The  <em>azure</em>  network policy option is used. To use Calico as the network policy option instead, use the  <code>--network-policy calico</code>  parameter.</li>
  </ul></li>
  </ul>
</blockquote>

<p>As for automating flags, maybe this would work for you.</p>

<p>You can see labels here:</p>

<pre><code>$ kubectl describe pods console-partial | grep -A3 Labels
Labels:             allow80=true
                    app=console-partial
                    name=console-partial
                    pod-template-hash=6c6dc7d94f
</code></pre>

<p>When I've started to edit  <a href="https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/" rel="nofollow noreferrer"><code>Labels</code></a>, using <code>kubectl labels</code>.</p>

<p>Removed a label <code>allow80="true"</code>:</p>

<pre><code>$ kubectl label pods console-partial-6c6dc7d94f-v8k5q allow80-
pod/console-partial-6c6dc7d94f-v8k5q labeled
$ kubectl describe pods console-partial | grep -A3 Labels
Labels:             app=console-partial
                    name=console-partial
                    pod-template-hash=6c6dc7d94f
</code></pre>

<p>and add a label <code>allow81= true</code></p>

<pre><code>kubectl label pods console-partial-6c6dc7d94f-v8k5q "allow81=true"
pod/console-partial-6c6dc7d94f-v8k5q labeled

$ kubectl describe pods console-partial | grep -3 Labels
Labels:             allow81=true
                    app=console-partial
                    name=console-partial
                    pod-template-hash=6c6dc7d94f
</code></pre>

<p>You can see from a test that the policy works as you wanted.</p>

<pre><code>$ kubectl exec -it console-partial-6c6dc7d94f-v8k5q curl http://dual-nginx:81
&lt;!DOCTYPE html&gt;
&lt;html&gt;
&lt;head&gt;
&lt;title&gt;nginx, instance2&lt;/title&gt;
&lt;/head&gt;
&lt;body&gt;
  &lt;h1&gt;nginx, instance 2, port 81&lt;/h1&gt;
&lt;/body&gt;
&lt;/html&gt;
$ kubectl exec -it console-partial-6c6dc7d94f-v8k5q curl http://dual-nginx:80
^Ccommand terminated with exit code 130
</code></pre>

<p>I hope this is remotely helpful.</p>
