<p>I am currently trying to move my calico based clusters to the new Dataplane V2, which is basically a managed Cilium offering.
For local testing, I am running k3d with open source cilium installed, and created a set of NetworkPolicies (k8s native ones, not CiliumPolicies), which lock down the desired namespaces.</p>
<p>My current issue is, that when porting the same Policies on a GKE cluster (with DataPlane enabled), those same policies don't work.</p>
<p>As an example let's take a look into the connection between some app and a database:</p>
<pre class="lang-yaml prettyprint-override"><code>---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: db-server.db-client
  namespace: BAR
spec:
  podSelector:
    matchLabels:
      policy.ory.sh/db: server
  policyTypes:
    - Ingress
  ingress:
    - ports: []
      from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: FOO
          podSelector:
            matchLabels:
              policy.ory.sh/db: client
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: db-client.db-server
  namespace: FOO
spec:
  podSelector:
    matchLabels:
      policy.ory.sh/db: client
  policyTypes:
    - Egress
  egress:
    - ports:
        - port: 26257
          protocol: TCP
      to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: BAR
          podSelector:
            matchLabels:
              policy.ory.sh/db: server
</code></pre>
<p>Moreover, using GCP monitoring tools we can see the expected and actual effect the policies have on connectivity:</p>
<p><strong>Expected:</strong>
<a href="https://i.stack.imgur.com/AOOXu.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/AOOXu.png" alt="Expected" /></a></p>
<p><strong>Actual:</strong>
<a href="https://i.stack.imgur.com/bD0aS.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/bD0aS.png" alt="Actual" /></a></p>
<p>And logs from the application trying to connect to the DB, and getting denied:</p>
<pre class="lang-json prettyprint-override"><code>{
  &quot;insertId&quot;: &quot;FOO&quot;,
  &quot;jsonPayload&quot;: {
    &quot;count&quot;: 3,
    &quot;connection&quot;: {
      &quot;dest_port&quot;: 26257,
      &quot;src_port&quot;: 44506,
      &quot;dest_ip&quot;: &quot;172.19.0.19&quot;,
      &quot;src_ip&quot;: &quot;172.19.1.85&quot;,
      &quot;protocol&quot;: &quot;tcp&quot;,
      &quot;direction&quot;: &quot;egress&quot;
    },
    &quot;disposition&quot;: &quot;deny&quot;,
    &quot;node_name&quot;: &quot;FOO&quot;,
    &quot;src&quot;: {
      &quot;pod_name&quot;: &quot;backoffice-automigrate-hwmhv&quot;,
      &quot;workload_kind&quot;: &quot;Job&quot;,
      &quot;pod_namespace&quot;: &quot;FOO&quot;,
      &quot;namespace&quot;: &quot;FOO&quot;,
      &quot;workload_name&quot;: &quot;backoffice-automigrate&quot;
    },
    &quot;dest&quot;: {
      &quot;namespace&quot;: &quot;FOO&quot;,
      &quot;pod_namespace&quot;: &quot;FOO&quot;,
      &quot;pod_name&quot;: &quot;cockroachdb-0&quot;
    }
  },
  &quot;resource&quot;: {
    &quot;type&quot;: &quot;k8s_node&quot;,
    &quot;labels&quot;: {
      &quot;project_id&quot;: &quot;FOO&quot;,
      &quot;node_name&quot;: &quot;FOO&quot;,
      &quot;location&quot;: &quot;FOO&quot;,
      &quot;cluster_name&quot;: &quot;FOO&quot;
    }
  },
  &quot;timestamp&quot;: &quot;FOO&quot;,
  &quot;logName&quot;: &quot;projects/FOO/logs/policy-action&quot;,
  &quot;receiveTimestamp&quot;: &quot;FOO&quot;
}
</code></pre>
<p>EDIT:</p>
<p>My local env is a k3d cluster created via:</p>
<pre class="lang-sh prettyprint-override"><code>k3d cluster create --image ${K3SIMAGE} --registry-use k3d-localhost -p &quot;9090:30080@server:0&quot; \
            -p &quot;9091:30443@server:0&quot; foobar \
            --k3s-arg=--kube-apiserver-arg=&quot;enable-admission-plugins=PodSecurityPolicy,NodeRestriction,ServiceAccount@server:0&quot; \
            --k3s-arg=&quot;--disable=traefik@server:0&quot; \
            --k3s-arg=&quot;--disable-network-policy@server:0&quot; \
            --k3s-arg=&quot;--flannel-backend=none@server:0&quot; \
            --k3s-arg=feature-gates=&quot;NamespaceDefaultLabelName=true@server:0&quot;

docker exec k3d-server-0 sh -c &quot;mount bpffs /sys/fs/bpf -t bpf &amp;&amp; mount --make-shared /sys/fs/bpf&quot;
kubectl taint nodes k3d-ory-cloud-server-0 node.cilium.io/agent-not-ready=true:NoSchedule --overwrite=true
skaffold run --cache-artifacts=true -p cilium --skip-tests=true --status-check=false
docker exec k3d-server-0 sh -c &quot;mount --make-shared /run/cilium/cgroupv2&quot;
</code></pre>
<p>Where cilium itself is being installed by skaffold, via helm with the following parameters:</p>
<pre class="lang-yaml prettyprint-override"><code>name: cilium
remoteChart: cilium/cilium
namespace: kube-system
version: 1.11.0
upgradeOnChange: true
wait: false
setValues:
  externalIPs.enabled: true
  nodePort.enabled: true
  hostPort.enabled: true
  hubble.relay.enabled: true
  hubble.ui.enabled: true
</code></pre>
<p>UPDATE:
I have setup a third environment: a GKE cluster using the old calico CNI (Legacy dataplane) and installed cilium manually as shown <a href="https://docs.cilium.io/en/v1.10/gettingstarted/k8s-install-helm/" rel="nofollow noreferrer">here</a>. Cilium is working fine, even hubble is working out of the box (unlike with the dataplane v2...) and I found something interesting. The rules behave the same as with the GKE managed cilium, but with hubble working I was able to see this:</p>
<p><a href="https://i.stack.imgur.com/SwhKv.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/SwhKv.png" alt="Hubble db connection" /></a></p>
<p>For some reason cilium/hubble cannot identify the db pod and decipher its labels. And since the labels don't work, the policies that rely on those labels, also don't work.</p>
<p>Another proof of this would be the trace log from hubble:</p>
<p><a href="https://i.stack.imgur.com/pQu0m.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/pQu0m.png" alt="kratos -&gt; db" /></a></p>
<p>Here the destination app is only identified via an IP, and not labels.</p>
<p>The question now is why is this happening?</p>
<p>Any idea how to debug this problem? What could be difference coming from? Do the policies need some tuning for the managed Cilium, or is a bug in GKE?
Any help/feedback/suggestion appreciated!</p>

<p>Update: I was able to solve the mystery and it was ArgoCD all along. Cilium is creating an Endpoint and Identity for each object in the namespace, and Argo was deleting them after deploying the applications.</p>
<p>For anyone who stumbles on this, the solution is to add this exclusion to ArgoCD:</p>
<pre class="lang-yaml prettyprint-override"><code>  resource.exclusions: |
    - apiGroups:
      - cilium.io
      kinds:
      - CiliumIdentity
      - CiliumEndpoint
      clusters:
      - &quot;*&quot;
</code></pre>
