<p>I'm sending UDP packets (statsd) from pods on a host to <code>&lt;hostIP&gt;:8125</code>. On the other end, a collector (datadog-agent using <code>hostPort</code>; one per host via DaemonSet) picks up the packets and does it's thing.</p>

<p>Generally this works fine, but if I ever delete + re-create the collector (<code>kubectl delete pod datadog-agent-xxxx</code>; new pod is started on same IP/port a few seconds later), traffic from <em>existing</em> client-sockets stop arriving at the collector (UDP sockets created <em>after</em> the pod-rescheduling works fine).</p>

<p>Re-starting just the agent inside the collector pod (<code>kubectl exec -it datadog-agent-xxxxx agent stop</code>; auto-restarts after ~30s) the same old traffic <em>does</em> show up. So containers somehow must have an impact.</p>

<p>While UDP are (supposedly) stateless, something, somewhere is obviously keeping state around!? Any ideas/pointers?</p>

<p>Each "client" pod has something like this in the deployment/pod:</p>

<pre><code>kind: Deployment
...
spec:
  template:
    spec:
      containers:
        - name: webservice
          env:
            # Statsd defaults to localhost:8125, but that's this pod. Use `hostPort` on collector + hostIP here to get around that.
            DD_AGENT_HOST:
              valueFrom:
                fieldRef:
                  fieldPath: 'status.hostIP'

</code></pre>

<p>On the collector (following <a href="https://docs.datadoghq.com/agent/kubernetes/daemonset_setup/?tab=k8sfile" rel="nofollow noreferrer">datadog's k8s docs</a>):</p>

<pre><code>kind: DaemonSet
...
spec:
  template:
    spec:
      containers:
        - image: datadog/agent:6.140.0
          ports:
            - containerPort: 8125
              hostPort: 8125
              protocol: UDP
          env:
            - name: DD_DOGSTATSD_NON_LOCAL_TRAFFIC
              value: "true"
            - ...
</code></pre>

<p>This happens on Kubernetes 1.12 on Google Kubernetes Engine.</p>

<p>This is likely related to this <a href="https://github.com/containernetworking/plugins/issues/123" rel="nofollow noreferrer">issue in the portmap plugin</a>. The current working theory is that a conntrack entry is created when the client pod reaches out for the UDP host port, and that entry becomes stale when the server pod is deleted, but it's not deleted, so clients keep hitting it, essentially blackholing the traffic.</p>

<p>You can try removing the conntrack entry with something like <code>conntrack -D -p udp --dport 8125</code> on one of the impacted host. If that solves the issue then that was the root cause of your problem.</p>

<p>This workaround described in the GitHub issue should mitigate the issue until a fix is merged:</p>

<p>You can add an initContainer to the server's pod to run the conntrack command when it starts:</p>

<pre><code>initContainers: 
        - image: &lt;conntrack-image&gt;
          imagePullPolicy: IfNotPresent 
          name: conntrack 
          securityContext: 
            allowPrivilegeEscalation: true 
            capabilities: 
              add: ["NET_ADMIN"] 
          command: ['sh', '-c', 'conntrack -D -p udp']
</code></pre>
