<p>Short summary: My ZMQ sockets do not receive (probably also send) messages when I deploy the code on kubernetes.</p>
<p>I have an application that involves multiple clients and servers. It is developed on node and uses ZeroMQ as communication layer. It works on my local machine, it works on docker and I am trying to deploy the application using kubernetes.</p>
<p>When deploying the application, the pods, the deployments and the kubernete's services launch. Apparently everything goes ok, but the clients send an initial message that never arrives to the server. The deployments are in the same namespace and I use Flannel as CNI. To the best of my knowledge, the cluster was properly initialized but the messages never arrive.</p>
<p>I read this <a href="https://stackoverflow.com/questions/63430835/2-minutes-for-zmq-pub-sub-to-connect-in-kubernetes">question</a> about ZMQ sockets having problems binding on kubernetes. I have tried playing with the ZMQ_CONNECT_TIMEOUT parameter but it does not do anything. Also, unlike the question I cite, my messages never arrive.</p>
<p>I could provide some code, but there is a lot of code and I don't think the application is the problem. I think I am missing something on the Kubernetes configuration since it's my firts time using it . Let me know if you need more information.</p>
<p>Edit 1. 12/01/2021</p>
<p>As @anemyte suggests I will try to provide a simplified version of the code:</p>
<p>Client Side:</p>
<pre><code>initiate () {
        return new Promise(resolve =&gt; {
            this.N_INCOMING = 0;
            this.N_OUTGOING = 0;
            this.rrCounter = 0;
            this.PULL_SOCKET.bind(&quot;tcp://*:&quot; + this.MY_PORT, error =&gt; {
                utils.handleError(error);
                this.PUB_SOCKET.bind(&quot;tcp://*:&quot; + (this.MY_PORT + 1), error =&gt; {
                    utils.handleError(error);
                    this.SUB_SOCKET.subscribe(&quot;&quot;);                    
                    this.SUB_SOCKET.connect(this.SERVER + &quot;:&quot; + (this.SERVER_PORT + 1),
                        error =&gt; {utils.handleError(error)});
                    this.PULL_SOCKET.on(&quot;message&quot;, (m) =&gt; this.handlePullSocket(m));
                    this.SUB_SOCKET.on(&quot;message&quot;, (m) =&gt; this.handleSubSocket(m));
                    this.SERVER_PUSH_SOCKET = zmq.socket(&quot;push&quot;);
                    this.SERVER_PUSH_SOCKET.connect(this.SERVER + &quot;:&quot; + this.SERVER_PORT,
                        error =&gt; {utils.handleError(error)});
                    this.sendHello();
                    resolve();
                });
            });
        });
</code></pre>
<p>Server side:</p>
<pre><code>    initiate () {
        return new Promise(resolve =&gt; {
            this.PULL_SOCKET.bind(this.MY_IP + &quot;:&quot; + this.MY_PORT, err =&gt; {
                if (err) {
                    console.log(err);
                    process.exit(0);
                }
                this.PUB_SOCKET.bind(this.MY_IP + &quot;:&quot; + (this.MY_PORT + 1), err =&gt; {
                    if (err) {
                        console.log(err);
                        process.exit(0);
                    }
                    this.PULL_SOCKET.on(&quot;message&quot;, (m) =&gt; this.handlePullSocket(m));
                    resolve();
                });
            });
        });
    }
</code></pre>
<p>Client initiates the connection by sending the Hello Message. Server's listener function <code>handlePullSocket</code> should handle those messages.</p>
<p>Edit 2. 12/01/2021
As requested, I am adding the deployment/service configurations.</p>
<p>Client-deployment.yaml</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    kompose.cmd: kompose convert -f docker-compose-resolved.yml
    kompose.version: 1.19.0 (f63a961c)
  creationTimestamp: null
  labels:
    io.kompose.service: c1
  name: c1
  namespace: fishtrace
spec:
  replicas: 1
  selector:
    matchLabels:
      app: c1
  strategy:
    type: Recreate
  template:
    metadata:
      annotations:
        kompose.cmd: kompose convert -f docker-compose-resolved.yml
        kompose.version: 1.19.0 (f63a961c)
      creationTimestamp: null
      labels:
        app: c1
    spec:
      containers:
      - env:
        - name: NODO_ADDRESS
          value: 0.0.0.0
        - name: NODO_PUERTO
          value: &quot;9999&quot;
        - name: NODO_PUERTO_CADENA
          value: &quot;8888&quot;
        - name: SERVER_ADDRESS
          value: tcp://servidor
        - name: SERVER_PUERTO
          value: &quot;7777&quot;
        image: registrogeminis.com/docker_c1_rpi:latest
        name: c1
        ports:
          - containerPort: 8888
          - containerPort: 9999
        resources: {}
        volumeMounts:
        - mountPath: /app/vol
          name: c1-volume
        imagePullPolicy: Always
      restartPolicy: Always
      imagePullSecrets:
        - name: myregistrykey
      volumes:
        - name: c1-volume
          persistentVolumeClaim:
            claimName: c1-volume
status: {}
</code></pre>
<p>Client-service.yaml</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Service
metadata:
  annotations:
    kompose.cmd: ./kompose convert
    kompose.version: 1.22.0 (955b78124)
  creationTimestamp: null
  labels:
    io.kompose.service: c1
  name: c1
spec:
  ports:
    - name: &quot;9999&quot;
      port: 9999
      targetPort: 9999
    - name: &quot;8888&quot;
      port: 8888
      targetPort: 8888
  selector:
    app: c1
  type: ClusterIP
</code></pre>
<p>Server-deployment.yaml</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    kompose.cmd: kompose convert -f docker-compose-resolved.yml
    kompose.version: 1.19.0 (f63a961c)
  creationTimestamp: null
  labels:
    io.kompose.service: servidor
  name: servidor
  namespace: fishtrace
spec:
  replicas: 1
  selector:
    matchLabels:
      app: servidor
  strategy:
    type: Recreate
  template:
    metadata:
      annotations:
        kompose.cmd: kompose convert -f docker-compose-resolved.yml
        kompose.version: 1.19.0 (f63a961c)
      creationTimestamp: null
      labels:
        app: servidor
    spec:
      containers:
      - env:
        - name: SERVER_ADDRESS
          value: tcp://*
        - name: SERVER_PUERTO
          value: &quot;7777&quot;
        image: registrogeminis.com/docker_servidor_rpi:latest
        name: servidor
        ports:
          - containerPort: 7777
          - containerPort: 7778
        resources: {}
        volumeMounts:
        - mountPath: /app/vol
          name: servidor-volume
        imagePullPolicy: Always
      restartPolicy: Always
      imagePullSecrets:
        - name: myregistrykey
      volumes:
      - name: servidor-volume
        persistentVolumeClaim:
          claimName: servidor-volume
status: {}

</code></pre>
<p>Server-service.yaml</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Service
metadata:
  annotations:
    kompose.cmd: ./kompose convert
    kompose.version: 1.22.0 (955b78124)
  creationTimestamp: null
  labels:
    io.kompose.service: servidor
  name: servidor
spec:
  ports:
    - name: &quot;7777&quot;
      port: 7777
      targetPort: 7777
  selector:
    app: servidor
  type: ClusterIP
</code></pre>

<p>At the end it was a DNS problem. It was always a DNS problem. Thanks to @Matt for pointing out the problem.</p>
<p>In the official Kubernetes <a href="https://v1-18.docs.kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/" rel="nofollow noreferrer">DNS doc</a> they state that there is a known issue with systems that use <code>/etc/resolv.conf</code> as a link to the real configuration file, <code>/run/systemd/resolve/resolv.conf</code> in my case. It is a well-known problem, and the recommended <a href="https://github.com/coredns/coredns/blob/master/plugin/loop/README.md#troubleshooting-loops-in-kubernetes-clusters" rel="nofollow noreferrer">solution</a> is to update the kubelet's configuration to point to <code>/run/systemd/resolve/resolv.conf</code>.</p>
<p>To do so, I added the line <code>resolvConf:/run/systemd/resolve/resolv.conf</code> in <code>/var/lib/kubelet/config.yaml</code>. I also edited <code>/etc/kubernetes/kubelet.conf</code> just to be sure. Finally, you are supposed to reload the service executing <code>sudo systemctl daemon-reload &amp;&amp; sudo systemctl restart kubelet</code> to propagate the changes.</p>
<p>However, I already did that before asking in SE. And it did not seem to work. <strong>I had to restart the whole cluster</strong> to make the changes take effect. Then the DNS worked perfectly and the ZMQ sockets behaved as expected.</p>
<p><strong>Update 31/04/2021</strong>: I discovered that you have to forcefully restart the coredns kubernetes' service to actually propagate the changes. So in the end <code>kubectl rollout restart coredns -n kube-system</code> after restarting the kubelet service was enough.</p>
