<p>I have a Kubernetes 1.18 cluster using weave as my CNI. I have a ZMQ based pub/sub app and I am often (not always) seeing it take 2 minutes before the subscriber can receive messages from the publisher. This seems to be some sort of socket timeout uniqe to my Kubernetes environment.</p>
<hr />
<p>Here is my trivial ZMQ app example</p>
<pre class="lang-py prettyprint-override"><code>#!/bin/env python2
import zmq, sys, time, argparse, logging, datetime, threading
from zmq.utils.monitor import recv_monitor_message

FORMAT = '%(asctime)-15s %(message)s'
logging.basicConfig(format=FORMAT)

if zmq.zmq_version_info() &lt; (4, 0):
    raise RuntimeError(&quot;monitoring in libzmq version &lt; 4.0 is not supported&quot;)

logging.error(&quot;libzmq-%s&quot; % zmq.zmq_version())
if zmq.zmq_version_info() &lt; (4, 0):
    raise RuntimeError(&quot;monitoring in libzmq version &lt; 4.0 is not supported&quot;)

EVENT_MAP = {}
logging.error(&quot;Event names:&quot;)
for name in dir(zmq):
    if name.startswith('EVENT_'):
        value = getattr(zmq, name)
        logging.error(&quot;%21s : %4i&quot; % (name, value))
        EVENT_MAP[value] = name


def event_monitor(monitor):
    while monitor.poll():
        evt = recv_monitor_message(monitor)
        evt.update({'description': EVENT_MAP[evt['event']]})
        logging.error(&quot;Event: {}&quot;.format(evt))
        if evt['event'] == zmq.EVENT_MONITOR_STOPPED:
            break
    monitor.close()
    logging.error(&quot;event monitor thread done!&quot;)

parser = argparse.ArgumentParser(&quot;Simple zmq pubsub example&quot;)
parser.add_argument(&quot;pub_or_sub&quot;, help=&quot;Either pub or sub&quot;)
parser.add_argument(&quot;host&quot;, help=&quot;host address to connect to if sub otherwise the address to bind to&quot;)
parser.add_argument(&quot;--port&quot;, &quot;-p&quot;, type=int, help=&quot;The port to use&quot;, default=4567)
args = parser.parse_args()

context = zmq.Context()

if args.pub_or_sub.lower() == &quot;sub&quot;:
    zmq_socket = context.socket(zmq.SUB)
    monitor = zmq_socket.get_monitor_socket()
    t = threading.Thread(target=event_monitor, args=(monitor,))
    t.setDaemon(True)
    t.start()

    zmq_socket.setsockopt(zmq.SUBSCRIBE, &quot;&quot;)
    zmq_socket.connect(&quot;tcp://{}:{}&quot;.format(args.host, args.port))
    while 1:
        if zmq_socket.poll(timeout=1000):
            logging.error(&quot;Received: {}&quot;.format(zmq_socket.recv()))
        else:
            logging.error(&quot;No message available&quot;)
elif args.pub_or_sub.lower() == &quot;pub&quot;:
    zmq_socket = context.socket(zmq.PUB)
    monitor = zmq_socket.get_monitor_socket()
    t = threading.Thread(target=event_monitor, args=(monitor,))
    t.setDaemon(True)
    t.start()
    zmq_socket.bind(&quot;tcp://{}:{}&quot;.format(args.host, args.port))
    i = 0
    while 1:
        logging.error(&quot;Sending message: {}&quot;.format(i))
        zmq_socket.send(&quot;Message {} at {}&quot;.format(i, datetime.datetime.now()))
        i += 1
        time.sleep(1.0)
else:
    raise RuntimeError(&quot;Needs to either be sub or pub nothing else allowed&quot;)
</code></pre>
<p>Here is how I am deploying it within Kubernetes:</p>
<pre><code>---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pub-deployment
  labels:
    app: pub
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pub
  template:
    metadata:
      labels:
        app: pub
    spec:
      containers:
      - name: pub
        image: bagoulla/zmq:latest
        command: [&quot;pubsub&quot;, &quot;pub&quot;, &quot;0.0.0.0&quot;]
---
apiVersion: v1
kind: Service
metadata:
  name: pub
spec:
  selector:
    app: pub
  ports:
    - protocol: TCP
      port: 4567
      targetPort: 4567
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sub-deployment
  labels:
    app: sub
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sub
  template:
    metadata:
      labels:
        app: sub
    spec:
      containers:
      - name: sub
        image: bagoulla/zmq:latest
        command: [&quot;pubsub&quot;, &quot;sub&quot;, &quot;pub&quot;]
</code></pre>
<p>What I would expect to see from the subscriber, and I do see when running outside of Kubernetes on the same host (though still in Docker), is the following repeated in quick succession until the pub container is ready and routed:</p>
<pre><code>2020-08-16 08:12:09,141 Event: {'endpoint': 'tcp://127.0.0.1:4567', 'event': 2, 'value': 115, 'description': 'EVENT_CONNECT_DELAYED'}
2020-08-16 08:12:09,141 Event: {'endpoint': 'tcp://127.0.0.1:4567', 'event': 128, 'value': 12, 'description': 'EVENT_CLOSED'}
2020-08-16 08:12:09,142 Event: {'endpoint': 'tcp://127.0.0.1:4567', 'event': 4, 'value': 183, 'description': 'EVENT_CONNECT_RETRIED'}
2020-08-16 08:12:09,328 Event: {'endpoint': 'tcp://127.0.0.1:4567', 'event': 2, 'value': 115, 'description': 'EVENT_CONNECT_DELAYED'}
</code></pre>
<p>However what I see in Kubernetes instead is:</p>
<pre><code>│ 2020-08-16 05:54:51,724 Event: {'endpoint': 'tcp://pub:4567', 'event': 2, 'value': 115, 'description': 'EVENT_CONNECT_DELAYED'}                                                                                                            │
.... 2 minutes later....
│ 2020-08-16 05:56:59,038 No message available                                                                                                                                                                                               │
│ 2020-08-16 05:56:59,056 Event: {'endpoint': 'tcp://pub:4567', 'event': 128, 'value': 12, 'description': 'EVENT_CLOSED'}                                                                                                                    │
│ 2020-08-16 05:56:59,056 Event: {'endpoint': 'tcp://pub:4567', 'event': 4, 'value': 183, 'description': 'EVENT_CONNECT_RETRIED'}                                                                                                            │
│ 2020-08-16 05:56:59,243 Event: {'endpoint': 'tcp://pub:4567', 'event': 2, 'value': 115, 'description': 'EVENT_CONNECT_DELAYED'}                                                                                                            │
│ 2020-08-16 05:56:59,245 Event: {'endpoint': 'tcp://pub:4567', 'event': 1, 'value': 12, 'description': 'EVENT_CONNECTED'}                                                                                                                   │
│ 2020-08-16 05:56:59,286 Received: Message 127 at 2020-08-16 05:56:59.286036  
</code></pre>
<p>Clearly something within Kubernetes is preventing the &quot;EVENT_CLOSED&quot; event from occurring in a timely manor. What could this be?</p>

Youssef Bagoulla
<p>The issue is that when the service comes up it essentially creates a TCP black hole where tcp connections can be started but never end up connecting. Users should set a timeout on TCP connections so that they can retry the connection until the underlying deployment or pod is up and routed properly. For ZMQ this can be done with the <code>ZMQ_CONNECT_TIMEOUT</code> socket option.</p>
