<p>I'm trying to write an application where all pods are connected to each other. I have read in <a href="https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/" rel="nofollow noreferrer">documentation</a>:</p>

<blockquote>
  <p>The Pods’ ordinals, hostnames, SRV records, and A record names have not changed, but the IP addresses associated with the Pods may have changed. In the cluster used for this tutorial, they have. This is why it is important not to configure other applications to connect to Pods in a StatefulSet by IP address.</p>
  
  <p>If you need to find and connect to the active members of a StatefulSet, you should query the CNAME of the Headless Service (<code>nginx.default.svc.cluster.local</code>). The SRV records associated with the CNAME will contain only the Pods in the StatefulSet that are Running and Ready.</p>
  
  <p>If your application already implements connection logic that tests for liveness and readiness, you can use the SRV records of the Pods ( <code>web-0.nginx.default.svc.cluster.local</code>, <code>web-1.nginx.default.svc.cluster.local</code>), as they are stable, and your application will be able to discover the Pods’ addresses when they transition to Running and Ready.</p>
</blockquote>

<p>I though I can do it in following way:</p>

<ul>
<li>Lookup SRV records of service to check which pods are ready</li>
<li>Connect to all ready pods</li>
<li>Open the port which signify the readiness</li>
</ul>

<p>However when I started implementing it on minikube it seemed to be racy and when I query A/SRV records:</p>

<ul>
<li>On first node I get no records found error (sounds OK) before I open the port</li>
<li>On second node I sometimes get no records found sometimes one record</li>
<li>On third node I sometimes get one record and sometimes two records</li>
</ul>

<p>It seems to me that there is a race between updating of DNS records and statefulset startup. I'm not entirely sure what I'm doing wrong and how I misunderstood the documentation.</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: hello-world-lb
  labels:
    app: hello-world-lb
spec:
  ports:
  - port: 8080
    name: web
  type: LoadBalancer
  selector:
    app: hello-world
---
apiVersion: v1
kind: Service
metadata:
  name: hello-world
  labels:
    app: hello-world
spec:
  ports:
  - port: 8080
    name: web
  - port: 3080
    name: hello-world
  clusterIP: None
  selector:
    app: hello-world
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: hello-world
spec:
  selector:
    matchLabels:
      app: hello-world
  serviceName: "hello-world"
  replicas: 3
  template:
    metadata:
      labels:
        app: hello-world
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: hello-world
        image: hello-world
        imagePullPolicy: Never
        ports:
        - containerPort: 8080
          name: web
        - containerPort: 3080
          name: hello-world
        livenessProbe:
          tcpSocket:
            port: 8080
</code></pre>

<p><strong>EDIT</strong> Currently the code is as follows:</p>

<ul>
<li>Query A/SRV records of <code>hello-world.default.svc.cluster.local.</code>/<code>_hello-world._tcp.hello-world.default.svc.cluster.local.</code> and print them for debug</li>
<li>Bind to 3080 port and start to listen (connecting logic not implemented)</li>
<li>Open 8080 port</li>
</ul>

<p>I expected that A/SRV records for <code>hello-world-0</code> will be empty, for <code>hello-world-1</code> will contain hello-world-0 and for <code>hello-world-N+1</code> will contain <code>hello-world-0</code> to <code>hello-world-N</code>. During rolling update A/SRV records would contain all other peers.</p>

<p>However it seems that DNS records are updated asynchronously so even when the lifeness of pod <code>n</code> is detected and pod <code>n + 1</code> is started it is not guaranteed that pod <code>n + 1</code> will see address of pod <code>n</code> in DNS.</p>

<p>add the below annotation to service definition</p>

<pre><code>  annotations:
    service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
</code></pre>
