<p>I have the <a href="https://github.com/sasadangelo/patroni-k8s/blob/main/kustomize/spilo/iks/spilo.yaml" rel="nofollow noreferrer">following Kubernetes YAML</a> with a StatefulSet I use to deploy a PostgreSQL cluster with Patroni. However, the question is relative to how Kubernetes registers Pod names in CoreDNS.</p>
<p>According to <a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/" rel="nofollow noreferrer">this documentation</a> in the Stable Network ID section if I create a Headless service called <code>spilodemo-svc</code> for my Pods I can access them using the short hostname (podname.servicename):</p>
<pre><code>spilodemo-0.spilodemo-svc
</code></pre>
<p>Basically, my code worked properly for a long time on a K8s cluster deployed with kubeadm on VirtualBox and Vagrant. Today I wanted to deploy it on IBM Cloud but the hostname above didn't work and the strange thing is that when I repeated my tests on Vagrant/VirtualBox again I wasn't able to have it working anymore and I do not know why.</p>
<p>Now the YAML deploys Spilo that is an open-source project developed by Zalando that is a Docker image with Patroni and PostgreSQL. My code comes from their <a href="https://github.com/zalando/spilo/blob/master/kubernetes/spilo_kubernetes.yaml" rel="nofollow noreferrer">example here</a>.</p>
<p>Basically, they create a ClusterIP Service (and not a Headless) with no Selector. Under these conditions, Kubernetes doesn't create an Endpoint in it. For this reason, we have an Endpoint in the YAML with the same name of the service (it seems this is the binding Kubernetes expect).</p>
<p>Spilo has Python code that always keeps updated this Endpoint with the IP of the primary node.</p>
<p>The StatefulSet has the field serviceName equal to the name of the Service:</p>
<pre><code>serviceName: spilodemo-svc
</code></pre>
<p>and, according to the documentation, this guarantees that Kubernetes creates an entry in CoreDNS for this short hostname (podname.servicename):</p>
<pre><code>spilodemo-0.spilodemo-svc
</code></pre>
<p>and it worked for a long time until today and nothing happened in the meanwhile. To be honest I never fully understand how the DNS name <code>spilodemo-0.spilodemo-svc</code> worked so far since it uses a ClusterIP service instead of a Headless one.</p>
<p>Another strange thing is that the Zalando team uses another Headless service that I called <code>spilodemo-config</code> and according to a comment in their code, it should avoid that Kubernetes delete the Endpoint but this doesn't make much sense to me.</p>
<p>However, today I also tried to convert the Service into a Headless one removing the <code>spilodemo-config</code> one but no luck. Kubernetes only create the entry for the service in the CoreDNS:</p>
<pre><code>spilodemo.spilons.svc.cluster.local
</code></pre>
<p>but not the one for each Pod:</p>
<pre><code>spilodemo-0.spilodemo-svc
spilodemo-1.spilodemo-svc
spilodemo-2.spilodemo-svc
</code></pre>
<p>Can anyone help me to figure out what's going on with my YAML file and how I can get the three short hostnames above working in CoreDNS?</p>
<p>PS
On Stackoverflow I found these discussions:</p>
<ul>
<li><a href="https://stackoverflow.com/questions/67285052/hostname-of-pods-in-same-statefulset-can-not-be-resolved">Hostname of pods in the same StatefulSet can not be resolved</a></li>
<li><a href="https://stackoverflow.com/questions/58815171/stateful-pod-hostname-doesnt-resolve">Stateful Pod hostname doesn't resolve</a>
but they don't address my issue.</li>
</ul>

<p>After almost three days of tests, I found a solution. The solution depends on two things:</p>
<ol>
<li>how Kubernetes works;</li>
<li>how Patroni works.</li>
</ol>
<p><strong>How Kubernetes Works</strong></p>
<p>When you create a StatefulSet deployment (but this is true also for Deployment), let's say with 3 pods, Kubernetes register in CoreDNS three DNS names:</p>
<pre><code>IP-with-dashes.&lt;namespace&gt;.pod.cluster.local
</code></pre>
<p>however, these names are useless for me because I cannot set them in advance on my YAML files because it depends on the IP Kubernetes assigned to Pods.</p>
<p>However, for StatefulSet deployments, according to <a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/" rel="nofollow noreferrer">this documentation in the Stable Network ID section</a> if I create a Headless service for my Pod I can access them using the short hostname (podname.servicename) of FQDN (...svc.cluster.local).</p>
<p>Here is the Headless service I needed to create:</p>
<pre><code>---
apiVersion: v1
kind: Service
metadata:
  name: spilodemo-svc
  labels:
    application: spilo
    spilo-cluster: spilodemo
spec:
  clusterIP: None
  selector:
    application: spilo
    spilo-cluster: spilodemo
</code></pre>
<p>It's important here to set the selector to bind all three pods. Another important thing is to add the following line to your StatefulSet with a name equal to the headless service:</p>
<pre><code>serviceName: spilodemo-svc
</code></pre>
<p>This is the Kubernetes part. Now you can reference your Pods with DNS names:</p>
<pre><code>spilodemo-0.spilodemo-svc
spilodemo-1.spilodemo-svc
spilodemo-2.spilodemo-svc
</code></pre>
<p>or FQDN:</p>
<pre><code>spilodemo-0.spilodemo-svc.&lt;namespace&gt;.svc.cluster.local
spilodemo-1.spilodemo-svc.&lt;namespace&gt;.svc.cluster.local
spilodemo-2.spilodemo-svc.&lt;namespace&gt;.svc.cluster.local
</code></pre>
<p><strong>How Patroni Works</strong></p>
<p>However, using Pods' DNS name is not meaningful for clients because they need a single point of access. For this reason, the Patroni team suggest to create a ClusterIP service like this:</p>
<pre><code>---
apiVersion: v1
kind: Service
metadata:
  name: spilodemo
  labels:
    application: spilo
    spilo-cluster: spilodemo
spec:
  type: ClusterIP
  ports:
  - name: postgresql
    port: 5432
    targetPort: 5432
</code></pre>
<p><strong>Note</strong>: there is no selector. This is not an error. When you create a Service like this Kubernetes creates a ClusterIP service (then it can be referenced using an IP or hostname) but without an Endpoint. This means that you connect to its IP or its DNS name: <code>spilodemo.&lt;namespace&gt;.svc.cluster.local</code>, the connection hangs.</p>
<p>For this reason, the Patroni team asks you to add in your YAML file the following Endpoint having the same name as the ClusterIP service.</p>
<pre><code>apiVersion: v1
kind: Endpoints
metadata:
  name: spilodemo
  labels:
    application: spilo
    spilo-cluster: spilodemo
subsets: []
</code></pre>
<p>Patroni, internally, has a piece of code in Python that via Kubernetes API updates this endpoint with the Master Pod IP. Patroni is able to determine the Endpoint to update using its relative labels above (application, spilo-cluster) that you can even customize.</p>
<p>At this point, Patroni cluster clients only need to use this DNS name (the ClusterIP one) or the relative IP:</p>
<pre><code>spilodemo.spilons.svc.cluster.local
</code></pre>
<p>the connection is automatically redirected to the Pod master node IP.</p>
<p>So far so good. Now the confusing part. If you look at the Patroni Kubernetes <a href="https://github.com/zalando/spilo/blob/master/kubernetes/spilo_kubernetes.yaml" rel="nofollow noreferrer">sample file</a> in Spilo code, you node another headless service was already present.</p>
<pre><code>---
# headless service to avoid deletion of patronidemo-config endpoint
apiVersion: v1
kind: Service
metadata:
  name: spilodemo-config
  labels:
    application: spilo
    spilo-cluster: spilodemo
spec:
  clusterIP: None
</code></pre>
<p>What confuse me was the presence of this headless service. I didn't understand its purpose. In the beginning, I thought it was the headless service required to have the Pods DNS name mentioned above. But I was wrong. The purpose of this service is different. Basically, the Zalando team doesn't know how the user writes the YAML file to deploy Patroni. If the user creates the Endpoint but forgot to associate to it a Service, Kubernetes see it as an orphan and delete it. For this reason, the Patroni code itself creates this service on its own. In fact, if you don't define it in the YAML file, Patroni will create it for you.</p>
<p>So, <strong>if Patroni creates it for you why do they add it in the sample YAML above?</strong> The reason is permissions. If Pod doesn't have permissions cannot create it. This is the reason they added it in the YAML. It's a bit confusing but this is the whole story.</p>
