<p>this question is similar to <a href="https://stackoverflow.com/questions/38901877/kubernetes-petset-dns-not-working">Kubernetes PetSet DNS not working</a> but (I believe) distinct. My problem is: I want to use a Kubernetes PetSet to run a sharded database (RethinkDB). I need to pass each shard the dns address of another shard in the database, so that the shards can connect to each other run as a cluster. I also need other services to connect to the database and query it, and I'd like to do that through a k8s NodePort service (I think that if other pods connect to RethinkDB through a service, each client pod will connect to a random Rethink pod, providing a basic kind of load balancing. Using a NodePort service also means I can connect to the Rethink admin console from outside the cluster).</p>

<p>I believe Kubernetes should assign each RethinkDB shard a consistent domain name, and I should be able to pass each shard e.g. <code>rethink-0.rethink-service.default.svc.cluster.local</code> for clustering. However, I've tried two ways of configuring my PetSet and neither seems to assign the domain name <code>rethink-0.rethink-service.default.svc.local</code>:</p>

<p>1) I created a non-headless service for talking to the PetSet and that's it. In this configuration, the the only rethink pet I create seems to be getting a random name:</p>

<pre><code>$ kc get all
NAME       DESIRED   CURRENT   READY     AGE
rc/etcd    1         1         1         46s
rc/pachd   1         1         1         46s
NAME                  CLUSTER-IP   EXTERNAL-IP   PORT(S)                        AGE
svc/etcd              10.0.0.206   &lt;none&gt;        2379/TCP,2380/TCP              46s
svc/kubernetes        10.0.0.1     &lt;none&gt;        443/TCP                        3d
svc/pachd             10.0.0.176   &lt;nodes&gt;       650/TCP,651/TCP                46s
svc/rethink-service   10.0.0.3     &lt;nodes&gt;       8080/TCP,28015/TCP,29015/TCP   46s
NAME             READY     STATUS    RESTARTS   AGE
po/etcd-x02ou    1/1       Running   0          46s
po/pachd-cqdus   1/1       Running   1          46s
po/rethink-0     1/1       Running   0          46s
  info: 2 completed object(s) was(were) not shown in pods list. Pass --show-all to see all objects.


NAME                                 STATUS    VOLUME             CAPACITY   ACCESSMODES   AGE
pvc/rethink-volume-claim-rethink-0   Bound     rethink-volume-0   1Gi        RWO           46s

$ kubectl run -i --tty --image ubuntu dns-test --restart=Never /bin/sh
...
# nslookup -type=srv rethink-service.default.svc.cluster.local
Server:     10.0.0.10
Address:    10.0.0.10#53

rethink-service.default.svc.cluster.local   service = 10 100 0 3231383531646337.rethink-service.default.svc.cluster.local.
</code></pre>

<p>Here my RethinkDB pet seems to get the name <code>3231383531646337.rethink-service.default.svc.cluster.local</code></p>

<p>2) I created both a non-headless service (for external services to talk to Rethink) and a headless service (for domain name assignment) and I still seem to get random DNS names:</p>

<pre><code>$ kc get all
NAME       DESIRED   CURRENT   READY     AGE
rc/etcd    1         1         1         6m
rc/pachd   1         1         1         6m
NAME                   CLUSTER-IP   EXTERNAL-IP   PORT(S)                        AGE
svc/etcd               10.0.0.59    &lt;none&gt;        2379/TCP,2380/TCP              6m
svc/kubernetes         10.0.0.1     &lt;none&gt;        443/TCP                        3d
svc/pachd              10.0.0.222   &lt;nodes&gt;       650/TCP,651/TCP                6m
svc/rethink-headless   None         &lt;none&gt;                                       6m
svc/rethink-service    10.0.0.30    &lt;nodes&gt;       8080/TCP,28015/TCP,29015/TCP   6m
NAME             READY     STATUS    RESTARTS   AGE
po/etcd-anc7v    1/1       Running   0          6m
po/pachd-i1anr   1/1       Running   1          6m
po/rethink-0     1/1       Running   0          6m
  info: 2 completed object(s) was(were) not shown in pods list. Pass --show-all to see all objects.


NAME                                 STATUS    VOLUME             CAPACITY   ACCESSMODES   AGE
pvc/rethink-volume-claim-rethink-0   Bound     rethink-volume-0   1Gi        RWO           6m

$ kubectl run -i --tty --image ubuntu dns-test --restart=Never /bin/sh
...
# nslookup -type=srv rethink-service.default.svc.cluster.local
Server:     10.0.0.10
Address:    10.0.0.10#53

rethink-service.default.svc.cluster.local   service = 10 100 0 6638393531396237.rethink-service.default.svc.cluster.local.
</code></pre>

<p>Here my RethinkDB pet seems to get the name <code>6638393531396237.rethink-service.default.svc.cluster.local</code> which still seems arbitrary.</p>

<p>My basic questions are: <strong>Do I need to connect the nodes to a headless service, in addition to my non-headless NodePort service, to get stable DNS addresses? Can I even have two services for the same set of nodes? Why do neither of these setups give <code>rethink-0</code> the domain name <code>rethink-0.rethink-&lt;something&gt;.default.svc.cluster.local</code>?</strong></p>

<p>Thank you so much for your help!!!</p>

<p>Edit: two updates:</p>

<p>1) Here's the complete k8s manifest I'm using. It's long, but I'd be happy to extract certain parts if that's helpful: <a href="http://pastebin.com/nm73Xtxi" rel="nofollow noreferrer">http://pastebin.com/nm73Xtxi</a></p>

<p>2) I can't seem to do any DNS resolution related to my headless RethinkDB service, <code>rethink-headless</code>:</p>

<pre><code># nslookup rethink-headless.default
Server:     10.0.0.10
Address:    10.0.0.10#53

** server can't find rethink-headless.default: NXDOMAIN

# nslookup rethink-headless
Server:     10.0.0.10
Address:    10.0.0.10#53

** server can't find rethink-headless: SERVFAIL
</code></pre>

<p>You can also use a headless Service for the DB access between nodes, and a regular Service for external access. </p>

<p>There is no issue having 2 Services pointing to the same nodes for different purposes</p>
