<p>I have an HPC cluster application where I am looking to replace MPI and our internal cluster management software with a combination of Kubernetes and some middleware, most likely ZMQ or RabbitMQ.</p>

<p>I'm trying to design how best to do peer discovery on this system using Kubernetes' service discovery. </p>

<p>I know Kubernetes can provide a DNS name for a given service, and that's great, but is there a way to also dynamically discover ports? </p>

<p>For example, assuming I replaced the MPI middleware with ZeroMQ, I would need a way for ranks (processes on the cluster) to find each other. I know I could simply have the ranks issue service creation messages to the Kubernetes discovery mechanism and get a hostname like myapp_mypid_rank_42 fairly easily, but how would I handle the port?</p>

<p>If possible, it would be great if I could just do:</p>

<pre><code>zmqSocket.connect("tcp://myapp_mypid_rank_42");
</code></pre>

<p>but I don't think that would work since I have no port number information from DNS.</p>

<p>How can I have Kubernetes service discovery also provide a port in as simple a manner as possible to allow ranks in the cluster to discover each other?</p>

<p>Note: The registering process knows its port and can register it with the K8s service discovery daemon. The problem is a quick and easy way to get that port number back for the processes that want it. The question I'm asking is whether or not there is a mechanism <em>as simple</em> as a DNS host name, or will I need to explicitly query both hostname <em>and</em> port number from the k8s daemon rather than simply building a hostname based on some agreed upon rule (like building a string from myapp_mypid_myrank)?</p>

<p>Turns out the best way to do this is with a DNS SRV record:</p>

<p><a href="https://kubernetes.io/docs/concepts/services-networking/service/#discovering-services" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/services-networking/service/#discovering-services</a>
<a href="https://en.wikipedia.org/wiki/SRV_record" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/SRV_record</a></p>

<p>A DNS SRV record provides both a hostname/IP <em>and</em> a port for a given request.</p>

<p>Luckily, Kubernetes service discovery supports SRV records and provides them on the cluster's DNS.</p>
