<p>I have a Kubernetes cluster with 2 master nodes and 3 worker nodes. I have used HELM to install consul setup which has 3 consul-servers and 5 consul-clients running.</p>
<p>Here is how the consul server pods and consul client pods are placed on the Kubernetes nodes:</p>
<pre class="lang-sh prettyprint-override"><code>[root@k8masterg2m1 autoinstall]# kubectl get po -o wide | grep consul
consul-consul-4lxtr 1/1 Running 0 103m 192.168.139.139 k8masterg2m1
consul-consul-6wv9w 1/1 Running 0 103m 192.168.118.215 k8workerg2w3
consul-consul-pc562 1/1 Running 0 103m 192.168.108.162 k8workerg2w2
consul-consul-server-0 1/1 Running 0 107m 192.168.118.214 k8workerg2w3
consul-consul-server-1 1/1 Running 0 9m15s 192.168.227.91 k8workerg2w1
consul-consul-server-2 1/1 Running 0 107m 192.168.108.161 k8workerg2w2
consul-consul-tg4kz 1/1 Running 0 103m 192.168.139.72 k8masterg2m2
consul-consul-tj7h5 1/1 Running 0 103m 192.168.227.90 k8workerg2w1
</code></pre>
<p>On the other side I have installed consul client on a local VM, which is on the same networks as the Kubernetes nodes.</p>
<p>From the consul server pods running in Kubernetes, I have used the below command to join the local VM(10.0.20.102).</p>
<pre class="lang-sh prettyprint-override"><code>/ # consul join 10.0.20.102
Successfully joined cluster by contacting 1 nodes.
</code></pre>
<p>I could see the below output in both the VM and consul pods in the Kubernetes:</p>
<pre class="lang-sh prettyprint-override"><code>/ # consul members

Node Address Status Type Build Protocol DC Segment
consul-consul-server-0 192.168.118.214:8301 alive server 1.8.1 2 dc1
consul-consul-server-1 192.168.227.91:8301 alive server 1.8.1 2 dc1
consul-consul-server-2 192.168.108.161:8301 alive server 1.8.1 2 dc1
k8masterg1m2 10.0.20.102:8301 alive client 1.8.1 2 dc1
k8masterg2m1 192.168.139.139:8301 alive client 1.8.1 2 dc1
k8masterg2m2 192.168.139.72:8301 alive client 1.8.1 2 dc1
k8workerg2w1 192.168.227.90:8301 alive client 1.8.1 2 dc1
k8workerg2w2 192.168.108.162:8301 alive client 1.8.1 2 dc1
k8workerg2w3 192.168.118.215:8301 alive client 1.8.1 2 dc1
</code></pre>
<p>Now, when I try to list the services in Kubernetes consul pods it works fine as shown below:</p>
<pre class="lang-sh prettyprint-override"><code>/ # consul catalog services

consul
consul-consul-dns-default
consul-consul-server-default
consul-consul-ui-default
ha-rabbitmq-rabbitmq-ha-default
ha-rabbitmq-rabbitmq-ha-discovery-default
kubernetes-default
vault-agent-injector-svc-default
vault-internal-default
</code></pre>
<p>but, when I try to run the same command in local VM it gives the below error:</p>
<pre class="lang-sh prettyprint-override"><code>[root@k8masterg1m2 autoinstall]# consul catalog services
Error listing services: Unexpected response code: 500 (rpc error getting client: failed to get conn: rpc error: lead thread didn’t get connection)
</code></pre>
<p>Since on the consul agent running in the local VM, it is able to list the members but not services/nodes.
Is this the expected behavior or is there any other configuration which has to be done to get this work.</p>
<p>Also, I wanted to know how the communication happens between consul servers and consul agent which is outside Kubernetes cluster.</p>
<p>Any help is appreciated.</p>
<p>Thanks in Advance!!</p>

<p>It's an odd setup that you want to join the VM. But for everything to work you need to make sure that <a href="https://www.consul.io/docs/install/ports" rel="nofollow noreferrer">all the required ports</a> are exposed on the Consul Server nodes. It sounds like all of those ports are exposed within the Kubernetes cluster but not outside.</p>
<p>You didn't specify how you deployed Consul but you can use <a href="https://kubernetes.io/docs/concepts/services-networking/service/" rel="nofollow noreferrer">Kubernetes Services</a> to expose the ports. In your case, it's likely that maybe port <code>8500</code> is the only exposed from your VM but maybe not the other ports. You have to expose them all if you'd like to enable all the Consul features/functionality.</p>
<p>✌️</p>
