<p>I'm having trouble setting up this scenario. I have a server that's running 4 VMs:</p>
<ul>
<li>k8s master</li>
<li>k8s worker 1</li>
<li>k8s worker 2</li>
<li>vm1</li>
</ul>
<p>I've deployed a Consul cluster with Helm chart into k8s cluster, resulting in 1 server on each worker node, 1 client on each worker node. Here's current config (I'm trying all sorts of things so some might be commented/disabled)</p>
<pre class="lang-yaml prettyprint-override"><code>global:
  name: consul
  enabled: true
  datacenter: dc1
  gossipEncryption:
    autoGenerate: true
  tls:
    enabled: true
    enableAutoEncrypt: true
    verify: true
  acls:
    manageSystemACLs: true
# client:
#   exposeGossipPorts: true
server:
  replicas: 2
  # exposeGossipAndRPCPorts: true
  # ports:
  #   serflan:
  #     port: 9301
  extraConfig: |
    { &quot;log_level&quot;: &quot;debug&quot; }
  exposeService:
    enabled: true
    type: NodePort
    nodePort:
      http: 31500 # 8500 + 23k
      https: 31501 # 8501 + 23k
      grpc: 31503 # 8503 + 23k
      serf: 32301 # 9301 + 23k
      rpc: 31300 # 8300 + 23k
  securityContext:
    runAsNonRoot: false
    runAsUser: 0
  storageClass: nfs-synology-test
connectInject:
  enabled: true
controller:
  enabled: true
syncCatalog:
  enabled: true
dns:
  enabled: true
</code></pre>
<p>This works mostly fine in the cluster itself (I can run a Job that does a <code>dig</code> to a Consul service name and I do get a response). Now I wanted to install another Consul client, this time on the <code>vm1</code> VM and join it into my Consul in K8s cluster.</p>
<p>As you can see in the comments, I've tried exposing gossip and RPC ports as host ports, now instead exposing the server service as NodePort service on given ports.</p>
<p>From the VM I can verify with <code>nmap</code> that those ports are indeed open on TCP, but for the love of all that's holy I can't figure out what to configure in the <code>vm1</code> client. Here's my current config:</p>
<pre class="lang-json prettyprint-override"><code>{
  &quot;server&quot;: false,
  &quot;domain&quot;: &quot;consul&quot;,
  &quot;datacenter&quot;: &quot;dc1&quot;,
  &quot;data_dir&quot;: &quot;/etc/consul/data&quot;,
  &quot;tls&quot;: {
    &quot;defaults&quot;: {
      &quot;ca_file&quot;: &quot;/etc/consul/tls/ca/tls.crt&quot;,
      &quot;verify_incoming&quot;: false,
      &quot;verify_outgoing&quot;: true
    },
    &quot;internal_rpc&quot;: {
      &quot;verify_server_hostname&quot;: true
    }
  },
  &quot;auto_encrypt&quot;: { &quot;tls&quot;: true },
  &quot;encrypt&quot;: &quot;redacted&quot;,
  &quot;log_level&quot;: &quot;DEBUG&quot;,
  &quot;enable_syslog&quot;: true,
  &quot;leave_on_terminate&quot;: true,
  &quot;retry_join&quot;: [
    &quot;192.168.1.207:32301&quot;,
    &quot;192.168.1.208:32301&quot;,
    &quot;10.233.94.138:8300&quot;,
    &quot;10.233.119.94:8300&quot;
  ],
  &quot;advertise_addr&quot;: &quot;192.168.1.230&quot;,
  &quot;bind_addr&quot;: &quot;0.0.0.0&quot;,
  &quot;ports&quot;: { &quot;server&quot;: 31300 },
  &quot;acl&quot;: {
    &quot;tokens&quot;: {
      &quot;agent&quot;: &quot;redacted&quot;,
      &quot;default&quot;: &quot;redacted&quot;
    }
  }
}
</code></pre>
<p>I've taken the value of <code>encrypt</code> from the secret in k8s, same as the <code>tls.crt</code>. I've tried to generate a token with the GUI assigned to <code>client-policy</code>, defined as:</p>
<pre><code>node_prefix &quot;&quot; {
  policy = &quot;write&quot;
}
service_prefix &quot;&quot; {
  policy = &quot;read&quot;
}
</code></pre>
<p>But all to no avail. The client generally fails UDP connections, tries to connect to internal k8s cluster IPs (even without me adding them to <code>retry_join</code>, again just trying), overall geting timeouts and <code>rpc error: lead thread didn't get connection</code>.</p>
<p>I'm out of ideas and I'm at the stage of just trying random ports and configs until I hit jackpots. Can anyone help?</p>

Lukáš Stuchlík
<p>I somewhat figured out my answer so posting if anyone is stuck on the same thing. While I like exposing a service on node ports, it didn't quite work out. The servers were accessible from the <code>32301</code> port (so VM client was able to <em>join</em>), but the server themselves were advertising themselves on the pod IP which was inaccessible from outside.</p>
<p>The solution in my case was:</p>
<ul>
<li>actually use the commented out <code>client.exposeGossipPorts: true</code>, <code>server.exposeGossipAndRPCPorts: true</code> and <code>server.ports.serflan.port: 9301</code></li>
<li><code>retry_join</code> in the config should use the node IP + <code>9301</code> port</li>
<li>the way I set up the TLS cert, tokens and <code>encrypt</code> was probably right</li>
<li>as far as RPC and failing UDP, the problem was in the fact that Kube cluster failed to deploy the servers and clients correctly (Serflan-UDP ports weren't exposed via UDP). When I <code>kubectl patch</code>ed it, it started to work (eventually we fixed the cluster itself instead)</li>
</ul>
