<p>Assuming having 2 separate k3d clusters (namely: <code>vault</code>, <code>dev</code>)
is there is a way to have a distinct URL for each cluster (preferably with https) for example: <code>vault.cluster.internal</code> and <code>dev.cluster.internal</code>
and allow apps deployed in <code>dev.cluster.internal</code> to lookup something or interact with apps in the <code>vault.cluster.internal</code> ?</p>
<p>The cluster definitions are as follows:
<code>dev.yaml</code>:</p>
<pre><code>apiVersion: k3d.io/v1alpha4
kind: Simple
metadata:
  name: dev
servers: 1
agents: 3
network: k3d-cluster
kubeAPI:
  host: &quot;dev.cluster.internal&quot;
  hostIP: &quot;127.0.0.1&quot;
image: rancher/k3s:v1.24.3-k3s1
ports:
  - port: 3000:3000
    nodeFilters:
      - loadbalancer
options:
  k3d:
    wait: true
    timeout: &quot;60s&quot;
  k3s:
    extraArgs:
      - arg: --tls-san=dev.cluster.internal
        nodeFilters:
          - server:*
      - arg: --disable=metrics-server
        nodeFilters:
          - server:*
      - arg: --disable=traefik
        nodeFilters:
          - server:*
  kubeconfig:
    updateDefaultKubeconfig: true
    switchCurrentContext: false
</code></pre>
<p>and the <code>vault.yaml</code>:</p>
<pre><code>apiVersion: k3d.io/v1alpha4
kind: Simple
metadata:
  name: vault
servers: 1
agents: 3
network: k3d-cluster
kubeAPI:
  host: &quot;vault.cluster.internal&quot;
  hostIP: &quot;127.0.0.1&quot;
image: rancher/k3s:v1.24.3-k3s1
ports:
 - port: 8200:8200
    nodeFilters:
      - loadbalancer
options:
  k3d:
    wait: true
    timeout: &quot;60s&quot;
  k3s:
    extraArgs:
      - arg: --tls-san=vault.cluster.internal
        nodeFilters:
          - server:*
      - arg: --disable=metrics-server
        nodeFilters:
          - server:*
      - arg: --disable=traefik
        nodeFilters:
          - server:*
  kubeconfig:
    updateDefaultKubeconfig: true
    switchCurrentContext: false
</code></pre>
<ul>
<li>Can this be done without using service mesh?</li>
<li>Can I update the coredns in the clusters to allow resolving the other cluster host names, and how?</li>
<li>Can this be done with docker network configurations, and how?</li>
</ul>
<p>This is basically to simulate real world clusters (but for local development)</p>

<p>I found 3 solutions for the problem.</p>
<p>The first solution is to add <code>HostAliases</code> section to the <code>dev</code> cluster definition, and make it point to the external IP of the <code>vault</code> cluster loadbalancer:</p>
<p>for example:
you can run the following command on the <code>vault</code> cluster after initializing it</p>
<pre><code>$ kubectl --context k3d-vault --namespace vault get services
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     ...
...
vault        LoadBalancer   10.43.34.131   172.24.0.3      ...
                                           ^^^^^^^^^^
...
</code></pre>
<p><code>dev.yaml</code> would be</p>
<pre><code>#...
ports:
  - port: 3000:3000
    nodeFilters:
      - loadbalancer
hostAliases:
  - ip: 172.24.0.3
    hostnames:
      - vault.cluster.internal
#...
</code></pre>
<pre class="lang-bash prettyprint-override"><code># (alternatively, this can be automated using the following command without editing `dev.yaml` file)
$ KMS_IP=$(kubectl --context k3d-vault --namespace vault get services | grep LoadBalancer | awk -F &quot; &quot; '{ print $4 }')
$ k3d cluster create --config dev.yaml --host-alias $KMS_IP:vault.cluster.internal
</code></pre>
<p>this solution allow resolving of hostname (as you would expect in a production cluster)...</p>
<p>The second solution works similarly but using <code>docker network inspect k3d-cluster</code> (where <code>k3d-cluster</code> is the docker network name in cluster definition)</p>
<p>Similarly, run <code>docker network inspect k3d-cluster</code> and note down the IP of the loadbalancer subnet defined by docker:</p>
<pre><code>...
&quot;cad3f3XXXXXX&quot;: {
    &quot;Name&quot;: &quot;k3d-vault-serverlb&quot;,
    &quot;EndpointID&quot;: &quot;47d5XXXX&quot;
    &quot;MacAddress&quot;: &quot;02:42:ac:18:00:04&quot;,
    &quot;IPv4Address&quot;: &quot;172.24.0.4/16&quot;,  #&lt;&lt;&lt; This IP can be used in dev cluster HostAliases
    &quot;IPv6Address&quot;: &quot;&quot;
}
...
</code></pre>
<p>The last solution is simpler but less flexible.
it uses <code>host.k3d.internal</code> as the name for the other cluster (allowing to resolve it) but you have to take care of port mapping as all of the clusters would be resolving to use the same URL for the services (which isn't ideal, but easy enough to test multi-cluster communication/bugs/etc).
In other words, configure the <code>dev</code> cluster <code>VAULT_ADDR</code> to be <code>host.k3d.internal:8200</code> instead of <code>vault.cluster.internal:8200</code>
This is not flexible with TLS/HTTPS (AFAIK).</p>
