<p>I am trying to enable a multi cluster CockroachDB spawning 3 k8s clusters connected with Cilium Cluster Mesh. The idea of having a multi cluster CockroachDB is described on <a href="https://www.cockroachlabs.com/blog/cockroachdb-kubernetes-cilium/" rel="nofollow noreferrer">cockroachlabs.com - 1</a>, <a href="https://www.cockroachlabs.com/docs/stable/orchestrate-cockroachdb-with-kubernetes-multi-cluster" rel="nofollow noreferrer">2</a>. Given the fact that the article calls for a <a href="https://github.com/cockroachlabs-field/cockroachdb-multiregion-k3s-ciliumcni-azure/blob/master/example-configmap-eastus.yaml" rel="nofollow noreferrer">change in CoreDNS ConfigMap</a>, instead of using Cilium <a href="https://docs.cilium.io/en/latest/network/clustermesh/services/#load-balancing-service-discovery" rel="nofollow noreferrer">global-services</a> feels suboptimal.</p>
<p>Therefore the question arises, how to enable a multi cluster CockroachDB in a Cilium Cluster Mesh environment, using Cilium global services instead of hacking CoreDNS ConfigMap ?</p>
<p>With CockroachDB installed via helm, it deploys a <a href="https://github.com/cockroachdb/helm-charts/blob/master/cockroachdb/templates/statefulset.yaml#L171" rel="nofollow noreferrer">StatefulSet</a> with a carefully crafted <code>--join</code> parameter. It contains FQDNs of CockroachDB pods that are to join the cluster.</p>
<p>The pod FQDNs come from <a href="https://github.com/cockroachdb/helm-charts/blob/master/cockroachdb/templates/service.discovery.yaml" rel="nofollow noreferrer">service.discover</a> that is created with <code>clusterIP: None</code> and</p>
<blockquote>
<p>(...) only exists to create DNS entries for each pod in the StatefulSet such that they can resolve each other's IP addresses.</p>
</blockquote>
<p>The discovery service automatically registers DNS entries for all pods within the StatefulSet, so that they can be <a href="https://github.com/cockroachdb/helm-charts/blob/master/cockroachdb/templates/statefulset.yaml#L177" rel="nofollow noreferrer">easily referenced</a></p>
<p>Can a similar <em>discovery service</em> or alternative be created for a StatefulSet running on a <em>remote</em> cluster ? So that with cluster mesh enabled, pods J,K,L in cluster Β could be reached from pods X,Y,Z in cluster Α by their FQDN ?</p>
<p>As suggested in <a href="https://devops.stackexchange.com/questions/11997/create-service-per-pod-in-statefulset">create-service-per-pod-in-statefulset</a>, one could create services like</p>
<pre><code>{{- range $i, $_ := until 3 -}}
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.alpha.kubernetes.io/tolerate-unready-endpoints: &quot;true&quot;
    io.cilium/global-service: 'true'
    service.cilium.io/affinity: &quot;remote&quot;
  labels:
    app.kubernetes.io/component: cockroachdb
    app.kubernetes.io/instance: dbs
    app.kubernetes.io/name: cockroachdb
  name: dbs-cockroachdb-remote-{{ $i }}
  namespace: dbs
spec:
  ports:
  - name: grpc
    port: 26257
    protocol: TCP
    targetPort: grpc
  - name: http
    port: 8080
    protocol: TCP
    targetPort: http
  selector:
    app.kubernetes.io/component: cockroachdb
    app.kubernetes.io/instance: dbs
    app.kubernetes.io/name: cockroachdb
    statefulset.kubernetes.io/pod-name: cockroachdb-{{ $i }}
  type: ClusterIP
  clusterIP: None
  publishNotReadyAddresses: true
---
kind: Service
apiVersion: v1
metadata:
  name: dbs-cockroachdb-public-remote-{{ $i }}
  namespace: dbs
  labels:
    app.kubernetes.io/component: cockroachdb
    app.kubernetes.io/instance: dbs
    app.kubernetes.io/name: cockroachdb
  annotations:
    io.cilium/global-service: 'true'
    service.cilium.io/affinity: &quot;remote&quot;
spec:
  ports:
  - name: grpc
    port: 26257
    protocol: TCP
    targetPort: grpc
  - name: http
    port: 8080
    protocol: TCP
    targetPort: http
  selector:
    app.kubernetes.io/component: cockroachdb
    app.kubernetes.io/instance: dbs
    app.kubernetes.io/name: cockroachdb
{{- end -}}
</code></pre>
<p>So that they resemble the original <a href="https://github.com/cockroachdb/helm-charts/blob/master/cockroachdb/templates/service.discovery.yaml" rel="nofollow noreferrer">service.discovery</a> and <a href="https://github.com/cockroachdb/helm-charts/blob/master/cockroachdb/templates/service.public.yaml" rel="nofollow noreferrer">service.public</a></p>
<p>However, despite the presence of cilium annotations</p>
<pre><code>io.cilium/global-service: 'true'
service.cilium.io/affinity: &quot;remote&quot;
</code></pre>
<p>services look bound to the <em>local</em> k8s cluster, resulting in CockroachDB consisting of 3 instead of 6 nodes. (3 in cluster A + 3 in cluster B)</p>
<p><strong>hubble</strong>:
<a href="https://i.stack.imgur.com/7blRC.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/7blRC.png" alt="no cross connect" /></a>
<strong>CockroachDB</strong>:
<a href="https://i.stack.imgur.com/0b9lD.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/0b9lD.png" alt="CockroachDB dash" /></a></p>
<p>It does not matter how which service (<code>dbs-cockroachdb-public-remote-X</code>, or <code>dbs-cockroachdb-remote-X</code>) I use in my <a href="https://github.com/cockroachdb/helm-charts/blob/master/cockroachdb/values.yaml#L61" rel="nofollow noreferrer"><code>--join</code> command overwrite</a></p>
<pre><code>    join:
      - dbs-cockroachdb-0.dbs-cockroachdb.dbs:26257
      - dbs-cockroachdb-1.dbs-cockroachdb.dbs:26257
      - dbs-cockroachdb-2.dbs-cockroachdb.dbs:26257
      - dbs-cockroachdb-public-remote-0.dbs:26257
      - dbs-cockroachdb-public-remote-1.dbs:26257
      - dbs-cockroachdb-public-remote-2.dbs:26257
</code></pre>
<p>The result is the same, 3 nodes instead of 6.</p>
<p>Any ideas?</p>

Krystian Marek
<p>Apparently due to <a href="https://github.com/cilium/cilium/issues/7070" rel="nofollow noreferrer">7070</a>, patching CoreDNS ConfigMap is the most reasonable thing we can do. In the comments of that bug, an <a href="https://faun.pub/connecting-multiple-kubernetes-clusters-on-vsphere-with-the-cilium-cluster-mesh-964f95267df4" rel="nofollow noreferrer">article is mentioned</a>, that provides additional context.</p>
<p>My twist to this story is that I updated the config map with kubernetes plugin config:</p>
<pre><code>apiVersion: v1
data:
  Corefile: |-
    saturn.local {
      log
      errors
      kubernetes saturn.local {
        endpoint https://[ENDPOINT]
        kubeconfig [PATH_TO_KUBECONFIG]
      }
    }
    rhea.local {
      ...
</code></pre>
<p>So that I could resolve other names as well.
In my setup, each cluster has its own <code>domain.local</code>. <code>PATH_TO_KUBECONFIG</code> is a plane kubeconfig file. Generic secret has to be created in <code>kube-system</code> namespace and the secret volume has to be mounted under coredns deployment.</p>
<p><a href="https://i.stack.imgur.com/QIOBu.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/QIOBu.png" alt="it works" /></a></p>
