<blockquote>
  <p>Relates to
  <a href="https://stackoverflow.com/questions/31664060/how-to-call-a-service-exposed-by-a-kubernetes-cluster-from-another-kubernetes-cl?rq=1">How to call a service exposed by a Kubernetes cluster from another Kubernetes cluster in same project</a>.</p>
  
  <p>Asking again since Kubernetes has been changes a lot since July.</p>
</blockquote>

<p><strong>Context:</strong></p>

<p>I'm working on an infrastructure with multiple clusters serving different purposes, e.g.:</p>

<ul>
<li>Cluster A runs services/apps creating data for consumption</li>
<li>Cluster B runs services/apps consuming data created by apps in cluster A</li>
<li>Cluster C runs data services like Redis, Memcache, etc.</li>
</ul>

<p>All clusters are in the <code>default</code> namespace.</p>

<p><strong>Problem:</strong></p>

<p>In Kubernetes, each cluster gets its own kubernetes (in the <code>default</code> namespace) and kube-dns (in the <code>kube-system</code> namespace) service with a different IP.</p>

<p>What happens with this setup is that, services in cluster A and B above can't discover (in service discovery terminology), let's say, Redis in cluster C. So a <code>nslookup redis.default.svc.cluster.local</code> from one of the services in cluster A/B comes back with <code>** server can't find redis.default.svc.cluster.local: NXDOMAIN</code>. <em>Note:</em> This works from within cluster C.</p>

<p>I've read as many documents as I found about kube-dns, and pretty much all assume one cluster setup.</p>

<p><strong>Clusters info:</strong></p>

<p>Here are <code>/etc/resolv.conf</code> from two different clusters showing DNS nameservers with no common kube-dns ancestor:</p>

<p>Cluster A:</p>

<pre><code>nameserver 10.67.240.10
nameserver 169.254.169.254
nameserver 10.240.0.1
search default.svc.cluster.local svc.cluster.local cluster.local c.project-name.internal. 1025230764914.google.internal. google.internal.
</code></pre>

<p>Cluster C:</p>

<pre><code>nameserver 10.91.240.10
nameserver 169.254.169.254
nameserver 10.240.0.1
search default.svc.cluster.local svc.cluster.local cluster.local c.project-name.internal. google.internal.
options ndots:5
</code></pre>

<p>Both clusters have these services running with their respective IPs for their cluster in the <code>kube-system</code> namespace:</p>

<pre><code>NAME                  LABELS                                                                           SELECTOR           
kube-dns              k8s-app=kube-dns,kubernetes.io/cluster-service=true,kubernetes.io/name=KubeDNS   k8s-app=kube-dns
kube-ui               k8s-app=kube-ui,kubernetes.io/cluster-service=true,kubernetes.io/name=KubeUI     k8s-app=kube-ui
monitoring-heapster   kubernetes.io/cluster-service=true,kubernetes.io/name=Heapster                   k8s-app=heapster
</code></pre>

<p>What is the ideal fix/update to this setup that can get the shared services discovered across all Kubernetes clusters in a GCE environment?</p>

<p>This is one of the large problems that Kubernetes is trying to solve with <a href="https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/federation.md#cross-cluster-service-discovery" rel="nofollow noreferrer">Cross-Cluster Service Discovery</a> as a part of the Cluster Federation plans. You can also check out/contribute to the <a href="https://groups.google.com/forum/#!forum/kubernetes-sig-federation" rel="nofollow noreferrer">Federation SIG</a>.</p>

<p>If you've used one of the <s>hacks </s>solutions described <a href="https://stackoverflow.com/questions/31664060/how-to-call-a-service-exposed-by-a-kubernetes-cluster-from-another-kubernetes-cl?rq=1">here</a>, you might be able to hack up your <code>/etc/resolve.conf</code> to also search the nameserver from the other cluster. Be careful, because this may run you into truncation issues.</p>

<p>You might also be able to modify the <a href="https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/skydns-rc.yaml.in" rel="nofollow noreferrer">sky-dns RC</a> for your clusters to include an extra kube2sky pod that points at the other cluster's kubernetes service (I haven't tried this, or thought through all of the implications).</p>

<p>Neither of the two hacks I've described above would prevent name collision, so you'd have to manually prevent that.</p>
