<p>I am attempting to migrate my on premises cluster to GKE. In order to facilitate this transition I need to be able to resolve the names of legacy services.</p>

<p>Assume that the networking/VPN is a solved problem.</p>

<p>Is there are way to do this with GKE currently?</p>

<p>Effectively I am attempting to add a NS to every /etc/resolv.conf</p>

<p>I want to add to what Eric said, and mutate it a bit.</p>

<p>One of the realizations we had during the kubernetes 1.1 "settling period" is that there are not really specs for things like resolv.conf and resolver behavior.  Different resolver libraries do different things, and this was causing pain for our users.</p>

<p>Specifically, some common resolvers assume that all <code>nameserver</code>s are fungible and would break if you had nameservers that handled different parts of the DNS namespace.  We made a decision that for kube 1.2 we will NOT pass multiple <code>nameserver</code> lines into containers.  Instead, we pass only the kube-dns server, which handles <code>cluster.local</code> queries and forwards any other queries to an "upstream" nameserver.</p>

<p>How do we know what "upstream" is?  We use the <code>nameservers</code> of the node.  There is a per-pod dnsPolicy field that governs this choice.  The net result is that containers see a single <code>nameserver</code> in resolv.conf, which we own, and that nameserver handles the whole DNS namespace.</p>

<p>What this practically means is that there's not a great hook for you to interject your own nameserver. You could change the <code>--cluster-dns</code> flag to kubelets to point to your own DNS server, which would then forward to the kube-dns, which would then forward to "upstream".  The problem is that GKE doesn't really support changing flags that way.  If/when the node is updated, the flag will disappear in favor of the default.</p>

<p>Possible solutions:</p>

<ul>
<li><p>Have kubelets read their flags from an in-cluster config.  This is already plan of record, but is not in v1.2</p></li>
<li><p>Have kube-dns take a flag indicating what "upstream" is.  Kube-dns is a "cluster addon" and as such isn't really mutable by end users (we will update it with your cluster and lose your changes).</p></li>
<li><p>Have kube-dns read its flags from an in-cluster config, and take a flag indicating what "upstream" is.  This is a doable idea, but probably not for v1.2 (too late).  It <em>might</em> be possible to patch this into a v1.2.x but it's not really a bugfix, it's a feature.</p></li>
<li><p>Get your own DNS server into the resolv.conf on each node so that kube-dns would use you as upstream.  I don't think GKE has a way to configure this that won't also get lost on node upgrades.  You could write a controller that periodically SSH'ed to VMs and wrote that out, and subsequently checked your kube-dns container for correctness.  Blech.</p></li>
</ul>

<p>I think the right answer is to use in-cluster configmaps to inform either kubelets or DNS (or both).  If you think these might be workable answers (despite the timeframe issues), it would be great if you opened a GitHub issue to discuss.  It will get more visibility there.</p>
