<p>I have a Kubernetes cluster that I setup with <a href="https://coreos.com/kubernetes/docs/latest/kubernetes-on-aws.html" rel="noreferrer">kube-aws</a>. I'm trying to run a custom NGINX configuration which uses DNS resolutions to proxy_pass. Here is the NGINX block of code</p>

<pre><code>location /api/v1/lead {
  resolver 10.3.0.10 ipv6=off;
  set $container lead-api;
  proxy_pass http://$container:3000;
}
</code></pre>

<p>10.3.0.10 comes from the cluster IP of the DNS service found in Kubernetes. I've also tried 127.0.0.11 which is what we use in the docker-compose/docker environments. </p>

<pre><code>$ kubectl describe --namespace=kube-system service kube-dns
Name:                   kube-dns
Namespace:              kube-system
Labels:                 k8s-app=kube-dns
                        kubernetes.io/cluster-service=true
                        kubernetes.io/name=KubeDNS
Selector:               k8s-app=kube-dns
Type:                   ClusterIP
IP:                     10.3.0.10
Port:                   dns     53/UDP
Endpoints:              10.2.26.61:53
Port:                   dns-tcp 53/TCP
Endpoints:              10.2.26.61:53
Session Affinity:       None
</code></pre>

<p>This configuration works well on three different environments which use docker-compose. However I get the following error in the NGINX logs of the Kubernetes cluster</p>

<blockquote>
  <p>[error] 9#9: *20 lead-api could not be resolved (2: Server failure), client: 10.2.26.0, server: , request: "GET /api/v1/lead/661DF757-722B-41BB-81BD-C7FD398BBC88 HTTP/1.1"</p>
</blockquote>

<p>If I run nslookup within the NGINX pod I can resolve the host with the same dns server:</p>

<pre><code>$ kubectl exec nginx-1855584872-kdiwh -- nslookup lead-api
Server:         10.3.0.10
Address:        10.3.0.10#53

Name:   lead-api.default.svc.cluster.local
Address: 10.3.0.167
</code></pre>

<p>I don't know if it matters or not, but notice the "server" part of the error is empty. When I look at the pod logs for dnsmasq I don't see anything relevant. If I change the NGINX block to hardcode the proxy_pass then it resolves fine. However, I have other configurations that require dynamic proxy names. I could hard code every upstream this way, but I want to know how to make the DNS resolver work.</p>

<pre><code>location /api/v1/lead {
  proxy_pass http://lead-api:3000;
}
</code></pre>

<p>Resolving the name fails because you need to use the Full Qualified Domain name. That is, you should use:</p>

<p><code>lead-api.&lt;namespace&gt;.svc.cluster.local</code></p>

<p>not just </p>

<p><code>lead-api</code></p>

<p>Using just the hostname will usually work because in kubernetes the <code>resolv.conf</code> is configured with search domains so that you don't usually need to provide a service's FQDN. e.g:</p>

<pre><code>search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.3.240.10
options ndots:5
</code></pre>

<p>However, specifying the FQDN is necessary when you tell nginx to use a custom resolver because it does not get the benefit of these domain search specs.</p>
