<p>So, I would like to have <code>nginx</code> resolve hostnames for backends at request time.  I expect to get <code>HTTP 502 Bad Gateway</code> when back-end service is down and I expect service response, when it's up.</p>

<p>I use <code>nginx:1.15-alpine</code> image for <code>nginx</code> and here is what I have in it's config:</p>

<pre><code>server {

  resolver kube-dns.kube-system.svc.cluster.local valid=5s;

  server_name  mysystem.com;
  listen       80;

  client_max_body_size 20M;

  location = /nginx_status {
      stub_status on;
      access_log off;
  }

  # Services configuration

  location ~ /my-service/ {
      set $service_endpoint http://my-service.namespace:8080;
      proxy_pass $service_endpoint$request_uri;
      include includes/defaults-inc.conf;
      include includes/proxy-inc.conf;
  }

}
</code></pre>

<p>So, when I make the request to the nginx, I get 502 Bad Gateway response. Nginx's log say the name is not found:</p>

<pre><code>2018/06/28 19:49:18 [error] 7#7: *1 my-service.namespace could not be resolved (3: Host not found), client: 10.44.0.1, server: mysystem.com, request: "GET /my-service/version HTTP/1.1", host: "35.229.17.63:8080"
</code></pre>

<p>However, when I log into the container with shell (<code>kubectl exec ... -- sh</code>) and test the DNS resolution, it works perfectly.</p>

<pre><code> # nslookup my-service.namespace kube-dns.kube-system.svc.cluster.local
Server:    10.47.240.10
Address 1: 10.47.240.10 kube-dns.kube-system.svc.cluster.local

Name:      my-service.namespace
Address 1: 10.44.0.75 mysystem-namespace-mysystem-namespace-my-service-0.my-service.namespace.svc.cluster.local
</code></pre>

<p>Moreover,  I can <code>wget http://my-service.namespace:8080/</code> and get a response.</p>

<p>Why nginx cannot resolve the hostname?  </p>

<p><strong>Update:  How I managed to resolve it:</strong></p>

<p>In <code>nginx.conf</code> at the <code>server</code> level I have added a resolver setting:</p>

<pre><code>resolver kube-dns.kube-system.svc.cluster.local valid=10s;
</code></pre>

<p>Then I used a FQDN in <code>proxy_pass</code>:  </p>

<pre><code>proxy_pass http://SERVICE-NAME.YOUR-NAMESPACE.svc.cluster.local:8080;
</code></pre>

<p>It fails because you need to use the FQDN to Resolve the name.</p>

<p>Using just the hostname will usually work because in kubernetes the resolv.conf is configured with search domains so that you don't usually need to provide a service's FQDN.</p>

<p>However, specifying the FQDN is necessary when you tell nginx to use a custom name server because it does not get the benefit of these domain search specs.</p>

<p>In nginx.conf added at the server level:</p>

<pre><code>resolver kube-dns.kube-system.svc.cluster.local valid=10s;
</code></pre>

<p>Then used a FQDN in proxy_pass:</p>

<pre><code>proxy_pass http://SERVICE-NAME.YOUR-NAMESPACE.svc.cluster.local:8080;
</code></pre>
