<p>I have a <code>nginx</code> deployment in k8s cluster which proxies my <code>api/</code> calls like this:</p>
<pre><code>server {
  listen 80;

  location / {
    root /usr/share/nginx/html;
    index index.html index.htm;
    try_files $uri $uri/ /index.html =404;
  }

  location /api {
    proxy_pass http://backend-dev/api;
  }
}
</code></pre>
<p>This works most of the time, however sometimes when <code>api</code> pods aren't ready, nginx fails with error:</p>
<pre class="lang-sh prettyprint-override"><code>nginx: [emerg] host not found in upstream &quot;backend-dev&quot; in /etc/nginx/conf.d/default.conf:12
</code></pre>
<p>After couple of hours exploring internets, I found the <a href="https://ilhicas.com/2018/04/14/Nginx-Upstream-Unavalailble-Docker.html" rel="nofollow noreferrer">article</a> which pretty much the same issue. I've tried this:</p>
<pre><code>  location /api {
    set $upstreamName backend-dev;
    proxy_pass http://$upstreamName/api;
  }
</code></pre>
<p>Now nginx returns <strong>502</strong>.
And this:</p>
<pre><code>  location /api {
    resolver 10.0.0.10 valid=10s;
    set $upstreamName backend-dev;
    proxy_pass http://$upstreamName/api;
  }
</code></pre>
<p>Nginx returns <strong>503</strong>.</p>
<p>What's the correct way to fix it on k8s?</p>

<p>If your API pods are not ready, Nginx wouldn't be able to route traffic to them.</p>

<p>From Kubernetes <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/" rel="nofollow noreferrer">documentation</a>:</p>

<blockquote>
  <p>The kubelet uses readiness probes to know when a Container is ready to start accepting traffic. A Pod is considered ready when all of its Containers are ready. One use of this signal is to control which Pods are used as backends for Services. When a Pod is not ready, it is removed from Service load balancers.</p>
</blockquote>

<p>If you are not using liveness or readiness probes, then your pod will be marked as "ready" even if your application running inside the container has not finished it's startup process and is ready to accept traffic.</p>

<p>The relevant section regarding Pods and DNS records can be found <a href="https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/" rel="nofollow noreferrer">here</a></p>

<blockquote>
  <p>Because A records are not created for Pod names, hostname is required for the Pod’s A record to be created. A Pod with no hostname but with subdomain will only create the A record for the headless service (default-subdomain.my-namespace.svc.cluster-domain.example), pointing to the Pod’s IP address. Also, Pod needs to become ready in order to have a record unless publishNotReadyAddresses=True is set on the Service.</p>
</blockquote>

<p><strong>UPDATE:</strong> I would suggest using NGINX as an <a href="https://kubernetes.github.io/ingress-nginx/" rel="nofollow noreferrer">ingress controller</a>.</p>

<p>When you use NGINX as an ingress controller, the NGINX service starts successfully and whenever an <a href="https://kubernetes.io/docs/concepts/services-networking/ingress/#the-ingress-resource" rel="nofollow noreferrer">ingress rule</a> is deployed, the NGINX configuration is <a href="https://kubernetes.github.io/ingress-nginx/how-it-works/" rel="nofollow noreferrer">reloaded on the fly</a>.</p>

<p>This will help you avoid NGINX pod restarts.</p>
