<p>I have an ingress-nginx controller handling traffic to my Kubernetes cluster hosted on GKE. I set it up using helm installation instructions from docs:</p>

<p><a href="https://kubernetes.github.io/ingress-nginx/" rel="noreferrer">Docs here</a></p>

<p>For the most part everything is working, but if I try to set cache related parameters via a <code>server-snippet</code> annotation, all of the served content that should get the cache-control headers comes back as a <code>404</code>.</p>

<p>Here's my <code>ingress-service.yaml</code> file:</p>

<pre class="lang-yaml prettyprint-override"><code>apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-service
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-read-timeout: "4000"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "4000"
    nginx.ingress.kubernetes.io/server-snippet: |
      location ~* \.(js|css|gif|jpe?g|png)$ {
        expires 1M;
        add_header Cache-Control "public";
      }
spec:
  tls:
    - hosts:
        - example.com
      secretName: example-com
  rules:
    - host: example.com
      http:
       paths:
        - path: /
          backend:
            serviceName: client-cluster-ip-service
            servicePort: 5000
        - path: /api/
          backend:
            serviceName: server-cluster-ip-service
            servicePort: 4000
</code></pre>

<p>Again, it's only the resources that are matched by the regex that come back as <code>404</code> (all <code>.js</code> files, <code>.css</code> files, etc.).</p>

<p>Any thoughts on why this would be happening?</p>

<p>Any help is appreciated!</p>

Murcielago
<p>Those <code>location</code> blocks are <a href="https://nginx.org/en/docs/http/ngx_http_core_module.html#location" rel="noreferrer">last and/or longest match wins</a>, and since the ingress <strong>itself</strong> is not serving any such content, the nginx relies on a <a href="https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass" rel="noreferrer"><code>proxy_pass</code> directive</a> pointing at the upstream server. Thus, if you are getting 404s, it's very likely because <em>your</em> <code>location</code> is matching, thus interfering with the <code>proxy_pass</code> one. There's a pretty good chance you'd actually want <a href="https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#configuration-snippet" rel="noreferrer"><code>configuration-snippet:</code></a> instead, likely in combination with <a href="https://nginx.org/en/docs/http/ngx_http_rewrite_module.html#if" rel="noreferrer"><code>if ($request_uri ~* ...) {</code></a> to add the header.</p>

<p>One can try this locally with a trivial nginx.conf pointing at <code>python3 -m http.server 9090</code> or whatever fake upstream target.</p>

<p>Separately, for debugging nginx ingress problems, it is often invaluable to consult its actual <code>nginx.conf</code>, which one can grab from any one of the ingress Pods, and/or consulting the logs of the ingress Pods where nginx will emit helpful debugging text.</p>
