<p>I've set up a k8s cluster (1 bare metal node for now, which is both master and worker). I've also set up Nginx ingress controller as described here: <a href="https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-manifests/" rel="nofollow noreferrer">https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-manifests/</a> Below are the exact steps:</p>
<ol>
<li><code>kubectl apply -f common/ns-and-sa.yaml</code> <a href="https://github.com/nginxinc/kubernetes-ingress/blob/release-1.11/deployments/common/ns-and-sa.yaml" rel="nofollow noreferrer">https://github.com/nginxinc/kubernetes-ingress/blob/release-1.11/deployments/common/ns-and-sa.yaml</a> (no modifications)</li>
<li><code>kubectl apply -f rbac/rbac.yaml</code> <a href="https://github.com/nginxinc/kubernetes-ingress/blob/release-1.11/deployments/rbac/rbac.yaml" rel="nofollow noreferrer">https://github.com/nginxinc/kubernetes-ingress/blob/release-1.11/deployments/rbac/rbac.yaml</a> (no modifications)</li>
<li><code>kubectl apply -f common/default-server-secret.yaml</code> <a href="https://github.com/nginxinc/kubernetes-ingress/blob/release-1.11/deployments/common/default-server-secret.yaml" rel="nofollow noreferrer">https://github.com/nginxinc/kubernetes-ingress/blob/release-1.11/deployments/common/default-server-secret.yaml</a> (no modifications)</li>
<li><code>kubectl apply -f common/nginx-config.yaml</code> <a href="https://github.com/nginxinc/kubernetes-ingress/blob/release-1.11/deployments/common/nginx-config.yaml" rel="nofollow noreferrer">https://github.com/nginxinc/kubernetes-ingress/blob/release-1.11/deployments/common/nginx-config.yaml</a> Modified file:</li>
</ol>
<pre class="lang-yaml prettyprint-override"><code>kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-config
  namespace: nginx-ingress
data:
  ignore-invalid-headers: &quot;false&quot;
  use-forwarded-headers: &quot;true&quot;
  forwarded-for-header: &quot;CF-Connecting-IP&quot;
  proxy-real-ip-cidr: &quot;...IPs go here...&quot;
</code></pre>
<ol start="5">
<li><code>kubectl apply -f common/ingress-class.yaml</code> <a href="https://github.com/nginxinc/kubernetes-ingress/blob/release-1.11/deployments/common/ingress-class.yaml" rel="nofollow noreferrer">https://github.com/nginxinc/kubernetes-ingress/blob/release-1.11/deployments/common/ingress-class.yaml</a> Modified file:</li>
</ol>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: networking.k8s.io/v1beta1
kind: IngressClass
metadata:
  name: nginx
  annotations:
    ingressclass.kubernetes.io/is-default-class: &quot;true&quot;
spec:
  controller: nginx.org/ingress-controller
</code></pre>
<ol start="6">
<li>These commands:</li>
</ol>
<pre class="lang-sh prettyprint-override"><code>kubectl apply -f common/crds/k8s.nginx.org_virtualservers.yaml
kubectl apply -f common/crds/k8s.nginx.org_virtualserverroutes.yaml
kubectl apply -f common/crds/k8s.nginx.org_transportservers.yaml
kubectl apply -f common/crds/k8s.nginx.org_policies.yaml
</code></pre>
<p>No modifications, links:</p>
<ul>
<li><a href="https://github.com/nginxinc/kubernetes-ingress/blob/release-1.11/deployments/common/crds/k8s.nginx.org_virtualservers.yaml" rel="nofollow noreferrer">https://github.com/nginxinc/kubernetes-ingress/blob/release-1.11/deployments/common/crds/k8s.nginx.org_virtualservers.yaml</a></li>
<li><a href="https://github.com/nginxinc/kubernetes-ingress/blob/release-1.11/deployments/common/crds/k8s.nginx.org_virtualserverroutes.yaml" rel="nofollow noreferrer">https://github.com/nginxinc/kubernetes-ingress/blob/release-1.11/deployments/common/crds/k8s.nginx.org_virtualserverroutes.yaml</a></li>
<li><a href="https://github.com/nginxinc/kubernetes-ingress/blob/release-1.11/deployments/common/crds/k8s.nginx.org_transportservers.yaml" rel="nofollow noreferrer">https://github.com/nginxinc/kubernetes-ingress/blob/release-1.11/deployments/common/crds/k8s.nginx.org_transportservers.yaml</a></li>
<li><a href="https://github.com/nginxinc/kubernetes-ingress/blob/release-1.11/deployments/common/crds/k8s.nginx.org_policies.yaml" rel="nofollow noreferrer">https://github.com/nginxinc/kubernetes-ingress/blob/release-1.11/deployments/common/crds/k8s.nginx.org_policies.yaml</a></li>
</ul>
<ol start="7">
<li><code>kubectl apply -f daemon-set/nginx-ingress.yaml</code> <a href="https://github.com/nginxinc/kubernetes-ingress/blob/release-1.11/deployments/daemon-set/nginx-ingress.yaml" rel="nofollow noreferrer">https://github.com/nginxinc/kubernetes-ingress/blob/release-1.11/deployments/daemon-set/nginx-ingress.yaml</a> (no modifications)</li>
</ol>
<p>I've also set up cert-manager, which works fine (pretty sure this does not matter much).</p>
<p>Now, when I create some Ingress resource, it <em>almost</em> works. I can access it from the outer Internet, certificate issuing works, etc. But ConfigMap (common/nginx-config.yaml) is not applied, and annotations like <code>nginx.org/rewrite-target: /$1</code> are not applied, too.</p>
<p>Example:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-com
  namespace: example-com
  annotations:
    nginx.org/rewrite-target: /$1
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - example.com
    secretName: example-com-tls
  rules:
  - host: example.com
    http:
      paths:
      - path: /api/(.*)
        pathType: ImplementationSpecific
        backend:
          service:
            name: api
            port:
              number: 80
      - path: /(.*)
        pathType: ImplementationSpecific
        backend:
          service:
            name: frontend
            port:
              number: 80
</code></pre>
<p>Real domain names are used, of course. I get 404 nginx error in this example. In other Ingress I pass <code>/proxy-body-size</code> annotation, which does not work also (can not upload large files).</p>
<p>I've <code>exec</code>ed into ingress controller pod with <code>kubectl -n nginx-ingress exec -it nginx-ingress-snjjp bash</code> and looked at files in <code>/etc/nginx/conf.d</code>. None of the files contained configuration specified in ConfigMap or annotations.</p>
<p>This is what it look like (I removed extra blank lines and replaced domain names):</p>
<pre><code># configuration for example-com/example-com

upstream example-com-example-com-example.com-api-80 {
        zone example-com-example-com-example.com-api-80 256k;
        random two least_conn;

        server 10.32.0.4:80 max_fails=1 fail_timeout=10s max_conns=0;
}
upstream example-com-example-com-example.com-frontend-80 {
        zone example-com-example-com-example.com-frontend-80 256k;
        random two least_conn;

        server 10.32.0.27:80 max_fails=1 fail_timeout=10s max_conns=0;
}

server {
        listen 80;

        listen 443 ssl;

        ssl_certificate /etc/nginx/secrets/example-com-example-com-tls;
        ssl_certificate_key /etc/nginx/secrets/example-com-example-com-tls;

        server_tokens on;

        server_name example.com;

        set $resource_type &quot;ingress&quot;;
        set $resource_name &quot;example-com&quot;;
        set $resource_namespace &quot;example-com&quot;;

        if ($scheme = http) {
                return 301 https://$host:443$request_uri;
        }

        location /api/(.*) {
                set $service &quot;api&quot;;

                proxy_http_version 1.1;

                proxy_connect_timeout 60s;
                proxy_read_timeout 60s;
                proxy_send_timeout 60s;
                client_max_body_size 1m;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Host $host;
                proxy_set_header X-Forwarded-Port $server_port;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_buffering on;

                proxy_pass http://example-com-example-com-example.com-api-80;
        }
        location /(.*) {
                set $service &quot;frontend&quot;;

                proxy_http_version 1.1;

                proxy_connect_timeout 60s;
                proxy_read_timeout 60s;
                proxy_send_timeout 60s;
                client_max_body_size 1m;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Host $host;
                proxy_set_header X-Forwarded-Port $server_port;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_buffering on;

                proxy_pass http://example-com-example-com-example.com-frontend-80;
        }
}
</code></pre>
<p>I also tried <code>nginx.ingress.kubernetes.io/</code> annotations (I'm not a pro as you can see, and it was what I googled). No success.</p>
<p>I am updating my cluster, and with the older version of k8s (1.15 I think it was) everything worked a couple of days ago. I used the exact same configuration for every service, except ingress controller, of course.</p>
<p>Any ideas?</p>

<p>I've found out what is wrong. I was using Kubernetes Nginx Ingress Controller <a href="https://kubernetes.github.io/ingress-nginx/" rel="noreferrer">https://kubernetes.github.io/ingress-nginx/</a> with my old setup, and now I am using Nginx Ingress Controller <a href="https://www.nginx.com/products/nginx-ingress-controller/" rel="noreferrer">https://www.nginx.com/products/nginx-ingress-controller/</a> These implementations have different annotations (the latter is missing many useful annotations). This is really very confusing, as configuration is similar and one may think these are the same.</p>
