<p>I have deployed ingress-nginx helm chart ( 3.20.1 , <a href="https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx" rel="nofollow noreferrer">https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx</a> ) into k8s cluster.</p>
<p>Some of the Ingresses configured for applications use basic auth - which works as expected.
I.e. I can access some applications in the cluster without basic auth ( as configured )
and some applications require dedicated basic auth credentials ( as configured ).</p>
<p>But trying an unknown/unsupported URL/Path is handled awkward by default-backend.
Somehow the default backend is now requiring basic auth too ( as one of the ingress with basic-auth ) ?</p>
<p>How can I find out why/where this basic auth kicks in when it shouldn't ?
I do want default-backend to serve error pages for 404 without basic auth !</p>
<pre><code>$ kubectl -n ingress get svc
NAME                                 TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.0.235.5     xx.yy.zzz.zzz   80:31294/TCP,443:31732/TCP   39d
ingress-nginx-controller-admission   ClusterIP      10.0.190.204   &lt;none&gt;          443/TCP                      39d
ingress-nginx-controller-metrics     ClusterIP      10.0.21.184    &lt;none&gt;          9913/TCP                     12d
ingress-nginx-defaultbackend         ClusterIP      10.0.109.195   &lt;none&gt;          80/TCP                       39d
</code></pre>
<p>Currently deployed Ingresses:</p>
<pre><code>$ kubectl get ingress -A
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
NAMESPACE       NAME                                      CLASS    HOSTS                ADDRESS         PORTS     AGE
dev-app-a       app-a                                     &lt;none&gt;   my-host.com          xx.yy.zzz.zzz   80, 443   46d
dev-jaeger      jaeger                                    &lt;none&gt;   my-host.com          xx.yy.zzz.zzz   80, 443   47d
dev-app-b       app-b                                     &lt;none&gt;   my-host.com          xx.yy.zzz.zzz   80, 443   17d
dev-app-c       app-c                                     &lt;none&gt;   my-host.com          xx.yy.zzz.zzz   80, 443   12d
dev-app-d       app-d                                     &lt;none&gt;   my-host.com          xx.yy.zzz.zzz   80, 443   4d22h
prom            prometheus-grafana                        &lt;none&gt;   my-other-host.com    xx.yy.zzz.zzz   80, 443   20d
prom            prometheus-kube-prometheus-alertmanager   &lt;none&gt;   my-other-host.com    xx.yy.zzz.zzz   80, 443   20d
prom            prometheus-kube-prometheus-prometheus     &lt;none&gt;   my-other-host.com    xx.yy.zzz.zzz   80, 443   20d
stage-app-a     app-a                                     &lt;none&gt;   my-third-host.com    xx.yy.zzz.zzz   80, 443   39d
stage-jaeger    jaeger                                    &lt;none&gt;   my-third-host.com    xx.yy.zzz.zzz   80, 443   39d
</code></pre>
<p>Ingress details:</p>
<pre><code>apiVersion: v1
items:
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/enable-cors: &quot;false&quot;
      nginx.ingress.kubernetes.io/proxy-buffering: &quot;off&quot;
    labels:
      app.kubernetes.io/name: app-a
    name: app-a
    namespace: foo-app-a
  spec:
    rules:
    - host: my-host.com
      http:
        paths:
        - backend:
            serviceName: app-a-http
            servicePort: http
          path: /abc/
          pathType: ImplementationSpecific
    tls:
    - hosts:
      - my-host.com
      secretName: app-a-tls
  status:
    loadBalancer:
      ingress:
      - ip: xx.xx.xx.xx
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/add-base-url: &quot;true&quot;
      nginx.ingress.kubernetes.io/auth-realm: Authentication Required - Jaeger UI
      nginx.ingress.kubernetes.io/auth-secret: jaeger-basic-auth
      nginx.ingress.kubernetes.io/auth-type: basic
      nginx.ingress.kubernetes.io/enable-cors: &quot;false&quot;
      nginx.ingress.kubernetes.io/from-to-www-redirect: &quot;true&quot;
      nginx.ingress.kubernetes.io/proxy-buffering: &quot;off&quot;
    labels:
      app.kubernetes.io/name: jaeger
    name: jaeger
    namespace: dev-jaeger
  spec:
    rules:
    - host: my-host.com
      http:
        paths:
        - backend:
            serviceName: jaeger
            servicePort: http-ui
          path: /jaeger/
          pathType: ImplementationSpecific
    tls:
    - hosts:
      - my-host.com
      secretName: jaeger-tls
  status:
    loadBalancer:
      ingress:
      - ip: xx.xx.xx.xx
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/enable-cors: &quot;false&quot;
      nginx.ingress.kubernetes.io/proxy-buffering: &quot;off&quot;
      nginx.ingress.kubernetes.io/rewrite-target: /$1
    labels:
      app.kubernetes.io/name: app-b
    name: app-b
    namespace: dev-app-b
  spec:
    rules:
    - host: my-host.com
      http:
        paths:
        - backend:
            serviceName: app-b
            servicePort: http
          path: /app-b/(.*)
          pathType: ImplementationSpecific
    tls:
    - hosts:
      - my-host.com
      secretName: app-b-tls
  status:
    loadBalancer:
      ingress:
      - ip: xx.xx.xx.xx
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/enable-cors: &quot;false&quot;
      nginx.ingress.kubernetes.io/proxy-buffering: &quot;off&quot;
      nginx.ingress.kubernetes.io/rewrite-target: /$1
    labels:
      app.kubernetes.io/name: app-c
    name: app-c
    namespace: dev-app-c
  spec:
    rules:
    - host: my-host.com
      http:
        paths:
        - backend:
            serviceName: app-c
            servicePort: http
          path: /app-c/(.*)
          pathType: ImplementationSpecific
    tls:
    - hosts:
      - my-host.com
      secretName: app-c-tls
  status:
    loadBalancer:
      ingress:
      - ip: xx.xx.xx.xx
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
      meta.helm.sh/release-name: prometheus
      meta.helm.sh/release-namespace: prom
      nginx.ingress.kubernetes.io/add-base-url: &quot;true&quot;
      nginx.ingress.kubernetes.io/auth-realm: Authentication Required - Grafana UI
      nginx.ingress.kubernetes.io/auth-secret: monitoring-basic-auth
      nginx.ingress.kubernetes.io/auth-type: basic
      nginx.ingress.kubernetes.io/enable-cors: &quot;false&quot;
      nginx.ingress.kubernetes.io/from-to-www-redirect: &quot;true&quot;
      nginx.ingress.kubernetes.io/proxy-buffering: &quot;off&quot;
    labels:
      app.kubernetes.io/instance: prometheus
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: grafana
      app.kubernetes.io/version: 7.4.2
      helm.sh/chart: grafana-6.4.8
    name: prometheus-grafana
    namespace: prom
  spec:
    rules:
    - host: my-other-host.com
      http:
        paths:
        - backend:
            serviceName: prometheus-grafana
            servicePort: 80
          path: /monitoring/grafana/
          pathType: ImplementationSpecific
    tls:
    - hosts:
      - my-other-host.com
      secretName: prometheus-general-tls
  status:
    loadBalancer:
      ingress:
      - ip: xx.xx.xx.xx
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
      meta.helm.sh/release-name: prometheus
      meta.helm.sh/release-namespace: prom
      nginx.ingress.kubernetes.io/auth-realm: Authentication Required - AlertManager UI
      nginx.ingress.kubernetes.io/auth-secret: monitoring-basic-auth
      nginx.ingress.kubernetes.io/auth-type: basic
    labels:
      app: kube-prometheus-stack-alertmanager
      app.kubernetes.io/managed-by: Helm
      chart: kube-prometheus-stack-14.0.1
      heritage: Helm
      release: prometheus
    name: prometheus-kube-prometheus-alertmanager
    namespace: prom
  spec:
    rules:
    - host: my-other-host.com
      http:
        paths:
        - backend:
            serviceName: prometheus-kube-prometheus-alertmanager
            servicePort: 9093
          path: /monitoring/alertmanager/
          pathType: ImplementationSpecific
    tls:
    - hosts:
      - my-other-host.com
      secretName: prometheus-general-tls
  status:
    loadBalancer:
      ingress:
      - ip: xx.xx.xx.xx
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
      kubernetes.io/ingress.class: nginx
      meta.helm.sh/release-name: prometheus
      meta.helm.sh/release-namespace: prom
      nginx.ingress.kubernetes.io/auth-realm: Authentication Required - Prometheus UI
      nginx.ingress.kubernetes.io/auth-secret: monitoring-basic-auth
      nginx.ingress.kubernetes.io/auth-type: basic
    labels:
      app: kube-prometheus-stack-prometheus
      app.kubernetes.io/managed-by: Helm
      chart: kube-prometheus-stack-14.0.1
      heritage: Helm
      release: prometheus
    name: prometheus-kube-prometheus-prometheus
    namespace: prom
  spec:
    rules:
    - host: my-other-host.com
      http:
        paths:
        - backend:
            serviceName: prometheus-kube-prometheus-prometheus
            servicePort: 9090
          path: /monitoring/prometheus/
          pathType: ImplementationSpecific
    tls:
    - hosts:
      - my-other-host.com
      secretName: prometheus-general-tls
  status:
    loadBalancer:
      ingress:
      - ip: xx.xx.xx.xx
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/enable-cors: &quot;false&quot;
      nginx.ingress.kubernetes.io/proxy-buffering: &quot;off&quot;
    labels:
      app.kubernetes.io/managed-by: terraform
      app.kubernetes.io/name: app-a
    name: app-a
    namespace: stage-app-a
  spec:
    rules:
    - host: my-third-host.com
      http:
        paths:
        - backend:
            serviceName: app-a-http
            servicePort: http
          path: /controller/
          pathType: ImplementationSpecific
    tls:
    - hosts:
      - my-third-host.com
      secretName: app-a-tls
  status:
    loadBalancer:
      ingress:
      - ip: xx.xx.xx.xx
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/add-base-url: &quot;true&quot;
      nginx.ingress.kubernetes.io/auth-realm: Authentication Required - Jaeger UI
      nginx.ingress.kubernetes.io/auth-secret: jaeger-basic-auth
      nginx.ingress.kubernetes.io/auth-type: basic
      nginx.ingress.kubernetes.io/enable-cors: &quot;false&quot;
      nginx.ingress.kubernetes.io/from-to-www-redirect: &quot;true&quot;
      nginx.ingress.kubernetes.io/proxy-buffering: &quot;off&quot;
    labels:
      app.kubernetes.io/name: jaeger
    name: jaeger
    namespace: stage-jaeger
  spec:
    rules:
    - host: my-third-host.com
      http:
        paths:
        - backend:
            serviceName: jaeger
            servicePort: http-ui
          path: /jaeger/
          pathType: ImplementationSpecific
    tls:
    - hosts:
      - my-third-host.com
      secretName: jaeger-tls
  status:
    loadBalancer:
      ingress:
      - ip: xx.xx.xx.xx
</code></pre>
<p>E.g. trying a simple curl on an unknown path ( where I would expect default backend to given 404 response ) returns data that is coming from my Prometheus endpoint being basic auth protected :</p>
<p>(The real hostname/ip has been obfuscated by me !)</p>
<pre><code>$ curl -v &quot;https://my-other-host.com/bar&quot;
*   Trying xx.xx.xx.xx...
* TCP_NODELAY set
* Connected to my-other-host.com (xx.xx.xx.xx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=my-other-host.com
*  start date: Apr  6 14:28:20 2021 GMT
*  expire date: Jul  5 14:28:20 2021 GMT
*  subjectAltName: host &quot;my-other-host.com&quot; matched cert's &quot;my-other-host.com&quot;
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fec1300d600)
&gt; GET /bar HTTP/2
&gt; Host: my-other-host.com
&gt; User-Agent: curl/7.64.1
&gt; Accept: */*
&gt;
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
&lt; HTTP/2 401
&lt; date: Wed, 21 Apr 2021 15:37:24 GMT
&lt; content-type: text/html
&lt; content-length: 172
&lt; www-authenticate: Basic realm=&quot;Authentication Required - Prometheus UI&quot;
&lt; strict-transport-security: max-age=15724800; includeSubDomains
&lt;
&lt;html&gt;
&lt;head&gt;&lt;title&gt;401 Authorization Required&lt;/title&gt;&lt;/head&gt;
&lt;body&gt;
&lt;center&gt;&lt;h1&gt;401 Authorization Required&lt;/h1&gt;&lt;/center&gt;
&lt;hr&gt;&lt;center&gt;nginx&lt;/center&gt;
&lt;/body&gt;
&lt;/html&gt;
* Connection #0 to host my-other-host.com left intact
* Closing connection 0
</code></pre>
<p>or similar when trying on the other host I'll get 401 from something that looks like my Jaeger endpoint instead of default-backend 404:</p>
<pre><code>$ curl -v &quot;https://my-host.com/foo&quot;
*   Trying xx.xx.xx.xx...
* TCP_NODELAY set
* Connected to my-host.com (xx.xx.xx.xx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=my-host.com
*  start date: Mar 18 06:37:40 2021 GMT
*  expire date: Jun 16 06:37:40 2021 GMT
*  subjectAltName: host &quot;my-host.com&quot; matched cert's &quot;my-host.com&quot;
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7f968480d600)
&gt; GET /foo HTTP/2
&gt; Host: my-host.com
&gt; User-Agent: curl/7.64.1
&gt; Accept: */*
&gt; 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
&lt; HTTP/2 401 
&lt; date: Wed, 21 Apr 2021 15:59:21 GMT
&lt; content-type: text/html
&lt; content-length: 172
&lt; www-authenticate: Basic realm=&quot;Authentication Required - Jaeger UI&quot;
&lt; strict-transport-security: max-age=15724800; includeSubDomains
&lt; 
&lt;html&gt;
&lt;head&gt;&lt;title&gt;401 Authorization Required&lt;/title&gt;&lt;/head&gt;
&lt;body&gt;
&lt;center&gt;&lt;h1&gt;401 Authorization Required&lt;/h1&gt;&lt;/center&gt;
&lt;hr&gt;&lt;center&gt;nginx&lt;/center&gt;
&lt;/body&gt;
&lt;/html&gt;
* Connection #0 to host my-host.com left intact
* Closing connection 0
</code></pre>

<p>Fixed ( or worked around the problem ) by explicitly adding an Ingress route for every host that gets served in the cluster:</p>
<pre><code>apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  name: default-route-per-host
  namespace: ingress
spec:
  rules:
  - host: my-host.com
    http:
      paths:
      - backend:
          service:
            name: ingress-nginx-defaultbackend
            port:
              name: http
        path: /
        pathType: Prefix
  - host: my-other-host.com
    http:
      paths:
      - backend:
          service:
            name: ingress-nginx-defaultbackend
            port:
              name: http
        path: /
        pathType: Prefix
  - host: my-third-host.com
    http:
      paths:
      - backend:
          service:
            name: ingress-nginx-defaultbackend
            port:
              name: http
        path: /
        pathType: Prefix
</code></pre>
