<p>I have a baremetal cluster deployed using Kubespray with kubernetes 1.22.2, MetalLB, and ingress-nginx enabled. I am getting <code>404 Not found</code> when trying to access any service deployed via helm when setting <code>ingressClassName: nginx</code>. However, everything works fine if I don't use <code>ingressClassName: nginx</code> but <code>kubernetes.io/ingress.class: nginx</code> instead in the helm chart values.yaml. How can I get it to work using <code>ingressClassName</code>?</p>
<p>These are my kubespray settings for <code>inventory/mycluster/group_vars/k8s_cluster/addons.yml</code></p>
<pre><code># Nginx ingress controller deployment
ingress_nginx_enabled: true
ingress_nginx_host_network: false
ingress_publish_status_address: &quot;&quot;
ingress_nginx_nodeselector:
  kubernetes.io/os: &quot;linux&quot;
ingress_nginx_tolerations:
  - key: &quot;node-role.kubernetes.io/master&quot;
    operator: &quot;Equal&quot;
    value: &quot;&quot;
    effect: &quot;NoSchedule&quot;
  - key: &quot;node-role.kubernetes.io/control-plane&quot;
    operator: &quot;Equal&quot;
    value: &quot;&quot;
    effect: &quot;NoSchedule&quot;
ingress_nginx_namespace: &quot;ingress-nginx&quot;
ingress_nginx_insecure_port: 80
ingress_nginx_secure_port: 443
ingress_nginx_configmap:
  map-hash-bucket-size: &quot;128&quot;
  ssl-protocols: &quot;TLSv1.2 TLSv1.3&quot;
ingress_nginx_configmap_tcp_services:
  9000: &quot;default/example-go:8080&quot;
ingress_nginx_configmap_udp_services:
  53: &quot;kube-system/coredns:53&quot;
ingress_nginx_extra_args:
  - --default-ssl-certificate=default/mywildcard-tls
ingress_nginx_class: &quot;nginx&quot;
</code></pre>
<p>grafana helm values.yaml</p>
<pre><code>ingress:
  enabled: true
  # For Kubernetes &gt;= 1.18 you should specify the ingress-controller via the field ingressClassName
  # See https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/#specifying-the-class-of-an-ingress
  ingressClassName: nginx
  # Values can be templated
  annotations:
  #  kubernetes.io/ingress.class: nginx
  #  kubernetes.io/tls-acme: &quot;true&quot;
  labels: {}
  path: /

  # pathType is only for k8s &gt;= 1.1=
  pathType: Prefix

  hosts:
    - grafana.mycluster.org
  tls:
   - secretName: mywildcard-tls
     hosts:
       - grafana.mycluster.org
</code></pre>
<p><code>kubectl describe pod grafana-679bbfd94-p2dd7</code></p>
<pre><code>...
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  25m                default-scheduler  Successfully assigned default/grafana-679bbfd94-p2dd7 to node1
  Normal   Pulled     25m                kubelet            Container image &quot;grafana/grafana:8.2.2&quot; already present on machine
  Normal   Created    25m                kubelet            Created container grafana
  Normal   Started    25m                kubelet            Started container grafana
  Warning  Unhealthy  24m (x3 over 25m)  kubelet            Readiness probe failed: Get &quot;http://10.233.90.33:3000/api/health&quot;: dial tcp 10.233.90.33:3000: connect: connection refused
</code></pre>
<p><code>kubectl get svc</code></p>
<pre><code>NAME         TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
grafana      LoadBalancer   10.233.14.90   10.10.30.52   80:30285/TCP   55m
kubernetes   ClusterIP      10.233.0.1     &lt;none&gt;        443/TCP        9d
</code></pre>
<p><code>kubectl get ing</code> (no node address assigned)</p>
<pre><code>NAME      CLASS   HOSTS                    ADDRESS   PORTS     AGE
grafana   nginx   grafana.mycluster.org             80, 443   25m
</code></pre>
<p><code>kubectl describe ing grafana</code> (no node address assigned)</p>
<pre><code>Name:             grafana
Namespace:        default
Address:
Default backend:  default-http-backend:80 (&lt;error: endpoints &quot;default-http-backend&quot; not found&gt;)
TLS:
  mywildcard-tls terminates grafana.mycluster.org
Rules:
  Host                    Path  Backends
  ----                    ----  --------
  grafana.mycluster.org
                          /   grafana:80 (10.233.90.33:3000)
Annotations:              meta.helm.sh/release-name: grafana
                          meta.helm.sh/release-namespace: default
Events:                   &lt;none&gt;
</code></pre>
<p><code>kubectl get all --all-namespaces</code></p>
<pre><code>NAMESPACE        NAME                                                              READY   STATUS    RESTARTS   AGE
default          pod/grafana-b988b9b6-pxccw                                        1/1     Running   0          2m53s
default          pod/nfs-client-nfs-subdir-external-provisioner-68f44cd9f4-wjlpv   1/1     Running   0          17h
ingress-nginx    pod/ingress-nginx-controller-6m2vt                                1/1     Running   0          17h
ingress-nginx    pod/ingress-nginx-controller-xkgxl                                1/1     Running   0          17h
kube-system      pod/calico-kube-controllers-684bcfdc59-kmsst                      1/1     Running   0          17h
kube-system      pod/calico-node-dhlnt                                             1/1     Running   0          17h
kube-system      pod/calico-node-r8ktz                                             1/1     Running   0          17h
kube-system      pod/coredns-8474476ff8-9sbwh                                      1/1     Running   0          17h
kube-system      pod/coredns-8474476ff8-fdgcb                                      1/1     Running   0          17h
kube-system      pod/dns-autoscaler-5ffdc7f89d-vskvq                               1/1     Running   0          17h
kube-system      pod/kube-apiserver-node1                                          1/1     Running   0          17h
kube-system      pod/kube-controller-manager-node1                                 1/1     Running   1          17h
kube-system      pod/kube-proxy-hbjz6                                              1/1     Running   0          16h
kube-system      pod/kube-proxy-lfqzt                                              1/1     Running   0          16h
kube-system      pod/kube-scheduler-node1                                          1/1     Running   1          17h
kube-system      pod/kubernetes-dashboard-548847967d-qqngw                         1/1     Running   0          17h
kube-system      pod/kubernetes-metrics-scraper-6d49f96c97-2h7hc                   1/1     Running   0          17h
kube-system      pod/nginx-proxy-node2                                             1/1     Running   0          17h
kube-system      pod/nodelocaldns-64cqs                                            1/1     Running   0          17h
kube-system      pod/nodelocaldns-t5vv6                                            1/1     Running   0          17h
kube-system      pod/registry-proxy-kljvw                                          1/1     Running   0          17h
kube-system      pod/registry-proxy-nz4qk                                          1/1     Running   0          17h
kube-system      pod/registry-xzh9d                                                1/1     Running   0          17h
metallb-system   pod/controller-77c44876d-c92lb                                    1/1     Running   0          17h
metallb-system   pod/speaker-fkjqp                                                 1/1     Running   0          17h
metallb-system   pod/speaker-pqjgt                                                 1/1     Running   0          17h

NAMESPACE     NAME                                TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
default       service/grafana                     LoadBalancer   10.233.1.104    10.10.30.52   80:31116/TCP             2m53s
default       service/kubernetes                  ClusterIP      10.233.0.1      &lt;none&gt;        443/TCP                  17h
kube-system   service/coredns                     ClusterIP      10.233.0.3      &lt;none&gt;        53/UDP,53/TCP,9153/TCP   17h
kube-system   service/dashboard-metrics-scraper   ClusterIP      10.233.35.124   &lt;none&gt;        8000/TCP                 17h
kube-system   service/kubernetes-dashboard        ClusterIP      10.233.32.133   &lt;none&gt;        443/TCP                  17h
kube-system   service/registry                    ClusterIP      10.233.30.221   &lt;none&gt;        5000/TCP                 17h

NAMESPACE        NAME                                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
ingress-nginx    daemonset.apps/ingress-nginx-controller   2         2         2       2            2           kubernetes.io/os=linux   17h
kube-system      daemonset.apps/calico-node                2         2         2       2            2           kubernetes.io/os=linux   17h
kube-system      daemonset.apps/kube-proxy                 2         2         2       2            2           kubernetes.io/os=linux   17h
kube-system      daemonset.apps/nodelocaldns               2         2         2       2            2           kubernetes.io/os=linux   17h
kube-system      daemonset.apps/registry-proxy             2         2         2       2            2           &lt;none&gt;                   17h
metallb-system   daemonset.apps/speaker                    2         2         2       2            2           kubernetes.io/os=linux   17h

NAMESPACE        NAME                                                         READY   UP-TO-DATE   AVAILABLE   AGE
default          deployment.apps/grafana                                      1/1     1            1           2m53s
default          deployment.apps/nfs-client-nfs-subdir-external-provisioner   1/1     1            1           17h
kube-system      deployment.apps/calico-kube-controllers                      1/1     1            1           17h
kube-system      deployment.apps/coredns                                      2/2     2            2           17h
kube-system      deployment.apps/dns-autoscaler                               1/1     1            1           17h
kube-system      deployment.apps/kubernetes-dashboard                         1/1     1            1           17h
kube-system      deployment.apps/kubernetes-metrics-scraper                   1/1     1            1           17h
metallb-system   deployment.apps/controller                                   1/1     1            1           17h

NAMESPACE        NAME                                                                    DESIRED   CURRENT   READY   AGE
default          replicaset.apps/grafana-b988b9b6                                        1         1         1       2m53s
default          replicaset.apps/nfs-client-nfs-subdir-external-provisioner-68f44cd9f4   1         1         1       17h
kube-system      replicaset.apps/calico-kube-controllers-684bcfdc59                      1         1         1       17h
kube-system      replicaset.apps/coredns-8474476ff8                                      2         2         2       17h
kube-system      replicaset.apps/dns-autoscaler-5ffdc7f89d                               1         1         1       17h
kube-system      replicaset.apps/kubernetes-dashboard-548847967d                         1         1         1       17h
kube-system      replicaset.apps/kubernetes-metrics-scraper-6d49f96c97                   1         1         1       17h
kube-system      replicaset.apps/registry                                                1         1         1       17h
metallb-system   replicaset.apps/controller-77c44876d                                    1         1         1       17h
</code></pre>
<p><code>kubectl get ing grafana -o yaml</code></p>
<pre><code>apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/tls-acme: &quot;true&quot;
    meta.helm.sh/release-name: grafana
    meta.helm.sh/release-namespace: default
    nginx.ingress.kubernetes.io/force-ssl-redirect: &quot;true&quot;
    nginx.ingress.kubernetes.io/ssl-redirect: &quot;true&quot;
  creationTimestamp: &quot;2021-11-11T07:16:12Z&quot;
  generation: 1
  labels:
    app.kubernetes.io/instance: grafana
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: grafana
    app.kubernetes.io/version: 8.2.2
    helm.sh/chart: grafana-6.17.5
  name: grafana
  namespace: default
  resourceVersion: &quot;3137&quot;
  uid: 6c34d3bd-9ab6-42fe-ac1b-7620a9566f62
spec:
  ingressClassName: nginx
  rules:
  - host: grafana.mycluster.org
    http:
      paths:
      - backend:
          service:
            name: ssl-redirect
            port:
              name: use-annotation
        path: /*
        pathType: Prefix
      - backend:
          service:
            name: grafana
            port:
              number: 80
        path: /
        pathType: Prefix
status:
  loadBalancer: {}
</code></pre>

<blockquote>
<p>Running <code>kubectl get ingressclass</code> returned 'No resources found'.</p>
</blockquote>
<p>That's the main reason of your issue.</p>
<p><em>Why?</em></p>
<p>When you are specifying <code>ingressClassName: nginx</code> in your Grafana <code>values.yaml</code> file you are <a href="https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/#specifying-the-class-of-an-ingress" rel="noreferrer">setting your Ingress resource to use</a> <code>nginx</code> Ingress class which does not exist.</p>
<p>I replicated your issue using <a href="https://minikube.sigs.k8s.io/docs/start/" rel="noreferrer">minikube</a>, <a href="https://metallb.universe.tf/" rel="noreferrer">MetalLB</a> and <a href="https://kubernetes.github.io/ingress-nginx/" rel="noreferrer">NGINX Ingress</a> installed via modified <a href="https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.4/deploy/static/provider/cloud/deploy.yaml" rel="noreferrer">deploy.yaml file</a> with commented <code>IngressClass</code> resource + set NGINX Ingress controller name to <code>nginx</code> as in your example. The result was exactly the same - <code>ingressClassName: nginx</code> didn't work (no address), but annotation <code>kubernetes.io/ingress.class: nginx</code> worked.</p>
<hr />
<p>(For the below solution I'm using controller pod name <code>ingress-nginx-controller-86c865f5c4-qwl2b</code>, but in your case it will be different - check it using <code>kubectl get pods -n ingress-nginx</code> command. Also keep in mind it's kind of a workaround - usually <code>ingressClass</code> resource should be installed automatically with a whole installation of NGINX Ingress. I'm presenting this solution to understand <em>why</em> it's not worked for you before, and <em>why</em> it works with NGINX Ingress installed using helm)</p>
<p>In the logs of the Ingress NGINX controller I found (<code>kubectl logs ingress-nginx-controller-86c865f5c4-qwl2b -n ingress-nginx</code>):</p>
<pre><code>&quot;Ignoring ingress because of error while validating ingress class&quot; ingress=&quot;default/minimal-ingress&quot; error=&quot;no object matching key \&quot;nginx\&quot; in local store&quot;
</code></pre>
<p>So it's clearly shown that there is no matching key to <code>nginx</code> controller class - because there is no <code>ingressClass</code> resource which is the &quot;link&quot; between the NGINX Ingress controller and running Ingress resource.</p>
<p>You can verify which name of controller class is bidden to controller by running <code>kubectl get pod ingress-nginx-controller-86c865f5c4-qwl2b -n ingress-nginx -o yaml</code>:</p>
<pre class="lang-yaml prettyprint-override"><code>...
spec:
  containers:
  - args:
    - /nginx-ingress-controller
    - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
    - --election-id=ingress-controller-leader
    - --controller-class=k8s.io/nginx
...
</code></pre>
<p>Now I will create and apply following Ingress class resource:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: nginx
spec:
  controller: k8s.io/nginx
</code></pre>
<p>Now in the logs I can see that it's properly configured:</p>
<pre><code>I1115 12:13:42.410384       7 main.go:101] &quot;successfully validated configuration, accepting&quot; ingress=&quot;minimal-ingress/default&quot;
I1115 12:13:42.420408       7 store.go:371] &quot;Found valid IngressClass&quot; ingress=&quot;default/minimal-ingress&quot; ingressclass=&quot;nginx&quot;
I1115 12:13:42.421487       7 event.go:282] Event(v1.ObjectReference{Kind:&quot;Ingress&quot;, Namespace:&quot;default&quot;, Name:&quot;minimal-ingress&quot;, UID:&quot;c708a672-a8dd-45d3-a2ec-f2e2881623ea&quot;, APIVersion:&quot;networking.k8s.io/v1&quot;, ResourceVersion:&quot;454362&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'Sync' Scheduled for sync
</code></pre>
<p>I re-applied the ingress resource definition, I get IP address for Ingress resource.</p>
<hr />
<p>As I said before, instead of using this workaround, I'd suggest installing the NGINX Ingress resource using a solution that automatically installs <code>IngressClass</code> as well. As you have chosen helm chart, it has <a href="https://github.com/kubernetes/ingress-nginx/blob/main/charts/ingress-nginx/templates/controller-ingressclass.yaml" rel="noreferrer">Ingress Class</a> resource so the problem is gone. Other possible ways to install <a href="https://kubernetes.github.io/ingress-nginx/deploy/" rel="noreferrer">are here</a>.</p>
