<p>UPDATE:
The issue persists but I used another way (sub-domain name, instead of the path) to 'bypass' the issue:</p>
<pre><code>ubuntu@df1:~$ cat k8s-dashboard-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: k8s-dashboard-ingress
  namespace: kubernetes-dashboard
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: &quot;HTTPS&quot;
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: &quot;true&quot;
    nginx.ingress.kubernetes.io/secure-backends: &quot;true&quot;
spec:
  ingressClassName: nginx
  tls: 
    - hosts: 
      - dashboard.XXXX
      secretName: df1-tls
  rules:
  - host: dashboard.XXXX
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: kubernetes-dashboard
            port:
              number: 443


</code></pre>
<hr />
<p>This error bothers me for some time and I hope with your help I can come down to the bottom of it.</p>
<p>I have one K8S cluster (single node so far, to avoid any network related issues). I installed Grafana on it.</p>
<p>All pods are running fine:</p>
<pre><code>ubuntu:~$ k get po -A
NAMESPACE              NAME                                         READY   STATUS    RESTARTS   AGE
default                grafana-646c8874cb-h6tc5                     1/1     Running   0          11h
default                nginx-1-7bdc99b884-xh7kl                     1/1     Running   0          36h
kube-system            coredns-64897985d-4sk6l                      1/1     Running   0          2d16h
kube-system            coredns-64897985d-dx5h6                      1/1     Running   0          2d16h
kube-system            etcd-df1                                     1/1     Running   1          3d14h
kube-system            kilo-kb52f                                   1/1     Running   0          2d16h
kube-system            kube-apiserver-df1                           1/1     Running   1          3d14h
kube-system            kube-controller-manager-df1                  1/1     Running   4          3d14h
kube-system            kube-flannel-ds-fjkxv                        1/1     Running   0          3d13h
kube-system            kube-proxy-bd2xt                             1/1     Running   0          3d14h
kube-system            kube-scheduler-df1                           1/1     Running   10         3d14h
kubernetes-dashboard   dashboard-metrics-scraper-799d786dbf-5skdw   1/1     Running   0          2d16h
kubernetes-dashboard   kubernetes-dashboard-6b6b86c4c5-56zp2        1/1     Running   0          2d16h
nginx-ingress          nginx-ingress-5b467c7d7-qtqtq                1/1     Running   0          2d15h

</code></pre>
<p>As you saw, I installed nginx ingress controller.</p>
<p>Here is the ingress:</p>
<pre><code>ubuntu:~$ k describe ing grafana
Name:             grafana
Labels:           app.kubernetes.io/instance=grafana
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=grafana
                  app.kubernetes.io/version=8.3.3
                  helm.sh/chart=grafana-6.20.5
Namespace:        default
Address:          
Default backend:  default-http-backend:80 (&lt;error: endpoints &quot;default-http-backend&quot; not found&gt;)
Rules:
  Host           Path  Backends
  ----           ----  --------
  kalepa.k8s.io  
                 /grafana   grafana:80 (10.244.0.14:3000)
Annotations:     meta.helm.sh/release-name: grafana
                 meta.helm.sh/release-namespace: default
Events:          &lt;none&gt;
</code></pre>
<p>Here is the service that is defined in above ingress:</p>
<pre><code>ubuntu:~$ k get svc
NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
grafana       ClusterIP   10.96.148.1     &lt;none&gt;        80/TCP     11h
kubernetes    ClusterIP   10.96.0.1       &lt;none&gt;        443/TCP    3d14h
</code></pre>
<p>If I do a curl to the cluster ip of the service, it goes through without an issue:</p>
<pre><code>ubuntu:~$ curl 10.96.148.1
&lt;a href=&quot;/grafana/login&quot;&gt;Found&lt;/a&gt;.
</code></pre>
<p>If I do a curl to the hostname with the path to the service, I got the 404 error:</p>
<pre><code>ubuntu:~$ curl kalepa.k8s.io/grafana
&lt;html&gt;
&lt;head&gt;&lt;title&gt;404 Not Found&lt;/title&gt;&lt;/head&gt;
&lt;body&gt;
&lt;center&gt;&lt;h1&gt;404 Not Found&lt;/h1&gt;&lt;/center&gt;
&lt;hr&gt;&lt;center&gt;nginx/1.21.5&lt;/center&gt;
&lt;/body&gt;
&lt;/html&gt;
</code></pre>
<p>The hostname is resolved to the cluster ip of the nginx ingress service (nodeport):</p>
<pre><code>ubuntu:~$ grep kalepa.k8s.io /etc/hosts
10.96.241.112   kalepa.k8s.io
</code></pre>
<p>This is the nginx ingress service definition:</p>
<pre><code>ubuntu:~$ k describe -n nginx-ingress svc nginx-ingress
Name:                     nginx-ingress
Namespace:                nginx-ingress
Labels:                   &lt;none&gt;
Annotations:              &lt;none&gt;
Selector:                 app=nginx-ingress
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.96.241.112
IPs:                      10.96.241.112
Port:                     http  80/TCP
TargetPort:               80/TCP
NodePort:                 http  31803/TCP
Endpoints:                10.244.0.6:80
Port:                     https  443/TCP
TargetPort:               443/TCP
NodePort:                 https  31913/TCP
Endpoints:                10.244.0.6:443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   &lt;none&gt;
</code></pre>
<p>What am I missing? Thanks for your help!</p>

<p>This is happening as you are using /grafana and this path does not exist in the grafana application - hence 404. You need to first configure grafana to use this context path before you can forward your traffic to /grafana.</p>
<p>If you use / as path, it will work. That's why curl 10.96.148 works as you are not adding a route /grafana. But most likely that path is already used by some other service, that's why you were using /grafana to begin with.</p>
<p>Therefore, you need to update your grafana.ini file to set the context root explicitly as shown below.</p>
<p>You may put your grafana.ini in a configmap, mount it to the original grafana.ini location and recreate the deployment.</p>
<pre><code>[server]
domain = kalepa.k8s.io
root_url = http://kalepa.k8s.io/grafana/
</code></pre>
