<p>I created simple kubernetes cluster with demo app.
When creating the cluster I installed prometheus-stack and nginx-ingress-controller with helm (default values files).</p>
<p>After the cluster is set up I create Ingress object to expose prometheus, grafana and alertmanager with:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: monitoring-ingress
  namespace: monitoring
  annotations:
    kubernetes.io/ingress.class: &quot;nginx&quot;
    nginx.ingress.kubernetes.io/ssl-redirect: &quot;false&quot;
    nginx.ingress.kubernetes.io/use-regex: &quot;true&quot;
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
  - http:
      paths:
      - path: /prometheus(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: prometheus-operated
            port:
              number: 9090
      - path: /alertmanager(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: prometheus-stack-kube-prom-alertmanager
            port:
              number: 9093
      - path: /grafana(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: prometheus-stack-grafana
            port:
              number: 80
</code></pre>
<p>When I try to access prometheus via <code>&lt;ingress-controller's external IP&gt;/prometheus</code> it resolves to <code>&lt;ingress-controller's external IP&gt;/graph</code> and displays 404 - page not found.</p>
<p>If I use for example <code>kubectl port-forward svc/prometheus-operated 9090:9090 -n monitoring</code> I can reach prometheus without problem.</p>
<p>I can reach alertmanager through <code>&lt;ingress-controller's external IP&gt;/alertmanager</code>. The path is resolved to <code>&lt;ingress-controller's external IP&gt;/alertmanager/#/alerts</code></p>
<p>I suspect there is something wrong with the path rewriting but can't figure out what.</p>
<p>Please help...</p>

Artur Ferfecki
<p>In the end I was able to find 2 solutions to this problem.</p>
<p><strong>Option 1</strong></p>
<p>I had DNS zone in azure (where my cluster lives as well) and there I added subdomains for grafana, prometheus and alertmanager pointing to the ingress-controller external IP.</p>
<p>When deploying kube-prometheus-stack with Helm Chart I provided default root path for prometheus in values.yaml file with folowing configuration:</p>
<pre class="lang-yaml prettyprint-override"><code>prometheus:
  prometheusSpec:
    externalUrl: http://prometheus.mydomainname.something
</code></pre>
<p>The Ingress manifest for (for example Prometheus) then needs to include the host address and contain only root path &quot;/&quot;.</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: prometheus-new
  namespace: monitoring
  annotations:
    kubernetes.io/ingress.class: &quot;nginx&quot;
    nginx.ingress.kubernetes.io/ssl-redirect: &quot;false&quot;
spec:
  rules:
  - host: prometheus.mydomainname.something
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: prometheus-stack-kube-prom-prometheus
            port:
              number: 9090
</code></pre>
<p><strong>Option 2</strong></p>
<p>You can also provide default paths for each of the applications from kube-prometheus-stack (prometheus, grafana, alertmanager) as:</p>
<pre class="lang-yaml prettyprint-override"><code>prometheus:
  prometheusSpec:
    externalUrl: http://mydomainname.something/prometheus
</code></pre>
<p>and make Ingress manifest to redirect based on the path with:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: prometheus-new
  namespace: monitoring
  annotations:
    kubernetes.io/ingress.class: &quot;nginx&quot;
    nginx.ingress.kubernetes.io/ssl-redirect: &quot;false&quot;
spec:
  rules:
  - http:
      paths:
      - path: /prometheus
        pathType: Prefix
        backend:
          service:
            name: prometheus-stack-kube-prom-prometheus
            port:
              number: 9090
</code></pre>
