<p>I am currently playing around with a rpi based k3s cluster and I am observing a weird phenomenon. </p>

<p>I deployed two applications.
The first one is nginx which I can reach on the url <a href="http://external-ip/foo" rel="nofollow noreferrer">http://external-ip/foo</a> based on the following ingress rule:</p>

<pre><code>apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: foo
  namespace: foo
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/rule-type: "PathPrefixStrip"
    traefik.ingress.kubernetes.io/rewrite-target: "/"
spec:
  rules:
  - http:
      paths:
      - path: /foo
        backend:
          serviceName: foo-service
          servicePort: 8081
</code></pre>

<p>And the other one is grafana which I cannot reach on the url <a href="http://external-ip/grafana" rel="nofollow noreferrer">http://external-ip/grafana</a> based on the below ingress rule:</p>

<pre><code>apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: grafana
  namespace: grafana
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/rule-type: "PathPrefixStrip"
    traefik.ingress.kubernetes.io/rewrite-target: "/"
spec:
  rules:
  - http:
      paths:
      - path: /grafana
        backend: 
          serviceName: grafana-service
          servicePort: 3000
</code></pre>

<p>When I do a port-forward directly on the pod I can reach the grafana app, when I use the port-forward on the grafana service it also works. </p>

<p>However as soon as I try to reach it through the subpath I will get a gateway timeout.</p>

<p>Does anyone have a guess what I am missing?</p>

<p>Here the deployment and service for the grafana deployment:</p>

<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: grafana
  labels:
    app: grafana
    tier: frontend
    service: monitoring
spec:
  selector:
    matchLabels:
      app: grafana
      tier: frontend
  template:
    metadata:
      labels:
        app: grafana
        tier: frontend
        service: monitoring
    spec:
      containers:
      - image: grafana
        imagePullPolicy: IfNotPresent
        name: grafana
        envFrom:
        - configMapRef:
            name: grafana-config
        ports:
        - name: frontend
          containerPort: 3000
          protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: grafana-service
  namespace: grafana
  labels:
    app: grafana
    tier: frontend
    service: monitoring
spec:
  selector:
    app: grafana
    tier: frontend
  type: NodePort
  ports:
  - name: frontend
    port: 3000
    protocol: TCP
    targetPort: 3000
</code></pre>

<h1>Solution</h1>

<p>I had to add the following two parameters to my configmap to make it work:</p>

<pre><code>GF_SERVER_ROOT_URL=http://localhost:3000/grafana/   
GF_SERVER_FROM_SUB_PATH=true
</code></pre>

<p>As I mentioned in comments grafana is not listening on <code>/</code> like default nginx.</p>
<p>There is related <a href="https://github.com/helm/charts/issues/6264" rel="nofollow noreferrer">github issue</a> about this, and if you want to make it work you should specify root_url</p>
<pre><code>grafana.ini:
  server:
    root_url: https://subdomain.example.com/grafana
</code></pre>
<p>Specifically take a look at <a href="https://github.com/helm/charts/issues/6264#issuecomment-480530680" rel="nofollow noreferrer">this</a> and <a href="https://github.com/helm/charts/issues/6264#issuecomment-463755283" rel="nofollow noreferrer">this</a> comment.</p>
<hr />
<p>@tehemaroo add his own solution which include changing root url and sub_path in configmap</p>
<blockquote>
<p>I had to add the following two parameters to my configmap to make it work:</p>
</blockquote>
<pre><code>GF_SERVER_ROOT_URL=http://localhost:3000/grafana/   
GF_SERVER_FROM_SUB_PATH=true
</code></pre>
<p>And related <a href="https://grafana.com/tutorials/run-grafana-behind-a-proxy/#1" rel="nofollow noreferrer">documentation</a> about that</p>
<blockquote>
<p>To serve Grafana behind a sub path:</p>
<p>Include the sub path at the end of the root_url.</p>
<p>Set serve_from_sub_path to true.</p>
</blockquote>
<pre><code>[server]
domain = example.com
root_url = %(protocol)s://%(domain)s:%(http_port)s/grafana/
serve_from_sub_path = true
</code></pre>
