<p>I am trying to get a new cluster setup on Google Kubernetes Engine AKA GKE.
I am using helm and added to the repo ingress-nginx <a href="https://kubernetes.github.io/ingress-nginx" rel="nofollow noreferrer">https://kubernetes.github.io/ingress-nginx</a> and grafana <a href="https://grafana.github.io/helm-charts" rel="nofollow noreferrer">https://grafana.github.io/helm-charts</a></p>
<p>I was following this <a href="https://kubernetes.github.io/ingress-nginx/user-guide/monitoring/" rel="nofollow noreferrer">guide</a> but I was not able to access an internal ip or get the ingress working to where I could see the dashboard. I have then tried this <a href="https://grafana.com/docs/loki/latest/installation/helm/" rel="nofollow noreferrer">guide</a> using their helm deployment for Loki Stack (Loki, Promtail, Grafana, Prometheus) with persistent volume claim.</p>
<p>That combined with opening port 8443 up and I was still getting a 503 error. I made this ingress:</p>
<pre><code>apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-resource
  annotations:
    kubernetes.io/ingress.class: &quot;nginx&quot;
    nginx.ingress.kubernetes.io/use-regex: &quot;true&quot;
spec:
  rules:
    - http:
        paths:
          - path: /pro
            pathType: Prefix
            backend:
              service:
                name: loki-prometheus-server
                port:
                  number: 9090
          - path: /graf
            pathType: Prefix
            backend:
              service:
                name: loki-grafana
                port:
                  number: 3000

</code></pre>
<p>I tried many different ports including 80 but it was a no go.</p>
<p>loki-prometheus-server service:</p>
<pre><code>Name:              loki-prometheus-server
Namespace:         ingress-nginx
Labels:            app=prometheus
                   app.kubernetes.io/managed-by=Helm
                   chart=prometheus-11.16.9
                   component=server
                   heritage=Helm
                   release=loki
Annotations:       cloud.google.com/neg: {&quot;ingress&quot;:true}
                   meta.helm.sh/release-name: loki
                   meta.helm.sh/release-namespace: ingress-nginx
Selector:          app=prometheus,component=server,release=loki
Type:              ClusterIP
IP Families:       &lt;none&gt;
IP:                10.***
IPs:               10.***
Port:              http  80/TCP
TargetPort:        9090/TCP
Endpoints:         10.***:9090
Session Affinity:  None
Events:            &lt;none&gt;
</code></pre>
<p>loki-grafana service:</p>
<pre><code>Name:              loki-grafana
Namespace:         ingress-nginx
Labels:            app.kubernetes.io/instance=loki
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=grafana
                   app.kubernetes.io/version=7.5.0
                   helm.sh/chart=grafana-5.7.10
Annotations:       cloud.google.com/neg: {&quot;ingress&quot;:true}
                   meta.helm.sh/release-name: loki
                   meta.helm.sh/release-namespace: ingress-nginx
Selector:          app.kubernetes.io/instance=loki,app.kubernetes.io/name=grafana
Type:              ClusterIP
IP Families:       &lt;none&gt;
IP:                10.***
IPs:               10.***
Port:              service  80/TCP
TargetPort:        3000/TCP
Endpoints:         10.***:3000
Session Affinity:  None
Events:            &lt;none&gt;
</code></pre>
<p><strong>Step by Step:</strong></p>
<p>Create new cluster on gcloud:</p>
<ol>
<li>GKE Standard</li>
<li>default-pool-&gt;Nodes-&gt; Machine type: e2-small</li>
<li>default-pool-&gt;Security-&gt; Allow full access to all Cloud APIs selected Create</li>
<li>Create.</li>
</ol>
<p>Connect to console or cloud shell</p>
<p>Then:</p>
<pre><code>helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

kubectl create namespace ingress-nginx
helm install ingress-nginx ingress-nginx/ingress-nginx --namespace ingress-nginx
</code></pre>
<p>Everything seems normal so far.</p>
<p>Go to VPC network-&gt; Firewall -&gt; Find the one with 80,443 in Protocols/ports
Then add 8443 to it.</p>
<p>Install loki:</p>
<pre><code>helm upgrade --install loki grafana/loki-stack --namespace ingress-nginx  --set grafana.enabled=true,prometheus.enabled=true,prometheus.alertmanager.persistentVolume.enabled=false,prometheus.server.persistentVolume.enabled=false,loki.persistence.enabled=true,loki.persistence.storageClassName=standard,loki.persistence.size=5Gi
</code></pre>
<p>Everything still seems to be going fine but if I go to my external load balancer I get 404 not found / nginx</p>
<p>I add this ingress:</p>
<p>using <code>kubectl apply -f ingress.yml -n ingress-nginx</code></p>
<pre><code>apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-resource
  annotations:
    kubernetes.io/ingress.class: &quot;nginx&quot;
    prometheus.io/scrape: &quot;true&quot;
spec:
  rules:
    - http:
        paths:
          - path: /pro
            pathType: Prefix
            backend:
              service:
                name: loki-prometheus-server
                port:
                  number: 80
          - path: /graf
            pathType: Prefix
            backend:
              service:
                name: loki-grafana
                port:
                  number: 80
</code></pre>
<p>And get a 404 page not found in text on /pro
And get redirected to /login with a 404 not found nginx page at /graf</p>
<p>What am I doing wrong?</p>

mitchldtn
<p>Running:</p>
<pre><code>helm upgrade ingress-nginx ingress-nginx/ingress-nginx --namespace ingress-nginx  --set controller.metrics.enabled=true --set-string controller.podAnnotations.&quot;prometheus\.io/scrape&quot;=&quot;true&quot; --set-string controller.podAnnotations.&quot;prometheus\.io/port&quot;=&quot;10254&quot;
</code></pre>
<p>And using this ingress I was able to get it working:</p>
<pre><code>apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-resource
  annotations:
    kubernetes.io/ingress.class: &quot;nginx&quot;
    nginx.ingress.kubernetes.io/use-regex: &quot;true&quot;
    nginx.ingress.kubernetes.io/ssl-redirect: &quot;false&quot;
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    prometheus.io/port: &quot;10254&quot;
    prometheus.io/scrape: &quot;true&quot;
spec:
  rules:
    - http:
        paths:
          - path: /?(.*)
            pathType: Prefix
            backend:
              service:
                name: loki-grafana
                port:
                  number: 80

</code></pre>
