title: Getting prometheus/grafana and k3s to work together
body: <p>T learn kubernetes I've built myself a bare metal cluster using 4 Raspberry PIs set it up using k3s:</p>
  <pre><code># curl -sfL https://get.k3s.io | sh -
  </code></pre>
  <p>Added nodes etc., and everything comes up and I can see all the nodes and almost everything is working as expected.</p>
  <p>I wanted to monitor the PIs so I installed the kube-prometheus-stack with helm:</p>
  <pre><code>$ kubectl create namespace monitoring
  $ helm install prometheus --namespace monitoring prometheus-community/kube-prometheus-stack
  </code></pre>
  <p>And now everything looks fantastic:</p>
  <pre><code>$ kubectl get pods --all-namespaces 
  NAMESPACE     NAME                                                     READY   STATUS      RESTARTS   AGE
  kube-system   helm-install-traefik-crd-s8zw5                           0/1     Completed   0          5d21h
  kube-system   helm-install-traefik-rc9f2                               0/1     Completed   1          5d21h
  monitoring    prometheus-prometheus-node-exporter-j85rw                1/1     Running     10         28h
  kube-system   metrics-server-86cbb8457f-mvbkl                          1/1     Running     12         5d21h
  kube-system   coredns-7448499f4d-t7sp8                                 1/1     Running     13         5d21h
  monitoring    prometheus-prometheus-node-exporter-mmh2q                1/1     Running     9          28h
  monitoring    prometheus-prometheus-node-exporter-j4k4c                1/1     Running     10         28h
  monitoring    alertmanager-prometheus-kube-prometheus-alertmanager-0   2/2     Running     10         28h
  kube-system   svclb-traefik-zkqd6                                      2/2     Running     6          19h
  monitoring    prometheus-prometheus-node-exporter-bft5t                1/1     Running     10         28h
  kube-system   local-path-provisioner-5ff76fc89d-g8tm6                  1/1     Running     12         5d21h
  kube-system   svclb-traefik-jcxd2                                      2/2     Running     28         5d21h
  kube-system   svclb-traefik-mpbjm                                      2/2     Running     22         5d21h
  kube-system   svclb-traefik-7kxtw                                      2/2     Running     20         5d21h
  monitoring    prometheus-grafana-864598fd54-9548l                      2/2     Running     10         28h
  kube-system   traefik-65969d48c7-9lh9m                                 1/1     Running     3          19h
  monitoring    prometheus-prometheus-kube-prometheus-prometheus-0       2/2     Running     10         28h
  monitoring    prometheus-kube-state-metrics-76f66976cb-m8k2h           1/1     Running     6          28h
  monitoring    prometheus-kube-prometheus-operator-5c758db547-zsv4s     1/1     Running     6          28h
  </code></pre>
  <p>The services are all there:</p>
  <pre><code>$ kubectl get services --all-namespaces
  NAMESPACE     NAME                                                 TYPE           CLUSTER-IP      EXTERNAL-IP                                                   PORT(S)                        AGE
  default       kubernetes                                           ClusterIP      10.43.0.1       &lt;none&gt;                                                        443/TCP                        5d21h
  kube-system   kube-dns                                             ClusterIP      10.43.0.10      &lt;none&gt;                                                        53/UDP,53/TCP,9153/TCP         5d21h
  kube-system   metrics-server                                       ClusterIP      10.43.80.65     &lt;none&gt;                                                        443/TCP                        5d21h
  kube-system   prometheus-kube-prometheus-kube-proxy                ClusterIP      None            &lt;none&gt;                                                        10249/TCP                      28h
  kube-system   prometheus-kube-prometheus-kube-scheduler            ClusterIP      None            &lt;none&gt;                                                        10251/TCP                      28h
  monitoring    prometheus-kube-prometheus-operator                  ClusterIP      10.43.180.73    &lt;none&gt;                                                        443/TCP                        28h
  kube-system   prometheus-kube-prometheus-coredns                   ClusterIP      None            &lt;none&gt;                                                        9153/TCP                       28h
  kube-system   prometheus-kube-prometheus-kube-etcd                 ClusterIP      None            &lt;none&gt;                                                        2379/TCP                       28h
  kube-system   prometheus-kube-prometheus-kube-controller-manager   ClusterIP      None            &lt;none&gt;                                                        10252/TCP                      28h
  monitoring    prometheus-kube-prometheus-alertmanager              ClusterIP      10.43.195.99    &lt;none&gt;                                                        9093/TCP                       28h
  monitoring    prometheus-prometheus-node-exporter                  ClusterIP      10.43.171.218   &lt;none&gt;                                                        9100/TCP                       28h
  monitoring    prometheus-grafana                                   ClusterIP      10.43.20.165    &lt;none&gt;                                                        80/TCP                         28h
  monitoring    prometheus-kube-prometheus-prometheus                ClusterIP      10.43.207.29    &lt;none&gt;                                                        9090/TCP                       28h
  monitoring    prometheus-kube-state-metrics                        ClusterIP      10.43.229.14    &lt;none&gt;                                                        8080/TCP                       28h
  kube-system   prometheus-kube-prometheus-kubelet                   ClusterIP      None            &lt;none&gt;                                                        10250/TCP,10255/TCP,4194/TCP   28h
  monitoring    alertmanager-operated                                ClusterIP      None            &lt;none&gt;                                                        9093/TCP,9094/TCP,9094/UDP     28h
  monitoring    prometheus-operated                                  ClusterIP      None            &lt;none&gt;                                                        9090/TCP                       28h
  kube-system   traefik                                              LoadBalancer   10.43.20.17     192.168.76.200,192.168.76.201,192.168.76.202,192.168.76.203   80:31131/TCP,443:31562/TCP     5d21h
  </code></pre>
  <p>Namespaces:</p>
  <pre><code>$ kubectl get namespaces 
  NAME              STATUS   AGE
  kube-system       Active   5d21h
  default           Active   5d21h
  kube-public       Active   5d21h
  kube-node-lease   Active   5d21h
  monitoring        Active   28h
  </code></pre>
  <p>But I couldn't reach the grafana service.</p>
  <p>Fair enough I thought, let's define an Ingress but it didn't work:</p>
  <pre><code>---
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: grafana-ingress
    namespace: monitoring
    annotations:
      kubernetes.io/ingress.class: traefik
  spec:
    rules:
    - http:
        paths:
        - pathType: Prefix
          path: /
          backend:
            service:
              name: prometheus-grafana
              port:
                number: 80
  </code></pre>
  <p>I have no idea why it isn't getting to the service and I can't really see where the problem is, although I understand containers, etc. (I first had everything running on docker swarm), I don't really know where, if anywhere, it would be shown in the logs.</p>
  <p>I've spent the past couple of days trying all sorts of things and I finally found a hint about name spaces and problems calling services and something called &quot;type: ExternalName&quot;.</p>
  <p>I checked with curl from a pod inside the cluster and it is delivering the data inside of the &quot;monitoring&quot; name space but traefik can't get there or maybe even see it?</p>
  <p>Having looked at the Traefik documentation I found this regarding <a href="https://doc.traefik.io/traefik/providers/kubernetes-crd/#namespaces" rel="nofollow noreferrer">namespaces</a> but I have no idea where I would start to find the mentioned:</p>
  <pre><code>providers:
    kubernetesCRD:
      namespaces:
  </code></pre>
  <p>I'm assuming that k3s has set this up correctly as an empty array because I can't find anything on their site that tells me what to do with their combination of &quot;klipper-lb&quot; and &quot;traefik&quot;.</p>
  <p>I finally tried to define another service with an external name:</p>
  <pre><code>---
  apiVersion: v1
    kind: Service
    metadata:
      name: grafana-named
      namespace: kube-system
    spec:
      type: ExternalName
      externalName: prometheus-grafana.monitoring.svc.cluster.local
      ports:
      - name: service
        protocol: TCP
        port: 80
        targetPort: 80
  ---
  apiVersion: networking.k8s.io/v1
  kind: Ingress
    metadata:
      name: grafana-ingress
      namespace: kube-system
      annotations:
        kubernetes.io/ingress.class: traefik
    spec:
      rules:
      - http:
        paths:
        - pathType: Prefix
          path: /
          backend:
            service:
              name: grafana-named
              port:
                number: 80
  </code></pre>
  <p>After 2-3 days, I've tried everything I can think of, google everything under the sun and I can't get to grafana from outside of the internal cluster nodes.</p>
  <p>I am at a loss as to how I can make anything work with k3s. I installed Lens on my main PC and can see almost everything there, but I think that the missing metrics information requires an Ingress or something like that too.</p>
  <p>What do I have to do to get traefik to do what I think is basically it's job, route incoming requests to the backend services?</p>