<p>I am setting up Grafana on Kubernetes with some custom dashboards and Prometheus as a default data source.</p>
<p>All services are running behind Traefik. Prometheus is running at <a href="https://myapp.dev/svc/prometheus/" rel="nofollow noreferrer">https://myapp.dev/svc/prometheus/</a> and Grafana is running at <a href="https://myapp.dev/svc/grafana/" rel="nofollow noreferrer">https://myapp.dev/svc/grafana/</a></p>
<p>I can log in to both grafana(also able to see dashboards that were added to config) and prometheus, however when I test the Prometheus data source, I see the following error in grafana logs:</p>
<p><a href="https://i.stack.imgur.com/XCAuQ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/XCAuQ.png" alt="img1" /></a></p>
<pre class="lang-sh prettyprint-override"><code>lvl=eror msg=&quot;Data proxy error&quot; logger=data-proxy-log userId=1 orgId=1 uname=xxx path=/api/datasources/proxy/1/api/v1/query remote_addr=192.168.64.1 referer=https://myapp.dev/svc/grafana/datasources/edit/1/ error=&quot;http: proxy error: dial tcp: lookup prometheus on 10.96.0.10:53: no such host&quot;
</code></pre>
<p><strong>Grafana deployment</strong></p>
<pre class="lang-yaml prettyprint-override"><code>kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: default
  name: grafana-deployment
  labels:
    app: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana
        image: grafana/grafana:latest
        ports:
        - containerPort: 3000
        volumeMounts:
          - mountPath: /etc/grafana/provisioning/datasources/
            name: datasource-volume
          - mountPath: /etc/grafana/provisioning/dashboards/
            name: dashboard-volume
          - mountPath: /var/lib/grafana
            name: grafana-storage
      volumes:
        - name: grafana-storage
          emptyDir: {}
        - configMap:
            defaultMode: 420
            name: grafana-datasource-cm
          name: datasource-volume
        - configMap:
            defaultMode: 420
            name: grafana-dashboard-cm
          name: dashboard-volume

</code></pre>
<p><strong>Configmap</strong></p>
<pre class="lang-yaml prettyprint-override"><code>kind: ConfigMap
apiVersion: v1
metadata:
  name: grafana-datasource-cm
data:
  datasource.yml: |-
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      orgId: 1
      access: proxy
      url: http://prometheus:9090/svc/prometheus/
      basicAuth: false
</code></pre>
<p>I am not sure on how to fix this error and test Prometheus datasource.</p>

<p>Sounds like it cannot find the <code>prometheus</code> service in the cluster. Maybe:</p>
<ul>
<li>It's in a different namespace❓</li>
<li>It's defined but with a different name❓</li>
</ul>
<p>If it's a different namespace you can try changing your config with the full FQDN of the service. <code>prometheus.&lt;namespace&gt;.svc.cluster.local</code></p>
<pre class="lang-yaml prettyprint-override"><code>kind: ConfigMap
apiVersion: v1
metadata:
  name: grafana-datasource-cm
data:
  datasource.yml: |-
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      orgId: 1
      access: proxy
      url: http://prometheus.&lt;namespace&gt;.svc.cluster.local:9090/svc/prometheus/
      basicAuth: false
</code></pre>
<p>See if you can get to the service from your grafana pod 🔧:</p>
<pre class="lang-sh prettyprint-override"><code>$ kubectl exec -it &lt;grafana-pod&gt; sh
# curl http://prometheus.&lt;namespace&gt;.svc.cluster.local:9090
</code></pre>
<p>✌️☮️</p>
