<p>Use <code>helm</code> installed <code>Prometheus</code> and <code>Grafana</code> on <code>minikube</code> at local.</p>

<pre><code>$ helm install stable/prometheus
$ helm install stable/grafana
</code></pre>

<p>Prometheus server, alertmanager grafana can run after set port-forward:</p>

<pre><code>$ export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
$ kubectl --namespace default port-forward $POD_NAME 9090

$ export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus,component=alertmanager" -o jsonpath="{.items[0].metadata.name}")
$ kubectl --namespace default port-forward $POD_NAME 9093

$ export POD_NAME=$(kubectl get pods --namespace default -l "app=excited-crocodile-grafana,component=grafana" -o jsonpath="{.items[0].metadata.name}")
$ kubectl --namespace default port-forward $POD_NAME 3000
</code></pre>

<p><a href="https://i.stack.imgur.com/sD6Di.png" rel="noreferrer"><img src="https://i.stack.imgur.com/sD6Di.png" alt="enter image description here"></a>
<a href="https://i.stack.imgur.com/UPXUF.png" rel="noreferrer"><img src="https://i.stack.imgur.com/UPXUF.png" alt="enter image description here"></a></p>

<p>Add Data Source from grafana, got <code>HTTP Error Bad Gateway</code> error:</p>

<p><a href="https://i.stack.imgur.com/Cnn1B.png" rel="noreferrer"><img src="https://i.stack.imgur.com/Cnn1B.png" alt="enter image description here"></a></p>

<p>Import dashboard 315 from:</p>

<blockquote>
  <p><a href="https://grafana.com/dashboards/315" rel="noreferrer">https://grafana.com/dashboards/315</a></p>
</blockquote>

<p>Then check <code>Kubernetes cluster monitoring (via Prometheus)</code>, got <code>Templating init failed</code> error:</p>

<p><a href="https://i.stack.imgur.com/Wm2kt.png" rel="noreferrer"><img src="https://i.stack.imgur.com/Wm2kt.png" alt="enter image description here"></a></p>

<p>Why?</p>

<p>In the HTTP settings of Grafana you set <code>Access</code> to <code>Proxy</code>, which means that Grafana wants to access Prometheus. Since Kubernetes uses an overlay network, it is a different IP.</p>

<p>There are two ways of solving this:</p>

<ol>
<li>Set <code>Access</code> to <code>Direct</code>, so the browser directly connects to Prometheus.</li>
<li>Use the Kubernetes-internal IP or domain name. I don't know about the Prometheus Helm-chart, but assuming there is a <code>Service</code> named <code>prometheus</code>, something like <code>http://prometheus:9090</code> should work.</li>
</ol>
