<p>I've set up a Kubernetes cluster on Ubuntu (trusty) based on the <a href="http://kubernetes.io/docs/getting-started-guides/docker/" rel="nofollow">Running Kubernetes Locally via Docker</a> guide, deployed a DNS and run Heapster with an InfluxDB backend and a Grafana UI.</p>

<p>Everything seems to run smoothly except for Grafana, which doesn't show any graphs but the message <code>No datapoints</code> in its diagrams: <a href="http://i.stack.imgur.com/W2eLv.png" rel="nofollow">Screenshot</a></p>

<p>After checking the Docker container logs I found out that Heapster is is unable to access the kubelet API (?) and therefore no metrics are persisted into InfluxDB:</p>

<pre><code>user@host:~$ docker logs e490a3ac10a8
I0701 07:07:30.829745       1 heapster.go:65] /heapster --source=kubernetes:https://kubernetes.default --sink=influxdb:http://monitoring-influxdb:8086
I0701 07:07:30.830082       1 heapster.go:66] Heapster version 1.2.0-beta.0
I0701 07:07:30.830809       1 configs.go:60] Using Kubernetes client with master "https://kubernetes.default" and version v1
I0701 07:07:30.831284       1 configs.go:61] Using kubelet port 10255
E0701 07:09:38.196674       1 influxdb.go:209] issues while creating an InfluxDB sink: failed to ping InfluxDB server at "monitoring-influxdb:8086" - Get http://monitoring-influxdb:8086/ping: dial tcp 10.0.0.223:8086: getsockopt: connection timed out, will retry on use
I0701 07:09:38.196919       1 influxdb.go:223] created influxdb sink with options: host:monitoring-influxdb:8086 user:root db:k8s
I0701 07:09:38.197048       1 heapster.go:92] Starting with InfluxDB Sink
I0701 07:09:38.197154       1 heapster.go:92] Starting with Metric Sink
I0701 07:09:38.228046       1 heapster.go:171] Starting heapster on port 8082
I0701 07:10:05.000370       1 manager.go:79] Scraping metrics start: 2016-07-01 07:09:00 +0000 UTC, end: 2016-07-01 07:10:00 +0000 UTC
E0701 07:10:05.008785       1 kubelet.go:230] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://127.0.0.1:10255/stats/container/": Post http://127.0.0.1:10255/stats/container/: dial tcp 127.0.0.1:10255: getsockopt: connection refused
I0701 07:10:05.009119       1 manager.go:152] ScrapeMetrics: time: 8.013178ms size: 0
I0701 07:11:05.001185       1 manager.go:79] Scraping metrics start: 2016-07-01 07:10:00 +0000 UTC, end: 2016-07-01 07:11:00 +0000 UTC
E0701 07:11:05.007130       1 kubelet.go:230] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://127.0.0.1:10255/stats/container/": Post http://127.0.0.1:10255/stats/container/: dial tcp 127.0.0.1:10255: getsockopt: connection refused
I0701 07:11:05.007686       1 manager.go:152] ScrapeMetrics: time: 5.945236ms size: 0
W0701 07:11:25.010298       1 manager.go:119] Failed to push data to sink: InfluxDB Sink
I0701 07:12:05.000420       1 manager.go:79] Scraping metrics start: 2016-07-01 07:11:00 +0000 UTC, end: 2016-07-01 07:12:00 +0000 UTC
E0701 07:12:05.002413       1 kubelet.go:230] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://127.0.0.1:10255/stats/container/": Post http://127.0.0.1:10255/stats/container/: dial tcp 127.0.0.1:10255: getsockopt: connection refused
I0701 07:12:05.002467       1 manager.go:152] ScrapeMetrics: time: 1.93825ms size: 0
E0701 07:12:12.309151       1 influxdb.go:150] Failed to create infuxdb: failed to ping InfluxDB server at "monitoring-influxdb:8086" - Get http://monitoring-influxdb:8086/ping: dial tcp 10.0.0.223:8086: getsockopt: connection timed out
I0701 07:12:12.351348       1 influxdb.go:201] Created database "k8s" on influxDB server at "monitoring-influxdb:8086"
I0701 07:13:05.001052       1 manager.go:79] Scraping metrics start: 2016-07-01 07:12:00 +0000 UTC, end: 2016-07-01 07:13:00 +0000 UTC
E0701 07:13:05.015947       1 kubelet.go:230] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://127.0.0.1:10255/stats/container/": Post http://127.0.0.1:10255/stats/container/: dial tcp 127.0.0.1:10255: getsockopt: connection refused
...
</code></pre>

<p>I found a few issues on GitHub describing similar problems that made me understand that Heapster doesn't access the kubelet (via the node's loopback) but itself (via the container's loopback) instead. However, I fail to reproduce their solutions:</p>

<p><strong>github.com/kubernetes/heapster/issues/1183</strong></p>

<blockquote>
  <p>You should either use host networking for Heapster pod or configure your cluster in a way that the node has a regular name not 127.0.0.1. The current problem is that node name is resolved to Heapster localhost. Please reopen in case of more problems.</p>
</blockquote>

<p><em>-@piosz</em></p>

<ul>
<li>How do I enable "host networking" for my Heapster pod?</li>
<li>How do I configure my cluster/node to use a regular name not 127.0.0.1?</li>
</ul>

<p><strong>github.com/kubernetes/heapster/issues/744</strong></p>

<blockquote>
  <p>Fixed by using better options in hyperkube, thanks for the help!</p>
</blockquote>

<p><em>-@ddispaltro</em></p>

<ul>
<li>Is there a way to solve this issue by adding/modifying kubelet's option flags in <code>docker run</code>? <br> I tried setting<code>--hostname-override=&lt;host's eth0 IP&gt;</code> and <code>--address=127.0.0.1</code> (as suggested in the last answer of this GitHub issue) but Heapster's container log then states: <br> <br><code>I0701 08:23:05.000566       1 manager.go:79] Scraping metrics start: 2016-07-01 08:22:00 +0000 UTC, end: 2016-07-01 08:23:00 +0000 UTC
E0701 08:23:05.000962       1 kubelet.go:279] Node 127.0.0.1 is not ready
E0701 08:23:05.003018       1 kubelet.go:230] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://&lt;host's eth0 IP&gt;:10255/stats/container/": Post http://&lt;host's eth0 IP&gt;/stats/container/: dial tcp &lt;host's eth0 IP&gt;:10255: getsockopt: connection refused
</code></li>
</ul>

<p><strong>Namespace issue</strong></p>

<p>Could this problem be caused by the fact that I'm running Kubernetes API in <code>default</code> namespace and Heapster in <code>kube-system</code>?</p>

<pre><code>user@host:~$ kubectl get --all-namespaces pods
NAMESPACE     NAME                     READY     STATUS    RESTARTS   AGE
default       k8s-etcd-127.0.0.1       1/1       Running   0          18h
default       k8s-master-127.0.0.1     4/4       Running   1          18h
default       k8s-proxy-127.0.0.1      1/1       Running   0          18h
kube-system   heapster-lizks           1/1       Running   0          18h
kube-system   influxdb-grafana-e0pk2   2/2       Running   0          18h
kube-system   kube-dns-v10-4vjhm       4/4       Running   0          18h
</code></pre>

<hr>

<p><em>OS: Ubuntu 14.04.4 LTS (trusty) | 
Kubernetes: v1.2.5 | 
Docker: v1.11.2</em></p>

<p>Provide the below argument to your heapster configuration to resolve the issue.</p>

<p>--source=kubernetes:<a href="https://kubernetes.default:443?useServiceAccount=true&amp;kubeletHttps=true&amp;kubeletPort=10250&amp;insecure=true" rel="nofollow noreferrer">https://kubernetes.default:443?useServiceAccount=true&amp;kubeletHttps=true&amp;kubeletPort=10250&amp;insecure=true</a></p>
