<p>I deployed Prometheus and Grafana into my cluster.</p>

<p>When I open the dashboards I don't get data for pod CPU usage.</p>

<p>When I check Prometheus UI, it shows pods 0/0 up, however I have many pods running in my cluster.</p>

<p>What could be the reason? I have node exporter running in all of nodes.</p>

<p><a href="https://i.stack.imgur.com/HkXbL.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/HkXbL.png" alt="enter image description here"></a></p>

<p>Am getting this for kube-state-metrics,</p>

<pre><code>I0218 14:52:42.595711       1 builder.go:112] Active collectors: configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,jobs,limitranges,namespaces,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets
I0218 14:52:42.595735       1 main.go:208] Starting metrics server: 0.0.0.0:8080
</code></pre>

<p>Here is my Prometheus config file:
<a href="https://gist.github.com/karthikeayan/41ab3dc4ed0c344bbab89ebcb1d33d16" rel="nofollow noreferrer">https://gist.github.com/karthikeayan/41ab3dc4ed0c344bbab89ebcb1d33d16</a></p>

<p>I'm able to hit and get data for:</p>

<pre><code>http://localhost:8080/api/v1/nodes/&lt;my_worker_node&gt;/proxy/metrics/cadvisor
</code></pre>

karthikeayan
<p>As it was mentioned by <a href="https://stackoverflow.com/users/2436847/karthikeayan">karthikeayan</a> in comments:</p>
<blockquote>
<p>ok, i found something interesting in the <code>values.yaml</code> comments, <strong>prometheus.io/scrape:</strong> <em>Only scrape pods that have a value of true</em>, when i remove this relabel_config in k8s configmap, i got the data in prometheus ui.. unfortunately k8s configmap doesn't have comments, i believe helm will remove the comments before deploying it.</p>
</blockquote>
<p>And just for clarification:</p>
<blockquote>
<h3><a href="https://github.com/kubernetes/kube-state-metrics#kube-state-metrics-vs-metrics-server" rel="nofollow noreferrer">kube-state-metrics vs. metrics-server</a></h3>
<p>The <strong>metrics-server</strong> is a project that has been inspired by <strong>Heapster</strong> and is implemented to serve the goals of the Kubernetes Monitoring Pipeline. It is a cluster level component which periodically scrapes metrics from all Kubernetes nodes served by Kubelet through Summary API. The metrics are aggregated, stored in memory and served in Metrics API format. The metric-server stores the latest values only and is not responsible for forwarding metrics to third-party destinations.</p>
<p><strong>kube-state-metrics</strong> is focused on generating completely new metrics from Kubernetes' object state (e.g. metrics based on deployments, replica sets, etc.). It holds an entire snapshot of Kubernetes state in memory and continuously generates new metrics based off of it. And just like the metric-server it too is not responsibile for exporting its metrics anywhere.</p>
<p>Having kube-state-metrics as a separate project also enables access to these metrics from monitoring systems such as Prometheus.</p>
</blockquote>
