<p>I have a working Kubernetes cluster that I want to monitor with Grafana.</p>

<p>I have been trying out many dashboards from <a href="https://grafana.com/dashboards" rel="nofollow noreferrer">https://grafana.com/dashboards</a> but they all seem to have some problems: it looks like there's a mismatch between the Prometheus metric names and what the dashboard expects.</p>

<p>Eg if I look at this recently released, quite popular dashboard: <a href="https://grafana.com/dashboards/5309/revisions" rel="nofollow noreferrer">https://grafana.com/dashboards/5309/revisions</a></p>

<p>I end up with many "holes" when running it: </p>

<p><a href="https://i.stack.imgur.com/TJ2ls.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/TJ2ls.png" alt="grafana dashboard with missing values"></a></p>

<p>Looking into the panel configuration, I see that the issues come from  small key changes, eg <code>node_memory_Buffers</code> instead of <code>node_memory_Buffers_bytes</code>.</p>

<p>Similarly the dashboard expects <code>node_disk_bytes_written</code> when Prometheus provides <code>node_disk_written_bytes_total</code>.</p>

<p>I have tried out a <em>lot</em> of Kubernetes-specific dashboards and I have the same problem with almost all of them.</p>

<p>Am I doing something wrong?</p>

<p>The Prometheus node exporter changed a lot of the metric names in the 0.16.0 version to conform to new naming conventions.</p>

<p>From <a href="https://github.com/prometheus/node_exporter/releases/tag/v0.16.0" rel="nofollow noreferrer">https://github.com/prometheus/node_exporter/releases/tag/v0.16.0</a>:</p>

<blockquote>
  <p><strong>Breaking changes</strong></p>
  
  <p>This release contains major breaking changes to metric names. Many
  metrics have new names, labels, and label values in order to conform
  to current naming conventions.</p>
  
  <ul>
  <li>Linux node_cpu metrics now break out <code>guest</code> values into separate
  metrics. </li>
  <li>Many counter metrics have been renamed to <code>include _total</code>. </li>
  <li>Many metrics have been renamed/modified to include
  base units, for example <code>node_cpu</code> is now <code>node_cpu_seconds_total</code>.</li>
  </ul>
</blockquote>

<p>See also the <a href="https://github.com/prometheus/node_exporter/blob/v0.16.0/docs/V0_16_UPGRADE_GUIDE.md" rel="nofollow noreferrer">upgrade guide</a>. One of its suggestion is to use <a href="https://github.com/prometheus/node_exporter/blob/v0.16.0/docs/example-16-compatibility-rules.yml" rel="nofollow noreferrer">compatibility rules</a> that will create duplicate metrics with the old names.</p>

<p>Otherwise use version 0.15.x until the dashboards are updated, or fix them!</p>
