<p>I'm working on a setup where we run our Java services in docker containers hosted on a kubernetes platform. </p>

<p>On want to create a dashboard where I can monitor the heap usage of all instances of a service in my grafana. Writing metrics to statsd with the pattern:</p>

<p><code>&lt;servicename&gt;.&lt;containerid&gt;.&lt;processid&gt;.heapspace</code> works well, I can see all heap usages in my chart. </p>

<p>After a redeployment, the container names change, so new values are added to the existing graph. My problem is, that the old lines continue to exist at the position of the last value received, but the containers are already dead.</p>

<p>Is there any simple solution for this in grafana? Can I just say: if you didn't receive data for a metric for more than X seconds, abort the chart line?</p>

<p><img src="https://i.stack.imgur.com/VDKgv.png" alt="Many containers exit at 14:00, but the chart continues"></p>

<p>Update:
Upgrading to the newest Grafana Version and Setting "null" as value for "Null value" in Stacking and Null Value didn't work.</p>

<p>Maybe it's a problem with statsd?</p>

<p>I'm sending data to statsd in form of:</p>

<p><code>felix.javaclient.machine&lt;number&gt;-&lt;pid&gt;.heap:&lt;heapvalue&gt;|g</code></p>

<p>Is anything wrong with this?</p>

<p>This can happen for 2 reasons, because grafana is using the "connected" setting for null values, and/or (as is the case here) because statsd is sending the previously-seen value for the gauge when there are no updates in the current period.</p>

<h2>Grafana Config</h2>

<p>You'll want to make 2 adjustments to your graph config:</p>

<p>First, go to the "Display" tab and under "Stacking &amp; Null value" change "Null value" to "null", that will cause Grafana to stop showing the lines when there is no data for a series.</p>

<p>Second, if you're using a legend you can go to the "Legend" tab and under "Hide series" check the "With only nulls" checkbox, that will cause items to only be displayed in the legend if they have a non-null value during the graph period.</p>

<h2>statsd Config</h2>

<p>The <a href="https://github.com/etsy/statsd/blob/master/docs/metric_types.md#gauges" rel="nofollow">statsd documentation for gauge metrics</a> tells us:</p>

<blockquote>
  <p>If the gauge is not updated at the next flush, it will send the
  previous value. You can opt to send no metric at all for this gauge,
  by setting <code>config.deleteGauges</code></p>
</blockquote>

<p>So, the grafana changes alone aren't enough in this case, because the values in graphite aren't actually null (since statsd keeps sending the last reading).  If you change the statsd config to have <code>deleteGauges: true</code> then statsd won't send anything and graphite will contain the null values we expect.</p>

<h2>Graphite Note</h2>

<p>As a side note, a setup like this will cause your data folder to grow continuously as you create new series each time a container is launched.  You'll definitely want to look into removing old series after some period of inactivity to avoid filling up the disk.  If you're using graphite with whisper that can be as simple as a cron task running <code>find /var/lib/graphite/whisper/ -name '*.wsp' -mtime +30 -delete</code> to remove whisper files that haven't been modified in the last 30 days. </p>
