<p>I am looking for disk space usage metrics for EKS cluster that can be monitored.
I came across three different metric alerts : <br>
1.</p>
<pre><code>k8s-high-filesystem-usage:
  name: &quot;(k8s) High Filesystem Usage Detected&quot;
  type: metric alert
  query: |
    avg(last_10m):avg:kubernetes.filesystem.usage_pct{*} by {cluster_name} &gt; 90
  message: |
    {{#is_warning}}
    {{cluster_name.name}} filesystem usage greater than 80% for 10 minutes
    {{/is_warning}}
    {{#is_alert}}
    {{cluster_name.name}} filesystem usage greater than 90% for 10 minutes
    {{/is_alert}}
</code></pre>
<ol start="2">
<li></li>
</ol>
<pre><code>k8s-high-disk-usage:
  name: &quot;(k8s) High Disk Usage Detected&quot;
  type: metric alert
  query: |
    min(last_5m):min:kubelet.volume.stats.used_bytes{*} by {cluster_name} / avg:kubernetes.kubelet.volume.stats.capacity_bytes{*} by {cluster_name} * 100 &gt; 90
  message: |
    ({{cluster_name.name}} High disk usage detected

</code></pre>
<ol start="3">
<li></li>
</ol>
<pre><code>k8s-high-disk-usage:
  name: &quot;(k8s) High Disk Usage Detected&quot;
  type: metric alert
  query: |
    min(last_5m):min:system.disk.used{*} by {host,cluster_name} / avg:system.disk.total{*} by {host,cluster_name} * 100 &gt; 90
  message: |
    ({{cluster_name.name}} High disk usage detected on {{host.name}}
</code></pre>
<p>What does these three metrics mean? When can I use these?</p>

<p>This is a confusing area that is poorly documented. I'm glad you asked the question.</p>
<p>First, some background on each metric.</p>
<h3><code>system.disk.used</code></h3>
<p>This metric is the most straightforward: disk space used, in bytes, of the disk partitions on the k8s host. This is a core check collected by the Datadog agent. Find the source for this metric in <a href="https://github.com/DataDog/datadog-agent/blob/ffb7ee79992815a77533f7950b7c1c45ae91fcd8/pkg/collector/corechecks/system/disk/disk.go" rel="nofollow noreferrer"><code>corechecks/system/disk/disk.go</code></a>. The check will report disk usage for each volume on the host.</p>
<h3><code>kubernetes.filesystem.usage_pct</code></h3>
<p>This metric reports disk space used for each node in a k8s cluster. The data is pulled from the metrics published by the kubelet under <code>/stats/summary</code>. You can query the data directly using <code>kubectl</code>, e.g.</p>
<pre><code>kubectl get --raw /api/v1/nodes/&lt;node name&gt;/proxy/stats/summary
</code></pre>
<p>The code can be found by tracing the calls in <a href="https://github.com/DataDog/datadog-agent/blob/ffb7ee79992815a77533f7950b7c1c45ae91fcd8/pkg/collector/corechecks/cluster/orchestrator/" rel="nofollow noreferrer">the cluster orchestrator</a> and <a href="https://github.com/DataDog/datadog-agent/blob/ffb7ee79992815a77533f7950b7c1c45ae91fcd8/pkg/util/kubernetes/kubelet/kubelet.go" rel="nofollow noreferrer">kubelet util</a> files. This metric also reports disk usage percentage <em>by pod</em>, <em>device</em>, and other potentially useful tags.</p>
<h3><code>kubernetes.kubelet.volume.stats.used_bytes</code></h3>
<p>This metric reports data about pods persistent volume claims. You can find out how many bytes are used by each pvc. This metric will only exist for pods with persistent volume claims. This is also in the <a href="https://github.com/DataDog/datadog-agent/blob/ffb7ee79992815a77533f7950b7c1c45ae91fcd8/pkg/collector/corechecks/cluster/orchestrator/processors/k8s/persistentvolumeclaim.go" rel="nofollow noreferrer">cluster/orchestrator code base</a>.</p>
<p>So, with that background in mind, when would you use each metric?</p>
<p>Use <code>system.disk.used</code> to track <em>the disk usage at the node level</em>. If you want to monitor the disk usage of hosts, watch this value. You should monitor on the <code>device</code> tag - you will be most interested in the physical disk partitions and the Docker volumes. You can probably ignore the <code>shm</code> and <code>tmpfs</code> volumes (virtual memory). Note that since this is a core check, this metric is reported for <em>any</em> host with the datadog agent installed, not just k8s hosts.</p>
<p>Use <code>kubernetes.filesystem.usage_pct</code> to track <em>disk usage by k8s hosts</em>.  It probably makes sense to monitor with <code>cluster_name</code> <em>and</em> with <code>host</code>, and to use the max value, e.g. update your query to:</p>
<pre><code>avg(last_10m):max:kubernetes.filesystem.usage_pct{*} by {cluster_name,host}
</code></pre>
<p>If you want pod-level usage, you can also add <code>pod_name</code> to the query.</p>
<p>Finally, use <code>kubernetes.kubelet.volume.stats.used_bytes</code> to monitor disk space of persistent volume claims. You'll want to add the <code>persistentvolumeclaim</code> tag to the query so you know which claim you're looking at.</p>
