<p>I have a K8s 1.7 cluster using <strong>vSphere</strong> as persistent storage provider. I have also deployed <strong>Prometheus</strong>, <strong>node_exporter</strong> and <strong>kube-state-metrics</strong>.<p>
I'm trying to find a way to monitor a persistent volume's usage using <strong>Prometheus</strong>. I have added custom labels to some PVs, eg. <code>app=rabbitmq-0</code>, etc. </p>

<p>How can I combine <code>kube_persistentvolume_labels</code> with <code>node_filesystem_size</code> metrics so that I can query PV usage using my custom label?</p>

<p>PS.<br>
I know that K8s 1.8 directly exposes these metrics from kubelet as mentioned in <a href="https://stackoverflow.com/questions/44718268/how-to-monitor-disk-usage-of-kubernetes-persistent-volumes">How to monitor disk usage of kubernetes persistent volumes?</a> but currently a cluster upgrade is not an option.</p>

<p>Starting from (v1.3.0-rc.0 / 2018-03-23) in the <strong>kube-state-metrics</strong>, two metrics that can convert <a href="https://github.com/kubernetes/kube-state-metrics/blob/641f44d26ccbad2272d5bf3e10e43efd41e7976d/pkg/collectors/persistentvolume.go#L128" rel="nofollow noreferrer">PersistentVolume</a> and <a href="https://github.com/kubernetes/kube-state-metrics/blob/641f44d26ccbad2272d5bf3e10e43efd41e7976d/pkg/collectors/persistentvolumeclaim.go#L151" rel="nofollow noreferrer">PersistenVolumeClaims</a> labels to Prometheus labels accordingly were added:</p>

<pre><code>kube_persistentvolume_labels
kube_persistentvolumeclaim_lables
</code></pre>

<p>To get more details about implementing aggregation of metrics based on labels, consider reading these articles:</p>

<ul>
<li><a href="https://www.weave.works/blog/aggregating-pod-resource-cpu-memory-usage-arbitrary-labels-prometheus/" rel="nofollow noreferrer">Aggregating Pod resource (CPU, memory) usage by arbitrary labels with Prometheus</a></li>
<li><a href="https://www.robustperception.io/exposing-the-software-version-to-prometheus" rel="nofollow noreferrer">Exposing the software version to Prometheus</a></li>
</ul>
