<p>I am trying to figure out <strong>how to get pod labels into the metric tags from kubelet metrics using prometheus-stack</strong>. In our environment, we need to hash pod names (due to length limitations) so our app name, env, and unit name are saved in pod labels.</p>
<p>We are using prometheus-stack (helm installation) to collect metrics from kubelet (<code>/metrics</code>, <code>/metrics/cadvisor</code>) and due to the lack of pod labels in metrics tags, it's difficult to know which metric belongs to which application.</p>
<p>Prometheus-stack is using <code>sd_kubernetes_config</code> with endpoint rule to collect kubelet metrics, where <code>__meta</code> tags for pod labels cannot be used. Is there another way how to get that labels in metric tags?</p>
<p>I also tried to collect pod_labels metric using <code>kubeStateMetrics</code>, where I can get metric that contains pod labels, but I cannot figure out how to display both metrics in a way that metric from cadvisor will show its value and metric from <code>kubeStateMetrics</code> will be used to display its labels (in Prometheus graph).</p>
<p>Thanks for any advice.</p>

<p>As far as I know you can really use filtering metrics <a href="https://stackoverflow.com/questions/60067654/prometheus-filtering-based-on-labels">based on pod labels</a>. Look at the original answer:</p>
<blockquote>
<p>You can use  <code>+</code>  operator to join metrics. Here,  <code>group_left()</code>  will include the extra label:  <code>label_source</code>  from the right metric  <code>kube_pod_labels</code>. The metric you're joining is forced to zero ( i.e.  <code>0 * kube_pod_labels</code>  ) so that it doesn't affect the result of first metric.</p>
</blockquote>
<pre><code>(
kube_pod_info{namespace=&quot;test&quot;}
)
   + on(namespace) group_left(label_source)
(
   0 * kube_pod_labels
)
</code></pre>
<p>In fact, metrics then can be long and nasty. However, it is an effective method in prometheus to create what you expect, which is kubelet metrics with pod labels.</p>
<p>Look also at <a href="https://ypereirareis.github.io/blog/2020/02/21/how-to-join-prometheus-metrics-by-label-with-promql/" rel="nofollow noreferrer">this guide</a> - it is described how to join Prometheus metrics by label with PromQL and <a href="https://grafana.com/blog/2021/08/04/how-to-use-promql-joins-for-more-effective-queries-of-prometheus-metrics-at-scale/" rel="nofollow noreferrer">this another guide</a> - How to use PromQL joins for more effective queries of Prometheus metrics at scale.</p>
