<p>I annotate my Kubernetes objects with things like version and whom to contact when there are failures. How would I relay this information to Prometheus, knowing that these annotation values will frequently change? I can't capture this information in Prometheus labels, as they serve as the primary key for a target (e.g. if the version changes, it's a new target altogether, which I don't want). Thanks!</p>

<p>I just wrote a blog post about this exact topic! <a href="https://www.weave.works/aggregating-pod-resource-cpu-memory-usage-arbitrary-labels-prometheus/" rel="nofollow noreferrer">https://www.weave.works/aggregating-pod-resource-cpu-memory-usage-arbitrary-labels-prometheus/</a></p>

<p>The trick is Kubelet/cAdvisor doesn't expose them directly, so I run a little exporter which does, and join this with the pod name in PromQL.  The exporter is: <a href="https://github.com/tomwilkie/kube-api-exporter" rel="nofollow noreferrer">https://github.com/tomwilkie/kube-api-exporter</a></p>

<p>You can do a join in Prometheus like this:</p>

<pre><code>sum by (namespace, name) (
      sum(rate(container_cpu_usage_seconds_total{image!=""}[5m])) by (pod_name, namespace)
    * on (pod_name) group_left(name)
      k8s_pod_labels{job="monitoring/kube-api-exporter"}
   )
</code></pre>

<p>Here I'm using a label called "name", but it could be any label.  </p>

<p>We use the same trick to get metrics (such as error rate) by version, which we then use to drive our continuous deployment system.  kube-api-exporter exports a bunch of useful meta-information about Kubernetes objects to Prometheus.</p>

<p>Hope this helps!</p>
