<p>I'd love to rename or drop a label from a <code>/metrics</code> endpoint within my metric. The metric itself is from the <code>kube-state-metrics</code> application, so nothing extraordinary. The metric looks like this:</p>
<pre><code>kube_pod_container_resource_requests{container=&quot;alertmanager&quot;, instance=&quot;10.10.10.10:8080&quot;, funday_monday=&quot;blubb&quot;, job=&quot;some-kube-state-metrics&quot;, name=&quot;kube-state-metrics&quot;, namespace=&quot;monitoring&quot;, node=&quot;some-host-in-azure-1234&quot;, pod=&quot;alertmanager-main-1&quot;, resource=&quot;memory&quot;, uid=&quot;12345678-1234-1234-1234-123456789012&quot;, unit=&quot;byte&quot;} 209715200
</code></pre>
<p>The label I'd love to replace is <code>instance</code> because it refers to the host which runs the <code>kube-state-metrics</code> application and I don't care about that. I want to have the value of <code>node</code> in <code>instance</code> and I've been trying so for hours now and can't find a way - I wonder if it's not possible at all!?</p>
<p>The way I'm grabbing the <code>/metrics</code> endpoint is through the means of a scrape-config which looks as follows:</p>
<pre><code>- job_name: some-kube-state-metrics
      scrape_interval: 30s
      scrape_timeout: 10s
      metrics_path: /metrics
      kubernetes_sd_configs:
      - api_server: null
        role: pod
      scheme: http
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_labelpresent_kubeStateMetrics]
        regex: true
        action: keep
      - source_labels: [__meta_kubernetes_pod_label_name]
        regex: (.*)
        replacement: $1
        target_label: name
        action: replace
      - source_labels: [__meta_kubernetes_pod_container_port_name]
        separator: ;
        regex: http
        replacement: $1
        action: keep
      - source_labels: [node]
        regex: (.*)
        replacement: blubb
        target_label: funday_monday
        action: replace
      - action: labeldrop
        regex: &quot;unit=(.*)&quot;
      - source_labels: [ __name__ ]
        regex: 'kube\_pod\_container\_resource\_requests'
        action: drop
</code></pre>
<p>As you can tell, I've been trying to drop labels as well, namely the <code>unit</code>-label (just for testing purposes) and I also tried to drop the metrics all together.
The <code>funday_monday</code> is an example that changed because I wanted to know if static relabels are possible (it works!) - before it looked like this:</p>
<pre><code>- source_labels: [node]
        regex: (.*)
        replacement: $1
        target_label: funday_monday
        action: replace
</code></pre>
<p>Help is appreciated.</p>

<p>The problem is that you are doing those operations at the wrong time. <a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config" rel="nofollow noreferrer">relabel_configs</a> happens before metrics are actually gathered, so, at this time, you can only manipulate the labels that you got from <a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config" rel="nofollow noreferrer">service discovery</a>.</p>
<p>That <code>node</code> label comes from the exporter. Therefore, you need to do this relabeling action under <a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#metric_relabel_configs" rel="nofollow noreferrer">metric_relabel_configs</a>:</p>
<pre class="lang-yaml prettyprint-override"><code>metric_relabel_configs:
- source_labels: [node]
  target_label: instance
</code></pre>
<p>Same goes for dropping metrics. If you wish a bit more info, I answered a similar question here: <a href="https://stackoverflow.com/questions/70355080/prometheus-relabel-config-drop-action-not-working/70359287#70359287">prometheus relabel_config drop action not working</a></p>
