<p>I installed stable/prometheus from helm. By default the job_name <code>kubernetes-service-endpoints</code> contains <code>node-exporter</code> and <code>kube-state-metrics</code> as component label. I added the below configuration in prometheus.yml to include namespace, pod and node labels.</p>

<pre><code>      - source_labels: [__meta_kubernetes_namespace]
        separator: ;
        regex: (.*)
        target_label: namespace
        replacement: $1
        action: replace
      - source_labels: [__meta_kubernetes_pod_name]
        separator: ;
        regex: (.*)
        target_label: pod
        replacement: $1
        action: replace
      - source_labels: [__meta_kubernetes_pod_node_name]
        separator: ;
        regex: (.*)
        target_label: node
        replacement: $1
        action: replace
</code></pre>

<p><code>kube_pod_info{component="kube-state-metrics"}</code> already had namespace, pod and node labels and hence exported_labels were generated. And the metric <code>node_cpu_seconds_total{component="node-exporter"}</code> now correctly has labels namespace, pod and node. </p>

<p>To have these labels correctly, I need to have these 3 labels present in both the above metric names. To achieve that I can override value of exported_labels. I tried adding the below config but to no avail.</p>

<pre><code>      - source_labels: [__name__, exported_pod]
        regex: "kube_pod_info;(.+)"
        target_label: pod
      - source_labels: [__name__, exported_namespace]
        regex: "kube_pod_info;(.+)"
        target_label: namespace
      - source_labels: [__name__, exported_node]
        regex: "kube_pod_info;(.+)"
        target_label: node
</code></pre>

<p>Similar approach was mentioned <a href="https://stackoverflow.com/questions/54235797/how-to-rename-label-within-a-metric-in-prometheus">here</a>. I can't see the issue with my piece of code. Any directions to resolve would be very helpful.</p>

<p><strong>Updated - (adding complete job)</strong> </p>

<pre><code>    - job_name: kubernetes-service-endpoints
      kubernetes_sd_configs:
      - role: endpoints

      metric_relabel_configs:
      - source_labels: [__name__, exported_pod]
        regex: "kube_pod_info;(.+)"
        target_label: pod
      - source_labels: [__name__, exported_namespace]
        regex: "kube_pod_info;(.+)"
        target_label: namespace
      - source_labels: [__name__, exported_node]
        regex: "kube_pod_info;(.+)"
        target_label: node

      relabel_configs:
      - action: keep
        regex: true
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scrape
      - action: replace
        regex: (https?)
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scheme
        target_label: __scheme__
      - action: replace
        regex: (.+)
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_path
        target_label: __metrics_path__
      - action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        source_labels:
        - __address__
        - __meta_kubernetes_service_annotation_prometheus_io_port
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        regex: (.*)
        replacement: $1
        separator: ;
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: pod
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_node_name
        target_label: node
</code></pre>

<p><strong>And the result from promql</strong></p>

<p><a href="https://i.stack.imgur.com/y83a1.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/y83a1.png" alt="kube_pod_info results"></a></p>

<p>So your goal is to rename the metric labels <code>exported_pod</code> to <code>pod</code>, etc, for the <code>kube_pod_info</code> metric?</p>

<p>In that case, you need <a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#metric_relabel_configs" rel="noreferrer">metric relabelling</a> which is done when metrics are fetched from targets:</p>

<pre><code>- job_name: 'kubernetes-service-endpoints'

  kubernetes_sd_configs:
    - role: endpoints

  metric_relabel_configs:
    - source_labels: [__name__, exported_pod]
      regex: "kube_pod_info;(.+)"
      target_label: pod
    - source_labels: [__name__, exported_namespace]
      regex: "kube_pod_info;(.+)"
      target_label: namespace
    - source_labels: [__name__, exported_node]
      regex: "kube_pod_info;(.+)"
      target_label: node
  relabel_configs:
    # Insert the same what you have so far
</code></pre>

<p><strong>Background:</strong></p>

<p><a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config" rel="noreferrer">Normal relabelling</a> (<code>relabel_configs</code>) is applied at service discovery time to the <em>target labels</em> that are automatically discovered by the service discovery process. It  defines the definitive target labels. At scrape time, target labels are added to the metric labels of all the metrics from the target. Normal relabelling can be only used to work on labels of a target after service discovery, which are generally meta labels starting with <code>__</code>.</p>

<p><a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#metric_relabel_configs" rel="noreferrer">Metric relabelling</a> (<code>metric_relabel_configs</code>) is applied to the <em>metric labels</em> at scrape time. So, this can be used to rename labels that are defined by the applications exposing the metrics themselves. </p>
