<p>I have multiple kubernetes clusters metric data being scraped in prometheus. When i get metrics how can i differentiate the metrics from different clusters? I am not seeing any label that contains data regarding a specific cluster, so that i can filter out data of a particular cluster like below,</p>

<pre><code>container_cpu_usage_seconds_total{cluster-name="abcde"}
</code></pre>

<p>Is there any way where i can add a label "cluster-name" in my kubernetes_sd_configs. I have seen, labels can be added in static_config but can't find anything related to kubernetes_sd_configs.</p>

<p>I tried using relabel_config like below,</p>

<pre><code> - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: cluster-name
        replacement: my-cluster
</code></pre>

<p>This did not get reflected in metrics. When i do it with already existing label like,</p>

<pre><code> - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: domainname
        replacement: my-cluster
</code></pre>

<p>Then the domain name value is getting changed. Am i missing any configuration here?</p>

<p>You have to add that label yourself in your relabel_configs. Generally you do this on the intake cluster (i.e. the normal "cluster-level" ones) via a global relabel to add it, but it's also possible to add during the remote write if you're doing federation that way or via the jobs on the central cluster if you're doing a pull-based.</p>
