<p>I deployed prometheus server (+ kube state metrics + node exporter + alertmanager) through the <a href="https://github.com/helm/charts/tree/master/stable/prometheus" rel="nofollow noreferrer">prometheus helm chart</a> using the chart's default values, including the chart's default <a href="https://github.com/helm/charts/blob/ead4e79279a972ec71f6a58dd04ef4491686efbc/stable/prometheus/values.yaml#L796" rel="nofollow noreferrer">scrape_configs</a>.  The problem is that I expect certain metrics to be coming from a particular job but instead are coming from a different one.</p>

<p>For example, <code>node_cpu_seconds_total</code> is being provided by the <code>kubernetes-service-endpoints</code> job but I expect it to come from the <code>kubernetes-nodes</code> job, i.e. <code>node-exporter</code>.  The returned metric's values are accurate but the problem is I don't have the labels that would normally come from <code>kubernetes-nodes</code> (since <code>kubernetes-nodes</code> job has <code>role: node</code> vs <code>role: endpoint</code> for <code>kubernetes-service-endpoints</code>.  I need these missing labels for advanced querying + dashboards.</p>

<p>Output of <code>node_cpu_seconds_total{mode="idle"}</code>:</p>

<p><code>
node_cpu_seconds_total{app="prometheus",chart="prometheus-7.0.2",component="node-exporter",cpu="0",heritage="Tiller",instance="10.80.20.46:9100",job="kubernetes-service-endpoints",kubernetes_name="get-prometheus-node-exporter",kubernetes_namespace="default",mode="idle",release="get-prometheus"} | 423673.44
node_cpu_seconds_total{app="prometheus",chart="prometheus-7.0.2",component="node-exporter",cpu="0",heritage="Tiller",instance="10.80.20.52:9100",job="kubernetes-service-endpoints",kubernetes_name="get-prometheus-node-exporter",kubernetes_namespace="default",mode="idle",release="get-prometheus"} | 417097.16
</code></p>

<p>There are no errors in the logs and I do have other <code>kubernetes-nodes</code> metrics such as <code>up</code> and <code>storage_operation_errors_total</code> so <code>node-exporter</code> is getting scraped.</p>

<p>I also verified manually that <code>node-exporter</code> has this particular metric, <code>node_cpu_seconds_total</code>, with <code>curl &lt;node IP&gt;:9100/metrics | grep node_cpu</code> and it has results.</p>

<p>Does the job order definition matter?  Would one job override the other's metrics if they have the same name?  Should I be dropping metrics for the <code>kubernetes-service-endpoints</code> job?  I'm new to prometheus so any detailed help is appreciated.</p>

ravishi
<p>I was able to figure out how to add the "missing" labels by navigating to the prometheus service-discovery status UI page.  This page shows all the "Discovered Labels" that can be processed and kept through relabel_configs.  What is processed/kept shows next to "Discovered Labels" under "Target Labels".  So then it was just a matter of modifying the <code>kubernetes-service-endpoints</code> job config in <code>scrape_configs</code> so I add more taget labels.  Below is exactly what I changed in the chart's <code>scrape_configs</code>.  With this new config, I get <code>namespace</code>, <code>service</code>, <code>pod</code>, and <code>node</code> added to all metrics if the metric didn't already have them (see <code>honor_labels</code>).</p>

<pre><code>       - job_name: 'kubernetes-service-endpoints'
+        honor_labels: true

         kubernetes_sd_configs:
           - role: endpoints

         relabel_configs:
           - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
             action: keep
             regex: true
           - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
             action: replace
             target_label: __scheme__
             regex: (https?)
           - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
             action: replace
             target_label: __metrics_path__
             regex: (.+)
           - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
             action: replace
             target_label: __address__
             regex: ([^:]+)(?::\d+)?;(\d+)
             replacement: $1:$2
           - action: labelmap
             regex: __meta_kubernetes_service_label_(.+)
           - source_labels: [__meta_kubernetes_namespace]
             action: replace
-            target_label: kubernetes_namespace
+            target_label: namespace
           - source_labels: [__meta_kubernetes_service_name]
             action: replace
-            target_label: kubernetes_name
+            target_label: service
+          - source_labels: [__meta_kubernetes_pod_name]
+            action: replace
+            target_label: pod
+          - source_labels: [__meta_kubernetes_pod_node_name]
+            action: replace
+            target_label: node
</code></pre>
