title: How to get a pod's labels in Prometheus when pulling the metrics from Kube State Metrics
body: <p>I have a Prometheus pod running along with my Kube-State-Metrics (KSM) pod. The KSM collects all the metrics from all the pods across all the namespaces in the cluster. Prometheus simply scrapes the metrics from KSM - this way Prometheus doesn't need to scrape the individual pods.</p>
  
  <p>When pods are deployed, their deployment has certain pod-related labels as shown below. They have two important labels: <strong>APP</strong> and <strong>TEAM</strong>:</p>
  
  <pre><code>apiVersion: apps/v1
  kind: Deployment
  metadata:
    labels:
      APP: AppABC
      TEAM: TeamABC
  ...
  </code></pre>
  
  <p>&nbsp;</p>
  
  <p>Within Prometheus, my scrape configuration looks like this:</p>
  
  <pre><code>scrape_configs:
    - job_name: 'pod monitoring'
      honor_labels: true
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
  ...
  </code></pre>
  
  <p>&nbsp;</p>
  
  <p>Problem is, when Prometheus scrapes the information from kube-state-metrics, it overwrites the <code>APP</code> with <code>kube-state-metrics</code>. e.g. this metric below is actually for an app called <strong>"AppABC"</strong>, yet Prometheus overwrote the <code>app</code> label to <code>kube-state-metrics</code>.</p>
  
  <pre><code>kube_pod_container_status_restarts_total{
      app="kube-state-metrics",
      container="appabccontainer",
      job="pod monitoring",
      namespace="test-namespace",
      pod="appabc-766cbcb68d-29smr"
  }
  </code></pre>
  
  <p>Is there anyway for me to scrape the metrics from kube-state-metrics BUT keep the <strong>APP</strong> and <strong>TEAM</strong> labels together without overwriting them?</p>
  
  <p>&nbsp;</p>
  
  <p><strong>EDIT - I figured it out</strong></p>
  
  <p><strong>My Issue:</strong> My deployment and pods have certain labels defined (APP, TEAM). Kube-state-metrics gets these from K8 API. When Prometheus scrapes from kube-state-metrics, it doesn't have those labels.</p>
  
  <p><strong>My Objective:</strong> Expose those labels into Prometheus.</p>
  
  <p><strong>My Solution:</strong> Using PromQL you can do group by. So in my prometheus-rules.yaml, I changed this:</p>
  
  <pre><code>expr: kube_pod_status_phase{phase="Failed"} &gt; 0
  </code></pre>
  
  <p>to this:</p>
  
  <pre><code>expr: kube_pod_status_phase{phase="Failed"} * on (pod,namespace) group_right kube_pod_labels &gt; 0
  </code></pre>
  
  <p>So my new alert rule looks like this:</p>
  
  <pre><code>- name: Pod_Failed
    rules:
    - alert: pod_failed
      expr: kube_pod_status_phase{phase="Failed"} * on (pod,namespace) group_right kube_pod_labels &gt; 0
      labels:
        appname: '{{ $labels.label_APP }}' # This is what I wanted to capture
        teamname: '{{ $labels.label_TEAM }}' # This is what I wanted to capture
      annotations:
        summary: 'Pod: {{ $labels.pod }} is down'
        description: 'Pod: {{ $labels.pod }} is down in {{ $labels.namespace }} namespace.'
  </code></pre>