<p>I'm using Prometheus to scrape metrics from my pods. The application I'm interested in is replicated a couple of times with one service providing access. Prometheus uses this service to scrape the metrics. In my app the metrics are setup as follows:</p>

<pre><code>import * as Prometheus from 'prom-client';

const httpRequestDurationMicroseconds = new Prometheus.Histogram({
    name: 'transaction_amounts',
    help: 'Amount',
    labelNames: ['amount'],
    buckets: [0, 5, 15, 50, 100, 200, 300, 400, 500, 10000],
});

const totalPayments = new Prometheus.Counter('transaction_totals', 'Total payments');
</code></pre>

<p>I'm using helm to install Prometheus and the scrape config looks like this:</p>

<pre><code>prometheus.yml:
  rule_files:
    - /etc/config/rules
    - /etc/config/alerts

  scrape_configs:
    - job_name: prometheus
      static_configs:
        - targets:
          - localhost:9090
    - job_name: transactions
      scrape_interval: 1s
      static_configs:
        - targets:
          - transaction-metrics-service:3001
</code></pre>

<p>I can see the metrics inside prometheus, but it seems to be from just one pod. For example, in Prometheus, when I query for <code>transaction_totals</code> it gives:</p>

<p><a href="https://i.stack.imgur.com/Z7wOS.png" rel="noreferrer"><img src="https://i.stack.imgur.com/Z7wOS.png" alt="enter image description here"></a></p>

<p>I don't think that the <code>instance</code> label can uniquely identify my pods. What should I do to be able to query all pods?</p>

<p>Instead of using a <code>static_config</code> that scrapes just one host, try using <code>kubernetes_sd_configs</code> Kubernetes Service Discovery as provided by Prometheus.
Your config file would look something like this:</p>

<pre><code>- job_name: 'kubernetes-pods'

  kubernetes_sd_configs:
  - role: pod

  relabel_configs:
  # only scrape when annotation prometheus.io/scrape: 'true' is set
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: (.+):(?:\d+);(\d+)
    replacement: ${1}:${2}
    target_label: __address__
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod_name
</code></pre>

<p>and then add the annotation to your Kubernetes Deployment yaml config like this:</p>

<pre><code>kind: Deployment

...

spec:
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "&lt;&lt; PORT OF YOUR CONTAINER &gt;&gt;"

</code></pre>

<p>You can see a <a href="https://stuff.21zoo.com/posts/prometheus-kubernetes-configure-scrape-pods/" rel="noreferrer">full working example here</a>.</p>
