<p>I have a service running in a k8s cluster, which I want to monitor using Prometheus Operator. The service has a <code>/metrics</code> endpoint, which returns simple data like:</p>

<pre><code>myapp_first_queue_length 12
myapp_first_queue_processing 2
myapp_first_queue_pending 10
myapp_second_queue_length 4
myapp_second_queue_processing 4
myapp_second_queue_pending 0
</code></pre>

<p>The API runs in multiple pods, behind a basic <code>Service</code> object:</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: myapp-api
  labels:
    app: myapp-api
spec:
  ports:
  - port: 80
    name: myapp-api
    targetPort: 80
  selector:
    app: myapp-api
</code></pre>

<p>I've installed Prometheus using <code>kube-prometheus</code>, and added a <code>ServiceMonitor</code> object:</p>

<pre><code>apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: myapp-api
  labels:
    app: myapp-api
spec:
  selector:
    matchLabels:
      app: myapp-api
  endpoints:
  - port: myapp-api
    path: /api/metrics
    interval: 10s
</code></pre>

<p>Prometheus discovers all the pods running instances of the API, and I can query those metrics from the Prometheus graph. So far so good.</p>

<p>The issue is, those metrics are aggregate - each API instance/pod doesn't have its own queue, so there's no reason to collect those values from every instance. In fact it seems to invite confusion - if Prometheus collects the same value from 10 pods, it looks like the total value is 10x what it really is, unless you know to apply something like <code>avg</code>.</p>

<p>Is there a way to either tell Prometheus "this value is already aggregate and should always be presented as such" or better yet, tell Prometheus to just scrape the values once via the internal load balancer for that service, rather than hitting each pod?</p>

<p><strong>edit</strong></p>

<p>The actual API is just a simple <code>Deployment</code> object:</p>

<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-api
  labels:
    app: myapp-api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: myapp-api
  template:
    metadata:
      labels:
        app: myapp-api
    spec:
      imagePullSecrets:
      - name: mysecret
      containers:
      - name: myapp-api
        image: myregistry/myapp:2.0
        ports:
        - containerPort: 80
        volumeMounts:
        - name: config
          mountPath: "app/config.yaml"
          subPath: config.yaml
      volumes:
      - name: config
        configMap:
          name: myapp-api-config
</code></pre>

superstator
<p>Prometheus Operator developers are kindly working (as of Jan 2023) on a generic ScrapeConfig CRD that is designed to solve exactly the use case you describe: <a href="https://github.com/prometheus-operator/prometheus-operator/issues/2787" rel="nofollow noreferrer">https://github.com/prometheus-operator/prometheus-operator/issues/2787</a></p>
<p>In the meantime, you can use the &quot;<a href="https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/additional-scrape-config.md" rel="nofollow noreferrer">additional scrape config</a>&quot; facility of Prometheus Operator to setup a custom scrape target.</p>
<p>The idea is that the configured scrape target will be hit only once per scrape period and the service DNS will load-balance the request to one of the N pods behind the service, thus avoiding duplicate metrics.</p>
<p>In particular, you can override the <code>kube-prometheus-stack</code> Helm values as follows:</p>
<pre><code>prometheus:
  prometheusSpec:
    additionalScrapeConfigs:
      - job_name: 'myapp-api-aggregates':
        metrics_path: '/api/metrics'
        scheme: 'http'
        static_configs:
          - targets: ['myapp-api:80']
</code></pre>
