<p>I am wondering is there a way to limit the number of targets in prometheus. Searched prometheus documentation, but couldn't find such one. We are deploying prometheus using operators on k8s cluster and targets are added using servicemonitor crd. These yaml files are written by devs and I as a infra team can not control addition done by dev people. I am looking for global option in prometheus which can limit the addition of targets done in prometheus and throw error on prometheus dashboard saying something like upper limit has been reached.</p>

<p>Prometheus version-2.7</p>

<p>You cannot limit the number of targets; this would make a very odd feature. But you can monitor the number of targets scraped by Prometheus and trigger an error whenever this number exceeds a given threshold.</p>

<pre><code>- alert: TooManyTargetsInPrometheus
  expr: count(up) &gt; 42
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Prometheus instance has too much jobs"
    description: "Prometheus has {{ $value }} targets which is way too much."
</code></pre>

<p>If you want automation, you can trigger an action on the alert to rollback the configuration or downsize the offending processes.</p>

<p>In my opinion, you would alert if your rate of missed scrape start to increase or on the number of metrics ingested by seconds exceeds recommendations, not on a arbitrary limit.</p>
