<p>I have a Prometheus federation with 2 prometheus' servers - one per Kubernetes cluster and a central to rule them all.</p>

<p>Over time the scrape durations increase. At some point, the scrape duration exceeds the timeout duration and then metrics get lost and alerts fire.</p>

<p>I’m trying to reduce the scrape duration by dropping metrics but this is an uphill battle and more like sisyphus then Prometheus. </p>

<p>Does anyone know a way to reduce the scrape time without losing metrics and without having to drop more and more as times progresses?</p>

<p>Thanks in advance!</p>

BarakH
<p>Per Prometheus' <a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file" rel="nofollow noreferrer">documentation</a>, these settings determine the <em>global</em> timeout and alerting rules evaluation frequency:</p>

<pre><code>global:
  # How frequently to scrape targets by default.
  [ scrape_interval: &lt;duration&gt; | default = 1m ]

  # How long until a scrape request times out.
  [ scrape_timeout: &lt;duration&gt; | default = 10s ]

  # How frequently to evaluate rules.
  [ evaluation_interval: &lt;duration&gt; | default = 1m ]
</code></pre>

<p>...and for each scrape job the <a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E" rel="nofollow noreferrer">configuration</a> allows setting <em>job-specific</em> values:</p>

<pre><code># The job name assigned to scraped metrics by default.
job_name: &lt;job_name&gt;

# How frequently to scrape targets from this job.
[ scrape_interval: &lt;duration&gt; | default = &lt;global_config.scrape_interval&gt; ]

# Per-scrape timeout when scraping this job.
[ scrape_timeout: &lt;duration&gt; | default = &lt;global_config.scrape_timeout&gt; ]
</code></pre>

<p>Not knowing more about the number of targets and number of metrics per target...I can suggest to try to configure appropriate <code>scrape_timeout</code> per job and adjust the global <code>evaluation_interval</code> accordingly.</p>

<p>Another option, in combination with the suggestion above or on its own, can be to have prometheus instances dedicated on scraping non-overlapping set of targets. Thus, making it possible to scale prometheus and to have different <code>evaluation_interval</code> per set of targets. For example, longer <code>scrape_timeout</code> and less frequent <code>evaluation_interval</code> (higher value) for jobs that take longer so that they don't affect other jobs.</p>

<p>Also, check if an exporter isn't misbehaving by accumulating metrics over time instead of just providing current readings at the time of scraping - otherwise, the list of what's returned to prometheus will keep on growing over time.</p>
