title: Alert always "pending" never "firing" after only (!?) upgrading container versions
body: <blockquote>
  <p><strong>UPDATES</strong></p>
  <ol>
  <li>Using a different exporter (<a href="https://github.com/DazWilkin/fly-exporter" rel="nofollow noreferrer"><code>fly-exporter</code></a>), alerts fire. If I run the Prometheus|Alertmanager and the <code>gcp-exporter</code> on a different host with the ~same config, alerts fire correctly. My hypothesis is that <code>gcp-exporter</code> scrapes (!?) are failing (periodically) and the absence of data (I'm unable to confirm this) is what resets the alerts.</li>
  <li>Discovered <code>ALERTS</code> synthetic time-series. Prometheus' graph of this, shows gaps. Querying the API <code>min_over_time(ALERTS{alertname=&quot;gcp_cloud_run_services_running&quot;}[12h])</code> is 1.</li>
  <li>&quot;Active Since&quot; (<code>activeAt</code>) is resetting periodically for no reason that's obvious to me. The time-series behind the alerts don't appear to include any 0 values.</li>
  </ol>
  </blockquote>
  <p>I wrote an <a href="https://github.com/DazWilkin/gcp-exporter" rel="nofollow noreferrer">exporter</a> for <a href="https://cloud.google.com/" rel="nofollow noreferrer">Google Cloud</a> services that I use to alert when I'm consuming (more) resources than expected.</p>
  <p>It's been working for several months and alerting me (email|<a href="https://pushover.net/" rel="nofollow noreferrer">Pushover</a>).</p>
  <p>I think (!?) the only changes I've made recently have been to bump:</p>
  <ol>
  <li>container image versions</li>
  </ol>
  <ul>
  <li><code>docker.io/prom/prometheus:v2.37.0</code></li>
  <li><code>docker.io/prom/alertmanager:v0.24.0</code></li>
  </ul>
  <ol>
  <li><a href="https://pkg.go.dev/google.golang.org/api" rel="nofollow noreferrer">google.golang.org/api</a></li>
  </ol>
  <p>However, I'm no longer (~weeks?) being alerted when I expect to.</p>
  <p>This is because the alerts move to &quot;pending&quot; but never &quot;firing&quot;.</p>
  <p>I don't understand why this is.</p>
  <p>The issue is with the Prometheus alert not &quot;firing&quot; and not with AlertManager.</p>
  <p>I've definitively been running resources on 2 different monitored services (Cloud Functions; Cloud Run) for longer than 1 day:</p>
  <pre class="lang-bash prettyprint-override"><code>date --rfc-3339=seconds --utc
  2022-07-22 22:03:37+00:00
  
  gcloud functions list \
  --project=${PROJECT} \
  --format=&quot;value(updateTime)&quot;
  2020-12-03T20:23:30.940Z # 03-December-2020
  
  gcloud run services list --project=${PROJECT} \
  --format=&quot;value(status.conditions['lastTransitionTime'].slice(0))&quot;
  2022-07-21T16:21:24.293561Z # 21-July-2022
  2022-07-21T16:20:48.439194Z # 21-July-2022
  </code></pre>
  <p>And <code>/api/v1/alerts</code>:</p>
  <pre><code>{
    &quot;status&quot;: &quot;success&quot;,
    &quot;data&quot;: {
      &quot;alerts&quot;: [
        {
          &quot;labels&quot;: {
            &quot;alertname&quot;: &quot;gcp_cloud_functions_running&quot;,
            &quot;severity&quot;: &quot;page&quot;
          },
          &quot;annotations&quot;: {
            &quot;summary&quot;: &quot;GCP Cloud Functions running&quot;
          },
          &quot;state&quot;: &quot;pending&quot;,
          &quot;activeAt&quot;: &quot;2022-07-22T21:56:02.984175081Z&quot;,
          &quot;value&quot;: &quot;1e+00&quot;
        },
        {
          &quot;labels&quot;: {
            &quot;alertname&quot;: &quot;gcp_cloud_run_services_running&quot;,
            &quot;severity&quot;: &quot;page&quot;
          },
          &quot;annotations&quot;: {
            &quot;summary&quot;: &quot;GCP Cloud Run services running&quot;
          },
          &quot;state&quot;: &quot;pending&quot;,
          &quot;activeAt&quot;: &quot;2022-07-22T21:56:02.984175081Z&quot;,
          &quot;value&quot;: &quot;2e+00&quot;
        }
      ]
    }
  }
  </code></pre>
  <blockquote>
  <p><strong>NOTE</strong> As I'm typing this question, I've noticed that the <code>activeAt</code> values are <code>21:56:02</code> which is within the 6 hour window. But, why only since then? The resources have existed for longer than that and Prometheus has been been running since (2022-07-21 16:14) when I restarted it thinking it had become wedged.</p>
  </blockquote>
  <blockquote>
  <p><strong>UPDATE</strong> The alerts <a href="https://github.com/prometheus/prometheus/blob/414c3e549cf04a63e59af449d3ed13ac4d6d844a/rules/alerting.go#L86" rel="nofollow noreferrer"><code>activeAt</code></a> property appears to (be) reset every 30 minutes:</p>
  <pre><code>2022-07-23T01:26:02.984175081Z
  2022-07-23T01:26:02.984175081Z
  
  2022-07-23T00:56:02.984175081Z
  2022-07-23T00:56:02.984175081Z
  </code></pre>
  <p>Querying the 2 time-series, I don't find any 0 values that could reset the (<code>6h</code>) timer and it's curious that both alerts have the same <code>activeAt</code> values (both times)?</p>
  </blockquote>
  <p>And, if I query the time-series:</p>
  <pre class="lang-bash prettyprint-override"><code>QUERY=&quot;...&quot; # See below
  
  # Currently 22:03 so this range covers 22 hours' data
  START=&quot;2022-07-01T00:00:00.000Z&quot;
  END=&quot;2022-07-22T23:59:59.999Z
  STEP=&quot;5m&quot;
  
  # Results show by QUERY below
  curl \
  --silent \
  --data-urlencode &quot;query=${QUERY}&quot; \
  --data &quot;start=${START}&quot; \
  --data &quot;end=${END}&quot; \
  --data &quot;step=${STEP}&quot; \
  &quot;http://${HOST}:${PORT}/api/v1/query_range&quot; \
  | jq -r '.data.result[].values[][1]' \
  | sort \
  | uniq -c
  
  # Cloud Functions
  QUERY=&quot;min_over_time(gcp_cloud_functions_functions[15m]&gt;0&quot;
      348 1
       60 2
  
  # Cloud Run
  QUERY=&quot;min_over_time(gcp_cloud_run_services[15m])&gt;0&quot;
       60 10
      609 2
  </code></pre>
  <p>My interpretation of the above is that, for both alerts, the data (admittedly stepped in 5-minute increments) never includes zeros and thus is always <code>&gt;0</code> for both queries. Yes, the alerts don't fire.</p>
  <p>Questions</p>
  <ol>
  <li>Why is this?</li>
  <li>What is my misunderstanding?</li>
  <li>Is there a better way for me to debug this?</li>
  </ol>
  <hr/>
  <p><code>prometheus.yml</code>:</p>
  <pre><code>global:
    scrape_interval: 1m
    scrape_timeout: 10s
    evaluation_interval: 1m
  alerting:
    alertmanagers:
    - follow_redirects: true
      enable_http2: true
      scheme: http
      timeout: 10s
      api_version: v2
      static_configs:
      - targets:
        - localhost:9093
  rule_files:
  - /etc/alertmanager/rules.yml
  scrape_configs:
  - job_name: gcp-exporter
    honor_timestamps: true
    scrape_interval: 15m
    scrape_timeout: 30s
    metrics_path: /metrics
    scheme: http
    follow_redirects: true
    enable_http2: true
    static_configs:
    - targets:
      - localhost:9402
  </code></pre>
  <p><code>rules.yml</code>:</p>
  <pre><code>groups:
  - name: gcp_exporter
    rules:
    - alert: gcp_cloud_functions_running
      expr: min_over_time(gcp_cloud_functions_functions{}[15m]) &gt; 0
      for: 6h
      labels:
        severity: page
      annotations:
        summary: GCP Cloud Functions running
    - alert: gcp_cloud_run_services_running
      expr: min_over_time(gcp_cloud_run_services{}[15m]) &gt; 0
      for: 6h
      labels:
        severity: page
      annotations:
        summary: GCP Cloud Run services running
  </code></pre>
  <h5>Runtime Information</h5>
  <pre><code>{
    &quot;status&quot;: &quot;success&quot;,
    &quot;data&quot;: {
      &quot;startTime&quot;: &quot;2022-07-21T16:14:23.941571056Z&quot;,
      &quot;CWD&quot;: &quot;/prometheus&quot;,
      &quot;reloadConfigSuccess&quot;: true,
      &quot;lastConfigTime&quot;: &quot;2022-07-21T16:14:23Z&quot;,
      &quot;corruptionCount&quot;: 0,
      &quot;goroutineCount&quot;: 43,
      &quot;GOMAXPROCS&quot;: 4,
      &quot;GOGC&quot;: &quot;&quot;,
      &quot;GODEBUG&quot;: &quot;&quot;,
      &quot;storageRetention&quot;: &quot;15d&quot;
    }
  }
  </code></pre>
  <h5>Build Information</h5>
  <pre><code>{
    &quot;status&quot;: &quot;success&quot;,
    &quot;data&quot;: {
      &quot;version&quot;: &quot;2.37.0&quot;,
      &quot;revision&quot;: &quot;b41e0750abf5cc18d8233161560731de05199330&quot;,
      &quot;branch&quot;: &quot;HEAD&quot;,
      &quot;buildUser&quot;: &quot;root@0ebb6827e27f&quot;,
      &quot;buildDate&quot;: &quot;20220714-15:19:21&quot;,
      &quot;goVersion&quot;: &quot;go1.18.4&quot;
    }
  }
  </code></pre>
  <h5>Alertmanagers</h5>
  <pre><code>{
    &quot;status&quot;: &quot;success&quot;,
    &quot;data&quot;: {
      &quot;activeAlertmanagers&quot;: [
        {
          &quot;url&quot;: &quot;http://localhost:9093/api/v2/alerts&quot;
        }
      ],
      &quot;droppedAlertmanagers&quot;: []
    }
  }
  </code></pre>