<p>given the following <code>promql</code> with alertmanager integration:</p>
<p>what's a good way to alter this expression so that when a pod restart is indicated by this promql and an alert triggered (expected) that an alert isn't automatically resolved on the next check since on the next check it's likely a restart will no longer register (unless a pod is flapping or something). We want to be alerted so we can check it out so we want to manually resolve the alert:</p>
<p><code>sum(rate(kube_pod_container_status_restarts_total[2m])*100) by (namespace,container) &gt; 0 </code></p>
<p>A potential alertmanager rule would be the following, but we don't yet have an actual rule.</p>
<pre><code>    - alert: Unexpected POD restart(s) occurring
      expr: sum(rate(kube_pod_container_status_restarts_total[2m])*100) by (namespace,container) &gt; 0 
      for: 10s
      annotations:
        summary: POD restarts occurring
</code></pre>

Jim
<p>An alert is raised when your query produces at least one metric and the alert is resolved when it no longer produces a metric.</p>
<pre><code>sum by (...) (rate(kube_pod_container_status_restarts_total[2m])) &gt; 0
</code></pre>
<p>means that it calculates the rate over the last two minutes. So if a restart occurs, the this expression will result in <code>0</code> after 2 minutes and your alert is resolved.</p>
<p>What you could do is simply increase the time interval:</p>
<pre><code>sum by (...) (rate(kube_pod_container_status_restarts_total[1h])) &gt; 0
</code></pre>
<p>so the alert stays on for an hour.</p>
<p>May be you dont want to check for every single restart, but if there are pods that restart within a specific time at least twice, so that might be the better expression:</p>
<pre><code>sum by (...) (increase(kube_pod_container_status_restarts_total[24h])) &gt; 2
</code></pre>
<p>meaning if a container restert at least two times within a day that might be a situation you want to have a look at.</p>
