<p>I have job failure alerts in prometheus, which resolves itself right after 2 hours I got the alert where the alert actually is not resolved. How come Prometheus resolves it? Just so you know, this is only happening with this job alert.</p>
<p>Job Alert:</p>
<pre><code>  - alert: Failed Job Status
    expr: increase(kube_job_status_failed[30m]) &gt; 0
    for: 1m
    labels:
      severity: warning
    annotations:
      identifier: '{{ $labels.namespace }} {{ $labels.job_name }}'
      description: '{{ $labels.namespace }} - {{ $labels.job_name }} Failed'
</code></pre>
<p>An example of the alert:</p>
<pre><code>At 3:01 pm
[FIRING:1] Failed Job Status @ &lt;environment-name&gt; &lt;job-name&gt;
&lt;environment-name&gt; - &lt;job-name&gt; Failed

At 5:01 pm
[RESOLVED]
Alerts Resolved:
- &lt;environment-name&gt; - &lt;job-name&gt;: &lt;environment-name&gt; - &lt;job-name&gt; Failed
</code></pre>
<p>Here's the related pods as it can be seen that nothing seems to be resolved.</p>
<p><a href="https://i.stack.imgur.com/P2Gaz.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/P2Gaz.png" alt="Scription here" /></a></p>
<p>Thanks for your help in advance!</p>

<p><code>kube_job_status_failed</code> is a gauge representing the number of failed job pods at a given time. The expression <code>increase(kube_job_status_failed[30m]) &gt; 0</code> asks the question: &quot;have there been new failures in the last 30 minutes?&quot; If there haven't, it won't be true, even if old failures remain in the Kubernetes API.</p>
<p>A refinement of this approach is <code>sum(rate(kube_job_status_failed[5m])) by (namespace, job_name) &gt; 0</code>, plus an alert manager configuration to <em>not send resolved notices</em> for this alert. This is because a job pod failure is an event that can't be reversed - the job could be retried, but the pod can't be un-failed so resolution only means the alert has &quot;aged out&quot; or the pods have been deleted.</p>
<p>An expression that looks at the current number of failures recorded in the API server is <code>sum(kube_job_status_failed) by (namespace, job_name) &gt; 0</code>. An alert based on this could be &quot;resolved&quot;, but only by the <code>Job</code> objects being removed from the API (which doesn't necessarily mean that a process has succeeded...)</p>
