<p>I am running management tasks using Kubernetes CronJobs and have Prometheus alerting on when one of the spawned Jobs fails using <a href="https://github.com/kubernetes/kube-state-metrics" rel="nofollow noreferrer">kube-state-metrics</a>:</p>

<pre><code>kube_job_status_failed{job="kube-state-metrics"}  &gt; 0
</code></pre>

<p>I want to have it so that when a more recent Job passes then the failed ones are cleaned up so that the alert stops firing.</p>

<p>Does the CronJob resource support this behaviour on its own?</p>

<p>Workarounds would be to make the Job clean up failed ones as the last step or to create a much more complicated alert rule to take the most recent Job as the definitive status, but they are not the nicest solutions IMO.</p>

<p>Kubernetes version: <code>v1.15.1</code></p>

dippynark
<p>As a workaround the following query would show CronJobs where the last finished Job has failed</p>

<pre><code>(max by(owner_name, namespace) (kube_job_status_start_time * on(job_name) group_left(owner_name) ((kube_job_status_succeeded / kube_job_status_succeeded == 1) + on(job_name) group_left(owner_name) (0 * kube_job_owner{owner_is_controller="true",owner_kind="CronJob"}))))
&lt; bool
(max by(owner_name, namespace) (kube_job_status_start_time * on(job_name) group_left(owner_name) ((kube_job_status_failed / kube_job_status_failed == 1) + on(job_name) group_left(owner_name) (0 * kube_job_owner{owner_is_controller="true",owner_kind="CronJob"})))) == 1
</code></pre>
