<p>I want to get an alert when the recent job of a cronjob fails. The expr
<code>kube_job_status_failed{job_name=~"cronjobname.*"}==1</code> works for most of time. But if a job fails and it's kept, even the next job succeeds, I still get an alert because there are two records in prometheus, one of which is the failure record, the other one is the success record.</p>

<p>I found I can get the latest job timestamp from <code>kube_cronjob_status_last_schedule_time{cronjob="cronjobname"}</code>, then use <code>kube_job_status_failed{job_name="cronjobname-TIMESTAMP"}</code> to query the last job status.</p>

<p>I wonder whether we have a way in one query to concatenate the <code>jobname</code> from the result of the first query and filter in the second? like
<code>kube_job_status_failed{job_name=string_concatenate("cronjobname-", kube_cronjob_status_last_schedule_time{cronjob="cronjobname"})}
</code></p>

Aaron LUO
<p>With promql, you won't be able to have something the way you describe it. Moreover, I am not sure the last schedule time is always the same as the job start time; if there is a slowness or a reschedule somewhere by example.</p>

<p>You can follow the approach indicated in <a href="https://medium.com/@tristan_96324/prometheus-k8s-cronjob-alerts-94bee7b90511" rel="nofollow noreferrer">this article</a>. An alternative one would be using the <a href="https://github.com/kubernetes/kube-state-metrics/blob/master/docs/job-metrics.md" rel="nofollow noreferrer">job metrics</a> to determine:</p>

<p>the timestamp of the last failed job per cronjob</p>

<pre><code>- record: job_cronjob:kube_job_status_start_time:last_failed
  expr: max((kube_job_status_start_time AND kube_job_status_failed == 1)
            * ON(job,namespace) GROUP_LEFT
            kube_job_labels{label_cronjob!=""}
           ) BY(label_cronjob)
</code></pre>

<p>the timestamp of the last successful job per cronjob</p>

<pre><code>- record: job_cronjob:kube_job_status_start_time:last_suceeded
  expr: max((kube_job_status_start_time AND kube_job_status_suceeded == 1)
            * ON(job,namespace) GROUP_LEFT
            kube_job_labels{label_cronjob!=""}
           ) BY(label_cronjob)
</code></pre>

<p>And alert if failed one is more recent than successful one:</p>

<pre><code>- alert: CronJobStatusFailed
  expr:   job_cronjob:kube_job_status_start_time:last_failed
        &gt; job_cronjob:kube_job_status_start_time:last_suceeded
  for: 1m
  annotations:
    description: '{{ $labels.label_cronjob}} last run has failed.'
</code></pre>
