<p>What would be the best way to set up a <a href="https://cloud.google.com/monitoring/docs" rel="nofollow noreferrer">GCP monitoring alert policy</a> for a <a href="https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/" rel="nofollow noreferrer">Kubernetes CronJob</a> failing? I haven't been able to find any good examples out there.</p>
<p>Right now, I have an OK solution based on monitoring logs in the Pod with <code>ERROR</code> severity. I've found this to be quite flaky, however. Sometimes a job will fail for some ephemeral reason outside my control (e.g., an external server returning a temporary 500) and on the next retry, the job runs successfully.</p>
<p>What I really need is an alert that is only triggered when a CronJob is in a persistent failed state. That is, Kubernetes has tried rerunning the whole thing, multiple times, and it's still failing. Ideally, it could also handle situations where the Pod wasn't able to come up either (e.g., downloading the image failed).</p>
<p>Any ideas here?</p>
<p>Thanks.</p>

joeltine
<p>First of all, confirm the <strong>GKE’s</strong> version that you are running. For that, the following commands are going to help you to identify the <a href="https://cloud.google.com/kubernetes-engine/versioning#use_to_check_versions" rel="nofollow noreferrer"><strong>GKE’s</strong>
default version</a> and the available versions too:</p>
<p><strong>Default version.</strong></p>
<pre><code>gcloud container get-server-config --flatten=&quot;channels&quot; --filter=&quot;channels.channel=RAPID&quot; \
    --format=&quot;yaml(channels.channel,channels.defaultVersion)&quot;
</code></pre>
<p><strong>Available versions.</strong></p>
<pre><code>gcloud container get-server-config --flatten=&quot;channels&quot; --filter=&quot;channels.channel=RAPID&quot; \
    --format=&quot;yaml(channels.channel,channels.validVersions)&quot;
</code></pre>
<p>Now that you know your <strong>GKE’s</strong> version and based on what you want is an alert that is only triggered when a <strong>CronJob</strong> is in a persistent failed state, <a href="https://cloud.google.com/stackdriver/docs/solutions/gke/managing-metrics#workload-metrics" rel="nofollow noreferrer">GKE Workload Metrics</a> was the <strong>GCP’s</strong> solution that used to provide a fully managed and highly configurable solution for sending to <strong>Cloud Monitoring all Prometheus-compatible metrics</strong> emitted by <strong>GKE workloads</strong> (such as a <strong>CronJob</strong> or a Deployment for an application). But, as it is right now deprecated in <strong>G​K​E 1.24</strong> and was replaced with <a href="https://cloud.google.com/stackdriver/docs/managed-prometheus" rel="nofollow noreferrer">Google Cloud Managed Service for Prometheus</a>, then this last is the best option you’ve got inside of <strong>GCP</strong>, as it lets you monitor and alert on your workloads, using <strong>Prometheus</strong>, without having to manually manage and operate <strong>Prometheus</strong> at scale.</p>
<p>Plus, you have 2 options from the outside of GCP: <a href="https://prometheus.io/" rel="nofollow noreferrer">Prometheus</a> as well and <a href="https://rancher.com/docs/rancher/v2.5/en/best-practices/rancher-managed/monitoring/#prometheus-push-gateway" rel="nofollow noreferrer">Ranch’s Prometheus Push Gateway</a>.</p>
<p>Finally and just FYI, it can be done manually by querying for the job and then checking it's start time, and compare that to the current time, this way, with bash:</p>
<pre><code>START_TIME=$(kubectl -n=your-namespace get job your-job-name -o json | jq '.status.startTime')
echo $START_TIME
</code></pre>
<p>Or, you are able to get the job’s current status as a JSON blob, as follows:</p>
<pre><code>kubectl -n=your-namespace get job your-job-name -o json | jq '.status'
</code></pre>
<p>You can see the following <a href="https://stackoverflow.com/questions/57959635/monitor-cronjob-running-on-gke">thread</a> for more reference too.</p>
<p>Taking the <strong>“Failed”</strong> state as the medullary point of your requirement, setting up a bash script with <code>kubectl</code> to send an email if you see a job that is in <strong>“Failed”</strong> state can be useful. Here I will share some examples with you:</p>
<pre><code>while true; do if `kubectl get jobs myjob -o jsonpath='{.status.conditions[?(@.type==&quot;Failed&quot;)].status}' | grep True`; then mail email@address -s jobfailed; else sleep 1 ; fi; done
</code></pre>
<p><strong>For newer K8s:</strong></p>
<pre><code>while true; do kubectl wait --for=condition=failed job/myjob; mail@address -s jobfailed; done
</code></pre>
