<p>we are running spark on k8s cluster with help of <a href="https://github.com/GoogleCloudPlatform/spark-on-k8s-operator" rel="nofollow noreferrer">spark-operator</a>. for monitoring we are using <a href="https://prometheus.io/" rel="nofollow noreferrer">prometheus</a>.</p>
<p>we want to configure an alert so that whenever any pod related to spark jobs transition to <strong>Failed</strong> state we should get an alert. and this alert rule should check for such failed pods over last 5 minutes duration.</p>
<p>we tried to leverage the <a href="https://github.com/kubernetes/kube-state-metrics" rel="nofollow noreferrer">kube-state-metrics</a> for this but we are not able to get metrics on time based. at any given point of time metric <strong>kube_pod_status_phase{namespace=&quot;spark-operator&quot;,phase=&quot;Failed&quot;}</strong> gives us the list of all the pods which are in failed state.</p>
<p>any suggestion or guidance on this are most welcome.</p>

chitender kumar
<pre><code>sum_over_time (kube_pod_status_phase{namespace=&quot;spark-operator&quot;,phase=&quot;Failed&quot;}[5m:1m]) &gt; 0
</code></pre>
