<p>I can see from the pod description that my pod &quot;Failed&quot; due to being &quot;Evicted&quot; due to Memory Pressure. but how can i test for too many &quot;Failed &amp;&amp; Evicted&quot; pods with prometheus alert rules or some other means?</p>
<p>I have Prometheus Operator installed, I can see metrics for Failed Pods but not Failed and Evicted</p>
<p>kubectl describe pod gives:</p>
<pre><code>Name:         besteffort-evictme-001
Namespace:    skyfii
Priority:     0
Node:         ip-172-17-2-169.ap-southeast-2.compute.internal/
Start Time:   Fri, 24 Sep 2021 15:28:53 +1000
Labels:       &lt;none&gt;
Annotations:  kubernetes.io/psp: eks.privileged
Status:       Failed
Reason:       Evicted
Message:      The node was low on resource: memory. Container termination-demo-container was using 17165108Ki, which exceeds its request of 0. 
IP:           
IPs:          &lt;none&gt;
Containers:

</code></pre>
<p>the prometheus rule :</p>
<pre><code>kube_pod_status_phase{phase=&quot;Failed&quot;} &gt; 0
</code></pre>
<p>shows the failed pod</p>
<pre><code>kube_pod_status_phase{endpoint=&quot;http&quot;,instance=&quot;172.17.3.141:8080&quot;,job=&quot;kube-state-metrics&quot;,namespace=&quot;skyfii&quot;,phase=&quot;Failed&quot;,pod=&quot;besteffort-evictme-001&quot;,service=&quot;prometheus-kube-state-metrics&quot;}
</code></pre>
<p>but nothing shows up for</p>
<pre><code>kube_pod_container_status_terminated_reason{reason=&quot;Evicted&quot;} &gt; 0
</code></pre>
<p>Any Ideas?</p>
<p>Thanks
Karl</p>

<p>So it seems I need to update my version of <code>kube-prometheus-stack</code> helm chart.</p>
<p>The &quot;Evicted&quot; <code>Reason</code> we see in pod description is hanging off podStatus
<a href="https://i.stack.imgur.com/Nolv6.png" rel="noreferrer"><img src="https://i.stack.imgur.com/Nolv6.png" alt="enter image description here" /></a></p>
<p>newer <code>kube-prometheus-stack</code> versions which brings in the later version (v.2) of kube-state-metrics (v.2) which in turn exposes the <a href="https://github.com/kubernetes/kube-state-metrics/blob/master/docs/pod-metrics.md" rel="noreferrer">kube_pod_status_reason</a></p>
<p>I'll upgrade then refactor my prometheus query to use this new metric and post back the answer when I get it working.</p>
<p>cheers
Karl</p>
<p>upgrading to kube-prometheus-stack v 18.1.0 allowed me to do this:-
<a href="https://i.stack.imgur.com/LyF8Y.png" rel="noreferrer"><img src="https://i.stack.imgur.com/LyF8Y.png" alt="enter image description here" /></a></p>
<p>so I'll be able to craft the query I need now</p>
<p>I added this to my prometheus alertmanager rules by adding it to the
<code>prometheusAdditionalRulesMap</code> section of the kube-prometheus-stack's Values.yaml</p>
<pre><code>
      - name: kubernetes-container-evictions

        rules:

        # Mem pressure evicted pods are left in a Failed state, alert if we see too many failed pods

        # NB you will need to delete the failed pods after investigating

        - alert: FailedEvictedPods

          expr: sum by(namespace, pod) (kube_pod_status_phase{phase=&quot;Failed&quot;} &gt; 0 and on(namespace, pod) kube_pod_status_reason{reason=&quot;Evicted&quot;} &gt; 0) &gt; 0

          for: 10m

          labels:

            severity: warning

          annotations:

            message: 'Failed Evicted pod:{{ $labels.pod }} namespace:{{ $labels.namespace }}'


        - alert: TooManyEvictedPods

          expr: sum(kube_pod_status_reason{reason=&quot;Evicted&quot;}) &gt;= 2

          labels:

            severity: high

          annotations:

            message: 'Too many Failed Evicted Pods: {{ $value }}'
</code></pre>
<p>and now I get the alerts I wanted :-)</p>
