<p>I wanted to create alerts in Grafana for My Kubernetes Clusters.
I have configured Prometheus, Node exporter, Kube-Metrics, Alert Manager in my k8s Cluster.
I wanted to setup Alerting on Unschedulable or Failed Pods.</p>
<ol>
<li>Cause of unschedulable or failed pods</li>
<li>Generating an alert after a while</li>
<li>Creating another alert to notify us when pods fail.
Can You guide me how to achieve this??</li>
</ol>

sayali_bhavsar
<p>Based on the comment from <a href="https://stackoverflow.com/users/8803619/suresh-vishnoi" title="14,017 reputation">Suresh Vishnoi</a>:</p>
<blockquote>
<p>it might be helpful <a href="https://awesome-prometheus-alerts.grep.to/rules.html#kubernetes" rel="noreferrer">awesome-prometheus-alerts.grep.to/rules.html#kubernetes</a></p>
</blockquote>
<p>yes, this could be very helpful. On this site you can find templates for <a href="https://awesome-prometheus-alerts.grep.to/rules.html#rule-kubernetes-1-17" rel="noreferrer">failed pods (not healthy)</a>:</p>
<blockquote>
<p>Pod has been in a non-ready state for longer than 15 minutes.</p>
</blockquote>
<pre class="lang-yaml prettyprint-override"><code>  - alert: KubernetesPodNotHealthy
    expr: min_over_time(sum by (namespace, pod) (kube_pod_status_phase{phase=~&quot;Pending|Unknown|Failed&quot;})[15m:1m]) &gt; 0
    for: 0m
    labels:
      severity: critical
    annotations:
      summary: Kubernetes Pod not healthy (instance {{ $labels.instance }})
      description: &quot;Pod has been in a non-ready state for longer than 15 minutes.\n  V
</code></pre>
<p>or for <a href="https://awesome-prometheus-alerts.grep.to/rules.html#rule-kubernetes-1-18" rel="noreferrer">crash looping</a>:</p>
<blockquote>
<p>Pod {{ $labels.pod }} is crash looping</p>
</blockquote>
<pre class="lang-yaml prettyprint-override"><code>  - alert: KubernetesPodCrashLooping
    expr: increase(kube_pod_container_status_restarts_total[1m]) &gt; 3
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: Kubernetes pod crash looping (instance {{ $labels.instance }})
      description: &quot;Pod {{ $labels.pod }} is crash looping\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}&quot;
</code></pre>
<p>See also <a href="https://sysdig.com/blog/kubernetes-monitoring-prometheus/" rel="noreferrer">this good guide about monitoring kubernetes cluster with Prometheus</a>:</p>
<blockquote>
<p>The Kubernetes API and the  <a href="https://github.com/kubernetes/kube-state-metrics" rel="noreferrer">kube-state-metrics</a>  (which natively uses prometheus metrics)  <strong>solve part of this problem</strong>  by exposing Kubernetes internal data, such as the number of desired / running replicas in a deployment, unschedulable nodes, etc.</p>
<p>Prometheus is a good fit for microservices because you just need to  <strong>expose a metrics port</strong>, and don’t need to add too much complexity or run additional services. Often, the service itself is already presenting a HTTP interface, and the developer just needs to add an additional path like  <code>/metrics</code>.</p>
</blockquote>
<p>If it comes to unschedulable nodes, you can use the metric <code>kube_node_spec_unschedulable</code>. It is described <a href="https://github.com/kubernetes/kube-state-metrics/blob/master/docs/node-metrics.md" rel="noreferrer">here</a> or <a href="https://www.sumologic.com/blog/kubernetes-monitoring/" rel="noreferrer">here</a>:
<code>kube_node_spec_unschedulable</code> - Whether a node can schedule new pods or not.</p>
<p>Look also at <a href="https://www.replex.io/blog/kubernetes-in-production-the-ultimate-guide-to-monitoring-resource-metrics" rel="noreferrer">this guide</a>.
Basically, you need to find the metric you want to monitor and set it appropriately in Prometheus. Alternatively, you can use templates, as I showed at the beginning of the answer.</p>
