<p>I have my kubernetes cluster setup on AWS where I am trying to monitor several pods, using cAdvisor + Prometheus + Alert manager. What I want to do it launch an email alert (with service/container name) if a container/pod goes down or stuck in Error or CarshLoopBackOff state or stcuk in anyother state apart from running.</p>

<p>Prometheus collects <a href="https://github.com/kubernetes/kube-state-metrics/blob/master/docs/pod-metrics.md" rel="noreferrer">a wide range of metrics</a>. As an example, you can use a metric <code>kube_pod_container_status_restarts_total</code> for monitoring restarts, which will reflect your problem.</p>
<p>It containing tags which you can use in the alert:</p>
<ul>
<li>container=<code>container-name</code></li>
<li>namespace=<code>pod-namespace</code></li>
<li>pod=<code>pod-name</code></li>
</ul>
<p>So, everything you need is to configure your <code>alertmanager.yaml</code> <a href="https://github.com/prometheus/alertmanager/blob/master/doc/examples/simple.yml" rel="noreferrer">config</a> by adding correct SMTP settings, receiver and rules like that:</p>
<pre><code>global:
  # The smarthost and SMTP sender used for mail notifications.
  smtp_smarthost: 'localhost:25'
  smtp_from: 'alertmanager@example.org'
  smtp_auth_username: 'alertmanager'
  smtp_auth_password: 'password'

receivers:
- name: 'team-X-mails'
  email_configs:
  - to: 'team-X+alerts@example.org'

# Only one default receiver
route:
  receiver: team-X-mails

# Example group with one alert
groups:
- name: example-alert
  rules:
    # Alert about restarts
  - alert: RestartAlerts
    expr: count(kube_pod_container_status_restarts_total) by (pod-name) &gt; 5
    for: 10m
    annotations:
      summary: &quot;More than 5 restarts in pod {{ $labels.pod-name }}&quot;
      description: &quot;{{ $labels.container-name }} restarted (current value: {{ $value }}s) times in pod {{ $labels.pod-namespace }}/{{ $labels.pod-name }}&quot;
</code></pre>
