<p>I have setup kube-prometheus in my cluster(<a href="https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus" rel="noreferrer">https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus</a>). It contains some default alerts like "CoreDNSdown etc". How to create my own alert?</p>

<p>Could any one provide me sample example to create an alert that will send an email to my gmail account?</p>

<p>I followed this <a href="https://stackoverflow.com/questions/49472375/alert-when-docker-container-pod-is-in-error-or-carshloopbackoff-kubernetes">Alert when docker container pod is in Error or CarshLoopBackOff kubernetes</a>. But I couldn't make it work.</p>

<p>To send an alert to your gmail account, you need to setup the alertmanager configuration in a file say alertmanager.yaml:</p>

<pre><code>cat &lt;&lt;EOF &gt; alertmanager.yml
route:
  group_by: [Alertname]
  # Send all notifications to me.
  receiver: email-me

receivers:
- name: email-me
  email_configs:
  - to: $GMAIL_ACCOUNT
    from: $GMAIL_ACCOUNT
    smarthost: smtp.gmail.com:587
    auth_username: "$GMAIL_ACCOUNT"
    auth_identity: "$GMAIL_ACCOUNT"
    auth_password: "$GMAIL_AUTH_TOKEN"
EOF
</code></pre>

<p>Now, as you're using kube-prometheus so you will have a secret named <code>alertmanager-main</code> that is default configuration for <code>alertmanager</code>. You need to create a secret <code>alertmanager-main</code> again with the new configuration using following command:</p>

<pre><code>kubectl create secret generic alertmanager-main --from-file=alertmanager.yaml -n monitoring
</code></pre>

<p>Now you're alertmanager is set to send an email whenever it receive alert from the prometheus.</p>

<p>Now you need to setup an alert on which your mail will be sent. You can set up DeadManSwitch alert which fires in every case and it is used to check your alerting pipeline</p>

<pre><code>groups:
- name: meta
  rules:
    - alert: DeadMansSwitch
      expr: vector(1)
      labels:
        severity: critical
      annotations:
        description: This is a DeadMansSwitch meant to ensure that the entire Alerting
          pipeline is functional.
        summary: Alerting DeadMansSwitch
</code></pre>

<p>After that the <code>DeadManSwitch</code> alert will be fired and should send email to your mail.</p>

<p>Reference link:</p>

<blockquote>
  <p><a href="https://coreos.com/tectonic/docs/latest/tectonic-prometheus-operator/user-guides/configuring-prometheus-alertmanager.html" rel="noreferrer">https://coreos.com/tectonic/docs/latest/tectonic-prometheus-operator/user-guides/configuring-prometheus-alertmanager.html</a></p>
</blockquote>

<p>EDIT:</p>

<p>The deadmanswitch alert should go in a config-map which your prometheus is reading. I will share the relevant snaps from my prometheus here:</p>

<pre><code>"spec": {
        "alerting": {
            "alertmanagers": [
                {
                    "name": "alertmanager-main",
                    "namespace": "monitoring",
                    "port": "web"
                }
            ]
        },
        "baseImage": "quay.io/prometheus/prometheus",
        "replicas": 2,
        "resources": {
            "requests": {
                "memory": "400Mi"
            }
        },
        "ruleSelector": {
            "matchLabels": {
                "prometheus": "prafull",
                "role": "alert-rules"
            }
        },
</code></pre>

<p>The above config is of my prometheus.json file which have the name of alertmanager to use and the <code>ruleSelector</code> which will select the rules based on <code>prometheus</code> and <code>role</code> label. So I have my rule configmap like:</p>

<pre><code>kind: ConfigMap
apiVersion: v1
metadata:
  name: prometheus-rules
  namespace: monitoring
  labels:
    role: alert-rules
    prometheus: prafull
data:
  alert-rules.yaml: |+
   groups:
   - name: alerting_rules
     rules:
       - alert: LoadAverage15m
         expr: node_load15 &gt;= 0.50
         labels:
           severity: major
         annotations:
           summary: "Instance {{ $labels.instance }} - high load average"
           description: "{{ $labels.instance  }} (measured by {{ $labels.job }}) has high load average ({{ $value }}) over 15 minutes."
</code></pre>

<p>Replace the <code>DeadManSwitch</code> in above config map.</p>
