<p>I have installed <a href="https://github.com/helm/charts/tree/master/stable/prometheus-operator" rel="nofollow noreferrer">prometheus-operator</a> via helm and now want to set email notifications for Alertmanager:</p>
<p>I edited Prometheus operator secret:</p>
<pre><code>    kubectl get secret alertmanager-prometheus-prometheus-oper-alertmanager -n monitoring -o yaml
    apiVersion: v1
    data:
      alertmanager.yaml: Z2xvYmFsOgogIHJlc29sdmVfdGltZW91dDogNW0KcmVjZWl2ZXJzOgotIG5hbWU6ICJudWxsIgpyb3V0ZToKICBncm91cF9ieToKICAtIGpvYgogIGdyb3VwX2ludGVydmFsOiA1bQogIGdyb3VwX3dhaXQ6IDMwcwogIHJlY2VpdmVyOiAibnVsbCIKICByZXBlYXRfaW50ZXJ2YWw6IDEyaAogIHJvdXRlczoKICAtIG1hdGNoOgogICAgICBhbGVydG5hbWU6IFdhdGNoZG9nCiAgICByZWNlaXZlcjogIm51bGwi
    kind: Secret
    metadata:
      creationTimestamp: &quot;2020-03-23T09:19:57Z&quot;
      labels:
        app: prometheus-operator-alertmanager
        chart: prometheus-operator-8.12.2
        heritage: Helm
        release: prometheus
      name: alertmanager-prometheus-prometheus-oper-alertmanager
      namespace: monitoring
      resourceVersion: &quot;1853097&quot;
      selfLink: /api/v1/namespaces/monitoring/secrets/alertmanager-prometheus-prometheus-oper-alertmanager
      uid: eb7f514e-6bf4-4791-9c1a-e45590ba2a36
    type: Opaque



 echo 'Z2xvYmFsOgogIHJlc29sdmVfdGltZW91dDogNW0KcmVjZWl2ZXJzOgotIG5hbWU6ICJudWxsIgpyb3V0ZToKICBncm91cF9ieToKICAtIGpvYgogIGdyb3VwX2ludGVydmFsOiA1bQogIGdyb3VwX3dhaXQ6IDMwcwogIHJlY2VpdmVyOiAibnVsbCIKICByZXBlYXRfaW50ZXJ2YWw6IDEyaAogIHJvdXRlczoKICAtIG1hdGNoOgogICAgICBhbGVydG5hbWU6IFdhdGNoZG9nCiAgICByZWNlaXZlcjogIm51bGwi' | base64 --decode
</code></pre>
<p>Created a new alertmanager.yaml file:</p>
<pre><code>global:
  resolve_timeout: 5m
route:
  group_by: [Alertname]
  # Send all notifications to me.
  receiver: email-alert

  group_by: ['job', 'alertname', 'service', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  receiver: email-alert
  routes:
  - match:
      severity: critical
    receiver: email-alert

receivers:
- name: email-alert
  email_configs:
  - to: email@example.com
    from: email@example.com
    # Your smtp server address
    smarthost: smtp.office365.com:587
    auth_username: email@example.com
    auth_identity: email@example.com
    auth_password: Pass
</code></pre>
<p>Created template:</p>
<pre><code>apiVersion: v1
data:
  alertmanager.yaml: ALERTMANAGER_CONFIG
kind: Secret
metadata:
  name: alertmanager-main
  namespace: monitoring
type: Opaque
</code></pre>
<p>Encoded it and applied:</p>
<pre><code>sed &quot;s/ALERTMANAGER_CONFIG/$(cat alertmanager.yaml | base64 -w0)/g&quot; alertmanager-secret-k8s.yaml | kubectl apply -f -
</code></pre>
<p>I exposed Alertmanager as Nodeport via port 30700.</p>
<p>So when I access it (http://IP:30700/#/status), I see this alertmanager.yaml is not applied, i.e., secret is not changed.</p>
<p>What needs to be done so I can edit this Prometheus Alertmanager secret?</p>
<p>tried with</p>
<pre><code>helm upgrade prometheus stable/prometheus-operator --namespace monitoring -f alertmanager.yaml 
</code></pre>
<p>without help</p>
<p><a href="https://i.stack.imgur.com/HanTR.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/HanTR.png" alt="enter image description here" /></a></p>

<p>Figured it out thanks to <a href="https://stackoverflow.com/questions/53558057/how-to-trigger-alert-in-kubernetes-using-prometheus-alert-manager">this</a> reference</p>
<ul>
<li><p>delete current secret:</p>
<pre><code>kubectl delete secret alertmanager-prometheus-prometheus-oper-alertmanager -n monitoring
</code></pre>
</li>
<li><p>create file alertmanager.yaml:</p>
<pre><code>  global:
    resolve_timeout: 5m
  route:
    receiver: 'email-alert'
    group_by: ['job']


    routes:
    - receiver: 'email-alert'
      match:
        alertname: etcdInsufficientMembers
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 12h  

  receivers:
  - name: email-alert
    email_configs:
    - to: receiver@example.com
      from: sender@example.com
      # Your smtp server address
      smarthost: smtp.office365.com:587
      auth_username: sender@example.com
      auth_identity: sender@example.com
      auth_password: pass
</code></pre>
</li>
<li><p>create new secret with same name as old one:</p>
<pre><code>kubectl create secret generic alertmanager-prometheus-prometheus-oper-alertmanager -n monitoring --from-file=alertmanager.yaml
</code></pre>
</li>
</ul>
