<p>I am deploying a monitoring stack from the <a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack" rel="nofollow noreferrer"><code>kube-prometheus-stack</code></a> helm chart and I am trying to configure alertmanager so that it has my custom configuration for alerting in a Slack channel.</p>
<p>The configuration in the pod is loaded from <code>/etc/alertmanager/config/alertmanager.yaml</code>.
From the pod description, this file is loaded from a secret automatically generated:</p>
<pre class="lang-yaml prettyprint-override"><code>...
  volumeMounts:
   - mountPath: /etc/alertmanager/config
     name: config-volume
...
volumes:
  - name: config-volume
    secret:
      defaultMode: 420
      secretName: alertmanager-prometheus-community-kube-alertmanager-generated
</code></pre>
<p>If I inspect the secret, it contains the default configuration found in the default values in <code>alertmanager.config</code>, which I intend to overwrite.</p>
<p>If I pass the following configuration to alertmanager to a fresh installation of the chart, it does not create the alertmanager pod:</p>
<pre class="lang-yaml prettyprint-override"><code>alertmanager:
  config:
    global:
      resolve_timeout: 5m
    route:
      group_by: ['job', 'alertname', 'priority']
      group_wait: 10s
      group_interval: 1m
      routes:
      - match:
          alertname: Watchdog
        receiver: 'null'
      - receiver: 'slack-notifications'
        continue: true
    receivers:
    - name: 'slack-notifications'
      slack-configs:
      - slack_api_url: &lt;url here&gt;
        title: '{{ .Status }} ({{ .Alerts.Firing | len }}): {{ .GroupLabels.SortedPairs.Values | join &quot; &quot; }}'
        text: '&lt;!channel&gt; {{ .CommonAnnotations.summary }}'
        channel: '#mychannel'
</code></pre>
<p>First of all, if I don't pass any configuration in the <code>values.yaml</code>, the alertmanager pod is successfully created.</p>
<p>How can I properly overwrite alertmanager's configuration so it mounts the correct file with my custom configuration into <code>/etc/alertmanger/config/alertmanager.yaml</code>?</p>

<p>The alertmanager requires certain non-default arguments to overwrite the default as it appears it fails in silence. Wrong configuration leads to the pod not applying the configuration (<a href="https://github.com/prometheus-community/helm-charts/issues/1998" rel="nofollow noreferrer">https://github.com/prometheus-community/helm-charts/issues/1998</a>). What worked for me was to carefully configure the alertmanager and add a watchdog child route and the null receiver</p>
<pre><code>route:
  group_by: [ '...' ]
  group_wait: 30s
  group_interval: 10s
  repeat_interval: 10s
  receiver: 'user1'
  routes:
    - match:
        alertname: Watchdog
        receiver: 'null'
receivers:
  - name: 'null'
  - ...
 
</code></pre>
