<p>I am trying to create alerts in Prometheus on Kubernetes and sending them to a Slack channel. For this i am using the <a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus" rel="nofollow noreferrer">prometheus-community</a> helm-charts (which already includes the alertmanager). As i want to use my own alerts I have also created an <em>values.yml</em> (shown below) strongly inspired from <a href="https://gist.github.com/l13t/d432b63641b6972b1f58d7c037eec88f" rel="nofollow noreferrer">here</a>.
If I port forward Prometheus I can see my Alert there going from inactive, to pending to firing, but no message is sent to slack. I am quite confident that my alertmanager configuration is fine (as I have tested it with some prebuild alerts of another chart and they were sent to slack). So my best guess is that I add the alert in the wrong way (in the serverFiles part), but I can not figure out how to do it correctly. Also, the alertmanager logs look pretty normal to me. Does anyone have an idea where my problem comes from?</p>
<pre><code>---
serverFiles:
  alerting_rules.yml: 
    groups:
    - name: example
      rules:
      - alert: HighRequestLatency
        expr: sum(rate(container_network_receive_bytes_total{namespace=&quot;kube-logging&quot;}[5m]))&gt;20000
        for: 1m
        labels:
          severity: page
        annotations:
          summary: High request latency

alertmanager:
  persistentVolume:
    storageClass: default-hdd-retain
  ## Deploy alertmanager
  ##
  enabled: true

  ## Service account for Alertmanager to use.
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/
  ##
  serviceAccount:
    create: true
    name: &quot;&quot;

  ## Configure pod disruption budgets for Alertmanager
  ## ref: https://kubernetes.io/docs/tasks/run-application/configure-pdb/#specifying-a-poddisruptionbudget
  ## This configuration is immutable once created and will require the PDB to be deleted to be changed
  ## https://github.com/kubernetes/kubernetes/issues/45398
  ##
  podDisruptionBudget:
    enabled: false
    minAvailable: 1
    maxUnavailable: &quot;&quot;

  ## Alertmanager configuration directives
  ## ref: https://prometheus.io/docs/alerting/configuration/#configuration-file
  ##      https://prometheus.io/webtools/alerting/routing-tree-editor/
  ##
  config:
    global:
      resolve_timeout: 5m
      slack_api_url: &quot;I changed this url for the stack overflow question&quot;
    route:
      group_by: ['job']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 12h
      #receiver: 'slack'
      routes:
      - match:
          alertname: DeadMansSwitch
        receiver: 'null'
      - match:
        receiver: 'slack'
        continue: true
    receivers:
    - name: 'null'
    - name: 'slack'
      slack_configs:
      - channel: 'alerts'
        send_resolved: false
        title: '[{{ .Status | toUpper }}{{ if eq .Status &quot;firing&quot; }}:{{ .Alerts.Firing | len }}{{ end }}] Monitoring Event Notification'
        text: &gt;-
          {{ range .Alerts }}
            *Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}`
            *Description:* {{ .Annotations.description }}
            *Graph:* &lt;{{ .GeneratorURL }}|:chart_with_upwards_trend:&gt; *Runbook:* &lt;{{ .Annotations.runbook }}|:spiral_note_pad:&gt;
            *Details:*
            {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
            {{ end }}
          {{ end }}

</code></pre>

<p>So I have finally solved the problem. The problem apparently was that the <a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack" rel="nofollow noreferrer">kube-prometheus-stack</a> and the <a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus" rel="nofollow noreferrer">prometheus</a> helm charts work a bit differently.
So instead of alertmanager.config I had to insert the code (everything starting from global) at alertmanagerFiles.alertmanager.yml.</p>
