<p>I am running Prometheus (bitnami/prometheus:2.25.2) with Prometheus Alertmanager (bitnami/alertmanager:0.21.0) on AKS 1.19.9.<br />
Alerting is handled by the Alertmanager which in turn routes the alerts to slack channels.</p>
<p>I have noticed that lately certain alerts have been ending up in the “Not grouped” section within the Prometheus Alertmanager WebUI, and not making it into the Slack channel.</p>
<p><a href="https://i.stack.imgur.com/bMgoV.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/bMgoV.png" alt="enter image description here" /></a></p>
<p>I am unable to explain this, as they are grouped by [cluster, alertname] and do contain these labels (in the screenshot blurred, but cluster contains the same value).</p>
<p>To make matters even more confusing (for me anyway) there are certain alerts that also have these labels and are sent correctly.</p>
<p><a href="https://i.stack.imgur.com/VBkir.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/VBkir.png" alt="enter image description here" /></a></p>
<p>The Alert manager routing tree in the config:</p>
<pre><code>spec:
  route:
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 3h
    receiver: fallback
    routes:
    - matchers:
      - name: team
        value: platform-engineering
      groupBy: [cluster, alertname]
      receiver: fallback
      routes:
      - matchers:
        - name: severity
          value: critical
        groupBy: [cluster, alertname]
        receiver: alerts-critical
      - matchers:
        - name: severity
          value: warning
        groupBy: [cluster, alertname]
        receiver: alerts-warning
</code></pre>
<p>Does anybody care to take a stab at what is wrong here? I am obviously missing something :-)<br />
Many thanks in advance!</p>

Morné Kruger
<p>Think I found the issue.
The Prometheus running on the cluster is provisioned by the Prometheus operator: <code>bitnami/prometheus-operator:0.53.1</code>
The routing tree listed above is what you see when looking at the Prometheus Alert manager configuration <em>before</em> deployment.</p>
<p>However … when you visit the Prometheus Alert manager WebUI <em>after deployment</em> and click the “status” tab at the top of the page, it tells a different story.
What I found was that the operator was injecting an extra matcher into the routing tree during the deployment.</p>
<p><a href="https://i.stack.imgur.com/uE67R.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/uE67R.png" alt="enter image description here" /></a></p>
<p>This obviously has consequences for the matching and grouping of alerts in particular if the alerts that are hitting the routing tree are not originating from the namespace monitoring.</p>
<p>In my case only monitoring workloads reside here and the bulk of the workloads comes from namespaces outside <strong>monitoring</strong>.</p>
<p>Reading GitHub issue <a href="https://github.com/prometheus-operator/prometheus-operator/issues/3737" rel="nofollow noreferrer">3737</a> on the Prometheus Operator repo, confirmed this suspicion.</p>
<p>As a workaround I tried Till Adam’s suggestion:</p>
<pre><code>kind: clustermanagement
namespace: prometheus
source_namespace: '{{ $labels.namespace }}'
</code></pre>
<p><em>With this, we have one alertmanagerconfig in the prometheus namespace responsible for all cluster related alerts, no matter what namespace label the metrics originally had.</em></p>
<p>Note that your alerting rules should also be adjusted when using this! The actual namespace where the alert stems from, will now be in <code>source_namespace</code>.</p>
<p>The only edge cases that I have encountered were when you end up losing the namespace label. This seems to be occurring when the alert expression is making use of aggregation operators (count for example).</p>
<p>If I am not mistaken <a href="https://github.com/prometheus-operator/prometheus-operator/pull/3821" rel="nofollow noreferrer">PR3821</a> will introduce the fix (Global alertmanagerconfig) for this challenge.</p>
