<p>We have a Grafana Service with two Grafana PODs in our Kubernetes Cluster. Everytime an alert is triggered both instances will fire the alert.</p>
<p>To prevent this from happening, we tried activating the HA alerting, which basically consists of the following configuration:</p>
<pre><code>[unified_alerting]
enabled = true
ha_listen_address = ${POD_IP}:9094
ha_peers = ${POD_IP}:9094

</code></pre>
<p>Since every POD only knows it's own IP address <code>${POD_IP}</code>, we are not able to set the ha_peers value correctly to contain all instances (like described in the Grafana documentation). Therefore we still get duplicate alerts.</p>
<p>Also, if one instance is terminated and another will start, I'm not quite sure how the ha_peers of the remaining active POD will be updated.</p>
<p>We'd like to avoid using work-arouds like fixed IPs because this would go against Kubernetes practices.</p>
<p>Does anyone one know how to circumvent or solve this problem?</p>

elcaos
<p>As Jan Garaj already mentioned, headless services are the way to go.</p>
<p>If the problem still occurs after using a headless service, it may have something to do with the network setup/firewall. Grafana uses gossip protocols to synchronize the pods. Gossip uses both TCP and UDP on port 9094. We only accepted TCP but not UDP. After allowing traffic with UDP on port 9094 the alerts were deduplicated.</p>
<p><strong>TL;DR:</strong> Allow TCP and UDP traffic on port 9094 as described in <a href="https://grafana.com/docs/grafana/latest/alerting/high-availability/enable-alerting-ha/" rel="nofollow noreferrer">Grafana Documentation</a>!</p>
