<p>We have a Grafana Service with two Grafana PODs in our Kubernetes Cluster. Everytime an alert is triggered both instances will fire the alert.</p>
<p>To prevent this from happening, we tried activating the HA alerting, which basically consists of the following configuration:</p>
<pre><code>[unified_alerting]
enabled = true
ha_listen_address = ${POD_IP}:9094
ha_peers = ${POD_IP}:9094

</code></pre>
<p>Since every POD only knows it's own IP address <code>${POD_IP}</code>, we are not able to set the ha_peers value correctly to contain all instances (like described in the Grafana documentation). Therefore we still get duplicate alerts.</p>
<p>Also, if one instance is terminated and another will start, I'm not quite sure how the ha_peers of the remaining active POD will be updated.</p>
<p>We'd like to avoid using work-arouds like fixed IPs because this would go against Kubernetes practices.</p>
<p>Does anyone one know how to circumvent or solve this problem?</p>

elcaos
<p>Headless service is the right solution. It is used in <a href="https://github.com/grafana/helm-charts/tree/main/charts/grafana#high-availability-for-unified-alerting" rel="nofollow noreferrer">official Grafana helm chart</a>:</p>
<pre><code>  unified_alerting:
    enabled: true
    ha_peers: {{ Name }}-headless:9094
</code></pre>
