<p>I want to find the sum number of alerts for all the pods starting with "sendsms" over 10minutes.</p>

<p>I am able to do use <a href="https://prometheus.io/docs/prometheus/latest/querying/functions/#label_replace" rel="nofollow noreferrer">label_replace()</a> to do this on the instant vector. But when i want to do this for over 10 minutes data, it cannot work as label_replace only works on instant vector.</p>

<p>Explaining the problem with an example:</p>

<pre><code>ALERTS{alertname="CPUThrottlingHigh",pod="sendsms-dbed"} 10
ALERTS{alertname="CPUThrottlingHigh",pod="sendsms-ebed"} 20
ALERTS{alertname="CPUThrottlingHigh",pod="sendsms-fbed"} 30
</code></pre>

<pre><code>ALERTS{alertname="CPUThrottlingHigh",pod="sendmail-gbed"} 60
ALERTS{alertname="CPUThrottlingHigh",pod="sendmail-hbed"} 70
ALERTS{alertname="CPUThrottlingHigh",pod="sendmail-ibed"} 80
</code></pre>

<p>Using label replace i can add a new label using the REGEX and then i can group it and get the results. </p>

<p><code>label_replace(ALERTS{alertname="CPUThrottlingHigh", "podname", "$1", "pod", "([a-z-A-Z]+)-.*")</code></p>

<pre><code>ALERTS{alertname="CPUThrottlingHigh",pod="sendsms-dbed", podname="sendsms"} 10
ALERTS{alertname="CPUThrottlingHigh",pod="sendsms-dbed", podname="sendsms"} 10
</code></pre>

<p><strong>How to do this for ALERTS in 10 minutes and calculate the sum?</strong></p>

<p>I want some result like this for last 10 minutes</p>

<pre><code>ALERTS{alertname="CPUThrottlingHigh",podname="sendsms"} 60
ALERTS{alertname="CPUThrottlingHigh",podname="sendmail"} 210
</code></pre>

<p><strong>Objective</strong>: Find the pods which are creating maximum no of alerts in last 1 week.</p>

Alok Kumar Singh
<p>I was able to solve this problem by doing label_replace after doing the sum </p>

<p><strong>Query</strong></p>

<pre><code>sort_desc(sum by (pod_set) (label_replace(sort_desc(sum by (namespace, pod) (avg_over_time(ALERTS{alertname=~"(KubeDeploymentReplicasMismatch|KubePodNotReady|KubePodCrashLooping|KubeJobFailed)", alertstate="firing"}[1w]))), "pod_set", "$1", "pod", "([a-z-A-Z]+)-.*" )))
</code></pre>

<p><strong>Result</strong></p>

<pre><code>{pod_set="sendsms"} 62
{pod_set="emailspreprocessor"}  32
{pod_set="sendmail"}    21
</code></pre>
