<p>I have already tried finding resources and articles online for how to create alerts using Grafana 8 UI about the CPU and/or memory usage of my kubernetes cluster pods, but I couldn't find anything, neither on youtube, google, discord, stackoverflow nor reddit.</p>
<p>Does anyone know any guide on how to do that?</p>
<p>The goal is to literally create an alert rule that will send a slack message when the CPU or Memory usage of my kubernetes cluster pods pass over X%. The slack app to receive the grafana message is working, but I have no idea how would be the grafana query.</p>
<p>PS.: I am using Prometheus and node-exporter.</p>

<p>You can try this query for creating an alert if the CPU or Memory usage is above threshold (let say 85%).</p>
<p><code>sum(rate(container_cpu_usage_seconds_total{namespace=&quot;$namespace&quot;, pod=&quot;$pod&quot;, container!=&quot;POD&quot;, container!=&quot;&quot;, pod!=&quot;&quot;}[1m])) by (pod) / sum(kube_pod_container_resource_limits{namespace=&quot;$namespace&quot;, pod=&quot;$pod&quot;, resource=&quot;cpu&quot;}) by (pod) * 100</code></p>
<p>You can check CPU utilization of all pods in the cluster by running:</p>
<p><code>sum(rate(container_cpu_usage_seconds_total{container_name!=&quot;POD&quot;,pod_name!=&quot;&quot;}[5m]))</code></p>
<p>If you want to check CPU usage of each running pod you can use using:</p>
<p><code>sum(rate(container_cpu_usage_seconds_total{container_name!=&quot;POD&quot;,pod_name!=&quot;&quot;}[5m])) by (pod_name).</code></p>
<p>To see actual CPU usage, look at metrics like <code>container_cpu_usage_seconds_total (per container CPU usage)</code> or maybe even <code>process_cpu_seconds_total (per process CPU usage).</code></p>
<p>You can create alert rule in grafana by following the steps provided in the <a href="https://grafana.com/docs/grafana/latest/alerting/alerting-rules/create-grafana-managed-rule/#add-grafana-managed-rule" rel="nofollow noreferrer">document</a> and refer to the <a href="https://stackoverflow.com/questions/61361263/grafana-for-kubernettes-shows-cpu-usage-higher-than-100">link</a> for more information.</p>
