<p>I have a kubernetes cluster in Amazon EKS, Autoscaling is set. So when there is load increase a new node spin-up in the cluster and spin-down with respect to load-running. We are monitoring it with Prometheus and send desired alerts with Alertmanager.</p>
<p>So help me with a query that will send alerts whenever Autoscaling is performed in my Cluster.</p>

iemkamran
<p>The logic is not so great, but this works for me in a non-EKS Self Hosted Kubernetes Cluster on AWS EC2s.</p>
<pre><code>(group by (kubernetes_io_hostname, kubernetes_io_role) (container_memory_working_set_bytes ) * 0
</code></pre>
<p>The above query fetches the currently up nodes and multiplies them by 0,</p>
<pre><code>or group by (kubernetes_io_hostname, kubernetes_io_role) (delta ( container_memory_working_set_bytes[1m]))) == 1
</code></pre>
<p>Here, it adds all nodes that existed in the last 1 minute through the <code>delta()</code> function. The default value of the nodes in the <code>delta()</code> function output will be 1, but the existing nodes will be overridden by the value 0, because of the <code>OR</code> precedence. So finally, only the newly provisioned node(s) will have the value 1, and they will get filtered by the equality condition. You can also extract whether the new node is master/worker by the <code>kubernetes_io_role</code> label</p>
<p>Full Query:</p>
<pre><code>(group by (kubernetes_io_hostname, kubernetes_io_role) (container_memory_working_set_bytes ) * 0 or group by (kubernetes_io_hostname, kubernetes_io_role) (delta ( container_memory_working_set_bytes[1m]))) == 1
</code></pre>
<p>You can reverse this query for downscaling of nodes, although that will collide with the cases in which your Kubernetes node Shuts Down Abruptly due to reasons other than AutoScaling</p>
