<p>Currently we have a set of microservice hosted on kubernetes cluster. We are setting hpa values based on rough estimates. I am planning to monitor horizontal pod autoscaling behavior using grafana to ensure we are not over/under allocating the resources like CPU/memory and come up with possible cost optimization recommendation. Need directions on how to achieve this.</p>
<p>I am new to Kubernetes world. Need directions on how to achieve this.</p>

<p><strong>tl;dr</strong></p>
<ol>
<li>Monitor resource consumption of each pod.</li>
<li>Monitor pod restarts and number of replicas.</li>
<li>Use a load test.</li>
</ol>
<p><strong>Memory</strong></p>
<p>As a starting point, you could monitor CPU- and memory-consumption of each pod. For example you can do something like this:</p>
<pre><code>sum by (pod) (container_memory_usage_bytes{container=...}/
sum by (pod) (kube_pod_container_resource_requests{container=...})
</code></pre>
<p>If you follow the advice given in <a href="https://blog.kubecost.com/blog/requests-and-limits/" rel="nofollow noreferrer">A Practical Guide to Setting Kubernetes Requests and Limits</a>, the limit setting is related to the request setting. With such a query you can analyse, if the requested memory per pod is roughly realistic. Depending on the configuration of the autoscaler, this could be helpful. You could define some grafana alert rule that triggers an alarm if the desired ratio between used and requested memory exceeds some threshold.</p>
<p><strong>Restarts</strong></p>
<p>If the pod exceeds a given memory limit, the pod will crash and kubernetes will trigger a restart. With the following metric you can monitor restarts:</p>
<pre><code>sum by (pod) (increase(kube_pod_container_status_restarts_total{...}[1h]))
</code></pre>
<p><strong>CPU</strong></p>
<p>CPU usage is also relevant:</p>
<pre><code>process_cpu_usage{container=&quot;...&quot;}
</code></pre>
<p>For additional queries, have a look at <a href="https://stackoverflow.com/q/55143656/11934850">Prometheus queries to get CPU and Memory usage in kubernetes pods</a>.</p>
<p><strong>Replicas</strong></p>
<p>Now, as you have basic metrics in place, what about the autoscaler itself? You'll be able to count the number of active pods like this:</p>
<pre><code>kube_horizontalpodautoscaler_status_current_replicas{}
</code></pre>
<p>Note that you might need to filter this metric by label <code>horizontalpodautoscaler</code>. But I recommend that you first run the metric without filters to get information about all running autoscalers.</p>
<p>To have better cost control, autoscaling is usually limited to a maximum of replicas. If you are running on maximum, you might want to check if the given maximum is to low. With kubectl you can check the status like this:</p>
<pre><code>kubectl describe hpa
</code></pre>
<p>Have a look at condition <code>ScalingLimited</code>.</p>
<p>With grafana:</p>
<pre><code>kube_horizontalpodautoscaler_status_condition{condition=&quot;ScalingLimited&quot;}
</code></pre>
<p>A list of kubernetes metrics can be found at <a href="https://github.com/kubernetes/kube-state-metrics/tree/master/docs#exposed-metrics" rel="nofollow noreferrer">kube-state-metrics</a>. Have a look at <a href="https://github.com/kubernetes/kube-state-metrics/blob/master/docs/horizontalpodautoscaler-metrics.md" rel="nofollow noreferrer">Horizontal Pod Autoscaler Metrics</a> and <a href="https://github.com/kubernetes/kube-state-metrics/blob/master/docs/replicationcontroller-metrics.md" rel="nofollow noreferrer">ReplicationController metrics</a>.</p>
<p><strong>Use a load test</strong></p>
<p>In the <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/" rel="nofollow noreferrer">HorizontalPodAutoscaler Walkthrough</a> there is a point where you need to increase the load on your application. There are several tools, that you may use for this, such as Apache Bench or JMeter.</p>
<p>In my experience, upscaling is easy to achieve, the tricky part is the downscaling. Therefore, you need to play with increasing <em>and decreasing</em> the load.</p>
