<p><strong>Use case / Problem</strong></p>

<p>I am in charge of maintaining a kubernetes cluster with 40 nodes (split across 2 zones). We have roughly 100 microservices and platform stuff like Kafka brokers running in this cluster. All microservices have defined resource request &amp; limits. Most of them however are burstable and don't have guaranteed RAM. Developers who deploy their services in our cluster defined limits far greater than the request (see example below) which eventually caused a lot of evicted pods on various nodes.  We still want to use burstable resources in our services though, as we can save money using burstable resources. Therefore I need a better monitoring possibility of all pods running on each node, containing these information:</p>

<ul>
<li>Node name &amp; CPU / RAM capacity</li>
<li>All pod names plus

<ul>
<li>pod's resource requests &amp; limits</li>
<li>pod's current cpu &amp; ram usage</li>
</ul></li>
</ul>

<p>This way I could easily identify two problematic kind of services:</p>

<p><strong>Case A:</strong> The microservice which just sets huge resource limits, because the developer was just testing stuff or is too lazy to bench/monitor his service</p>

<pre><code>resources:
  requests:
    cpu: 100m
    ram: 500Mi
  limits:
    cpu: 6
    ram: 20Gi
</code></pre>

<p><strong>Case B:</strong> Too many services on the same node which have set not accurate resource limits (e. g. 500Mi, but the service constantly uses 1.5Gi RAM). This case happened to us, because Java developers didn't notice the Java garbage collector will only start to cleanup when 75% of the available RAM has been used.</p>

<p><strong>My question:</strong></p>

<p>How could I properly monitor this and therefore identify misconfigured microservices in order to prevent such eviction problems? At a smaller scale I could simply run <code>kubectl describe nodes</code> and <code>kubectl top pods</code> to figure it out manually, but at this scale that doesn't work anymore.</p>

<p><em>Note:</em> I couldn't find any existing solution for this problem (including prometheus + grafana boards using kube metrics and similiar). I thought it's possible but visualizing this stuff in Grafana is really hard.</p>

kentor
<p>I ended up writing an own prometheus exporter for this purpose. While node exporter provides usage statistics and kube state metrics exposes metrics about your kubernetes resource objects it's not easy to combine and aggregate these metrics so that they provide valuable information to solve the described use case.</p>

<p>With Kube Eagle (<a href="https://github.com/google-cloud-tools/kube-eagle/" rel="nofollow noreferrer">https://github.com/google-cloud-tools/kube-eagle/</a>) you can easily create such a dashboard (<a href="https://grafana.com/dashboards/9871" rel="nofollow noreferrer">https://grafana.com/dashboards/9871</a>):</p>

<p><a href="https://i.stack.imgur.com/3J7ZQ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/3J7ZQ.png" alt="Grafana dashboard for Kubernetes resource monitoring"></a></p>

<p>I also wrote a medium article about how this has helped me saving lots of hardware resources: <a href="https://medium.com/@martin.schneppenheim/utilizing-and-monitoring-kubernetes-cluster-resources-more-effectively-using-this-tool-df4c68ec2053" rel="nofollow noreferrer">https://medium.com/@martin.schneppenheim/utilizing-and-monitoring-kubernetes-cluster-resources-more-effectively-using-this-tool-df4c68ec2053</a></p>
