<p>I have an application that is ~40 docker containers varying from NoSQL, RDBMS, C applications, Go apps, Python and so on, orchestrated using <code>Kubernetes</code>, Its all running on <code>GCP</code>. With a GLB(Load Balancer) at the frontend.</p>

<p>Now if I create a lot of replicas and give a lot of resources to these applications then everything runs properly. But if I give just enough resources then the frontend sometimes loads very slowly, the web application becomes unresponsive for sometime and then mysteriously comes back up again. </p>

<p>All this happens with no pod evictions or restarts.</p>

<p>When this happens I can see that the CPU/Memory are at 50%, so resources are not exhausted.</p>

<p>How to a go about debugging what is the reason for slowness? How to I calibrate which application requires how mush of resources?</p>

<p>You haven't mentioned about any monitoring tools implemented in Kubernetes cluster that you can use to check overall cluster performance or check application resource usage.</p>

<p>All the monitoring aspects based on the metrics characteristics, therefore Kubernetes offers <a href="https://kubernetes.io/docs/tasks/debug-application-cluster/resource-usage-monitoring/#resource-metrics-pipeline" rel="nofollow noreferrer">Resource metrics pipeline</a> gathered by <a href="https://github.com/kubernetes-incubator/metrics-server" rel="nofollow noreferrer">metrics-server</a> or <a href="https://kubernetes.io/docs/tasks/debug-application-cluster/resource-usage-monitoring/#full-metrics-pipelines" rel="nofollow noreferrer">Full metrics pipeline</a> for some more advanced metrics and <a href="https://prometheus.io/" rel="nofollow noreferrer">Prometheus</a> is a good example for that approach.</p>

<p>For GCP related environments you can use <a href="https://cloud.google.com/logging/" rel="nofollow noreferrer">Stackdriver logging</a> with lots of monitoring features and appropriate group of <a href="https://cloud.google.com/monitoring/api/metrics" rel="nofollow noreferrer">metrics</a>.</p>

<p>Therefore, I would start from checking monitoring metrics from underlying Kubernetes resources in order to collect measurements and take necessary actions to improve total cluster productivity. </p>
