<p>I deployed a web application on kubernetes cluster. This application consists of multiple nodes, and each node has multiple pods. How can I do the performance measurement of whole application? I can see some metrics results on prometheus/ grafana but these results for each node/pod, not for whole application. I am just trying to understand the bigger picture. For example, I see that application data is stored in redis(pod), but is it enough to look into only that pod to measure latency?</p>

user19717254
<p>Every kubelet has <code>cAdvisor</code> (<a href="https://github.com/google/cadvisor" rel="nofollow noreferrer">link</a>) integrated into the binary. <code>cAdvisor</code> provides container users an understanding of the resource usage and performance characteristics of their running containers. Combine <code>cAdvisor</code> with an additional exporter like <code>JMX exporter</code> (for java, <a href="https://github.com/prometheus/jmx_exporter" rel="nofollow noreferrer">link</a>), <code>BlackBox exporter</code> for probe-ing urls (response time, monitor http codes, etc. <a href="https://github.com/prometheus/blackbox_exporter" rel="nofollow noreferrer">link</a>). There are also frameworks, that provide metrics such as Java Springboot on path <code>/actuator/prometheus</code> and you can scrape these metrics. There are many different exporters (<a href="https://github.com/prometheus?q=exporter&amp;type=all&amp;language=&amp;sort=" rel="nofollow noreferrer">link</a>), with each doing something else. When you gather all these metrics, you can have a bigger overview about the state of your application. Couple this with handmade Grafana dashboards and Alerting (<code>AlertManager</code> e.g.) and you can monitor almost everything about your application.</p>
<p>As per the prometheus/grafana stack, I guess what you are talking about is the <code>kube-prom-stack</code> with default dashboards already implemented into them.</p>
