title: How to find the root cause of high CPU usage of Kafka brokers?
body: <p>I am in charge of operating two kafka clusters (one for prod and one for our dev environment). The setup is mostly similiar, but the dev environment has no SASL/SSL setup and uses just 4 instead of 8 brokers. Each broker is assigned to a dedicated google kubernetes node with 4 vCPU and 26GB RAM.</p>
  
  <p>On our dev environment we've got roughly 1000 messages in / sec and each of the 4 brokers uses pretty consistently 3 out of the 4 available CPU cores (75% CPU usage).</p>
  
  <p>On our prod environment we got about 1500 messages in / sec and the CPU usage is also 3 out of 4 cores there. </p>
  
  <p>It seems that CPU usage is at least the bottleneck for us and I'd like to know how I can perform a CPU profiling, so that I know what exactly is causing the high cpu usage. Since it's relatively consistent I guess it could be our snappy compression. </p>
  
  <p>I am interested in all ideas how I could investigate the cause of the high cpu usage and how I could tweak that in my cluster.</p>
  
  <ul>
  <li><p>Apache Kafka version: 2.1 (CPU load used to be similiar on Kafka 0.11.x too)</p></li>
  <li><p>Dev Cluster (Snappy compression, no SASL/SSL, 4 Brokers): 1000 messages in / sec, 3 CPU cores consistent usage</p></li>
  <li><p>Prod cluster (Snappy compression, SASL/SSL, 8 Brokers): 1500 messages in / sec, 3 CPU cores consistent usage</p></li>
  </ul>
  
  <p>Side note: I already made sure producers produce their messages snappy compressed. I have access to all JMX metrics, couldn't find anything useful for figuring out the CPU usage though.</p>
  
  <p>I already have metrics attached to my prometheus (this is where I got the CPU usage stats from too). The problem is that the container's CPU usage doesn't tell me WHY it is that high. I need more granularity e. g. what are CPU cycles being spent on (compression? broker communication? sasl/ssl?).</p>