<p>I use JMeter for HTTP-based performance testing in Kubernetes. Each JMeter instance has more than 80GB memory.</p>
<p>Everything works fine, but, every 10 minutes, the request rate drops from ~101Requests/Second to ~95Requests/Second:</p>
<pre><code>summary +   3116 in 00:00:30 =  103.9/s Avg:    20 Min:     0 Max:   293 Err:     0 (0.00%) Active: 20 Started: 20 Finished: 0
summary =  58661 in 00:10:13 =   95.6/s Avg:    29 Min:     0 Max:  2625 Err:    23 (0.04%)
summary +   2883 in 00:00:30 =   96.0/s Avg:    24 Min:     0 Max:  2330 Err:     0 (0.00%) Active: 20 Started: 20 Finished: 0
summary =  61544 in 00:10:43 =   95.6/s Avg:    28 Min:     0 Max:  2625 Err:    23 (0.04%)
summary +   3097 in 00:00:30 =  103.2/s Avg:    20 Min:     0 Max:   319 Err:     0 (0.00%) Active: 20 Started: 20 Finished: 0
</code></pre>
<p>I have no idea why the request rate is decreasing. I also measure the latency between JMeter and the SUT, but I can't see any increasing latency or anything. Also, the HTTP response code between JMeter and the SUT is always 200. Memory usage is ~30GB, so each instance has more than enough memory. CPU usage is ~80% per instance.</p>
<p>Any ideas?</p>

bennex
<p>It's hard to say what's wrong without seeing resources consumption in Kubernetes, JMeter, your application, etc so you need to set up a proper monitoring of everything including JMeter <a href="https://dzone.com/articles/diving-into-jvm-metrics" rel="nofollow noreferrer">JVM metrics</a>. If you don't have any monitoring toolchain you can consider using <a href="https://www.blazemeter.com/blog/how-monitor-your-server-health-performance-during-jmeter-load-test" rel="nofollow noreferrer">JMeter PerfMon plugin</a> for this.</p>
<p>If you need a guess - It might be due to <a href="https://www.oracle.com/webfolder/technetwork/tutorials/obe/java/gc01/index.html" rel="nofollow noreferrer">Garbage Collection</a> like it's described in <a href="https://howtojboss.wordpress.com/2012/07/31/concurrent-high-throughput-performance-testing-with-jmeter/" rel="nofollow noreferrer">Concurrent, High Throughput Performance Testing with JMeter</a></p>
