title: Java 8 sequential streams increase CPU usage very high
body: <p>In my spring boot service, I am validating incoming orders based upon order details and customer details.</p>
  <p>In customer details, I have different lists of objects like Services, Attributes, Products, etc. and for every list, I am doing something like below:</p>
  <pre><code>products.stream()  
         .filter(Objects::nonNull)  
         .map(Product::getResource)  
         .filter(Objects::nonNull)  
         .filter(&lt;SimplePredicate&gt;)  
         .collect(Collectors.toList());  
  </code></pre>
  <p>I am using streams like this many times for products, services &amp; attributes. We observed that performance-wise it is giving very high TPS and memory usage is also very optimal. But this is consuming CPU very much. We are running the service in Kubernetes pods and it is taking 90% of the CPU provided.</p>
  <p>One more interesting observation is, the more CPU we give, TPS achieved is higher and CPU usage also reaches 90%.</p>
  <p>Is it because Streams consume more CPU? Or is it because of high Garbage Collection because after every iteration of Streams the internal memory might be garbage collected?</p>
  <p><strong>EDIT-1:</strong></p>
  <p>Upon further investigation using Load Testing, it is observed that:</p>
  <ul>
  <li>Whenever we increase concurrent threads, due to high CPU usage the service starts not responding and followed by a sudden decrease in CPU and thus resulting in low TPS.</li>
  <li>Whenever we decrease concurrent threads, CPU usage still remains high but the service is performing in the most optimal way i.e. high TPS.</li>
  </ul>
  <p>The following are the statistics of TPS vs. CPU under different CPU/thread configuration.</p>
  <p>CPU: 1500m, Threads:70</p>
  <pre><code>| TPS | 176  | 140 | 125 | 79 | 63 |
  |----------------------------------|
  | CPU | 1052 | 405 | 201 | 84 | 13 |  
  </code></pre>
  <p>CPU: 1500m, Threads:35</p>
  <pre><code>| TPS | 500 | 510 | 500 | 530 |
  |-----------------------------|
  | CPU | 1172| 1349| 1310| 1214|  
  </code></pre>
  <p>CPU: 2500m, Threads:70</p>
  <pre><code>| TPS |  20 |  20 |  25 |  28 | 26 |
  |----------------------------------|
  | CPU | 2063| 2429| 2303| 879 | 35 |  
  </code></pre>
  <p>CPU: 2500m, Threads:35</p>
  <pre><code>| TPS | 1193 | 1200 | 1200 | 1230 |
  |---------------------------------|
  | CPU | 600  | 1908 | 2044 | 1949 | 
  </code></pre>
  <p><strong>Tomcat Configuration Used:</strong></p>
  <pre><code>server.tomcat.max-connections=100
  server.tomcat.max-threads=100
  server.tomcat.min-spare-threads=5
  </code></pre>
  <p><strong>EDIT-2:</strong><br />
  The thread dump analysis says: 80% of the <code>http-nio</code> threads are in <code>Waiting on condition</code> state. That means all the threads are waiting for something and no one is consuming any CPU that explains low CPU usage. <strong>But what could be causing the threads going for waiting? I'm not using any Asynchronous Calls in the service also.</strong> Even I'm not using any parallel streams, only sequential streams as mentioned above.</p>
  <p>The following is the Thread dump when CPU and TPS go down:</p>
  <pre><code>&quot;http-nio-8090-exec-72&quot; #125 daemon prio=5 os_prio=0 tid=0x00007f014001e800 nid=0x8f waiting on condition [0x00007f0158ae1000]
     java.lang.Thread.State: **TIMED_WAITING** (parking)
      at sun.misc.Unsafe.park(Native Method)
      - parking to wait for  &lt;0x00000000d7470b10&gt; (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
      at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
      at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:89)
      at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:33)
      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
      at java.lang.Thread.run(Thread.java:748)
  
     Locked ownable synchronizers:
      - None
  </code></pre>