title: Jenkins build job using 100% CPU on executors
body: <p>For a while now, we are seeing our Jenkins machines being pegged at 100% CPU (or 200% or 400%, depending on the number of cores) according to top:</p>
  
  <pre><code>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                      
  23376 ec2-user  20   0 4220m 100m    0 S 99.2  1.3 324945:21 java
  </code></pre>
  
  <p>These JVM processes linger around even after a build has completed and while no builds are currently running. This problem occurs for builds on both the master node as well as on slave nodes. The JVM running the slave agent itself has completely normal CPU usage.</p>
  
  <p>Once I was finally able to get a thread dump, there was only a single non-system thread that was runnable and not waiting on a lock:</p>
  
  <pre><code>"main" #1 prio=5 os_prio=0 tid=0x00007f0834008800 nid=0x5b51 runnable [0x00007f083abf3000]
  java.lang.Thread.State: RUNNABLE
  at java.util.HashMap.put(HashMap.java:611)
  at java.util.HashSet.add(HashSet.java:219)
  at net.bytebuddy.dynamic.scaffold.MethodGraph$Compiler$Default$Key$Harmonized.detach(MethodGraph.java:878)
  at net.bytebuddy.dynamic.scaffold.MethodGraph$Compiler$Default$Key$Store$Entry$Resolved.asNode(MethodGraph.java:1331)
  at net.bytebuddy.dynamic.scaffold.MethodGraph$Compiler$Default$Key$Store.asGraph(MethodGraph.java:1138)
  at net.bytebuddy.dynamic.scaffold.MethodGraph$Compiler$Default.compile(MethodGraph.java:507)
  at net.bytebuddy.dynamic.scaffold.MethodGraph$Compiler$AbstractBase.compile(MethodGraph.java:423)
  at net.bytebuddy.dynamic.scaffold.MethodRegistry$Default.prepare(MethodRegistry.java:489)
  at net.bytebuddy.dynamic.scaffold.subclass.SubclassDynamqicTypeBuilder.make(SubclassDynamicTypeBuilder.java:153)
  at net.bytebuddy.dynamic.DynamicType$Builder$AbstractBase$Delegator.make(DynamicType.java:2508)
  at org.mockito.internal.creation.bytebuddy.MockBytecodeGenerator.generateMockClass(MockBytecodeGenerator.java:60)
  at org.mockito.internal.creation.bytebuddy.CachingMockBytecodeGenerator$CachedBytecodeGenerator.generate(CachingMockBytecodeGenerator.java:72)
  at org.mockito.internal.creation.bytebuddy.CachingMockBytecodeGenerator$CachedBytecodeGenerator.getOrGenerateMockClass(CachingMockBytecodeGenerator.java:64)
  at org.mockito.internal.creation.bytebuddy.CachingMockBytecodeGenerator.get(CachingMockBytecodeGenerator.java:27)
  at org.mockito.internal.creation.bytebuddy.ByteBuddyMockMaker.createProxyClass(ByteBuddyMockMaker.java:54)
  at org.mockito.internal.creation.bytebuddy.ByteBuddyMockMaker.createMock(ByteBuddyMockMaker.java:27)
  at org.mockito.internal.util.MockUtil.createMock(MockUtil.java:32)
  at org.mockito.internal.MockitoCore.mock(MockitoCore.java:55)
  at org.mockito.Mockito.mock(Mockito.java:1449)
  at org.mockito.internal.configuration.MockAnnotationProcessor.process(MockAnnotationProcessor.java:33)
  at org.mockito.internal.configuration.MockAnnotationProcessor.process(MockAnnotationProcessor.java:16)
  at org.mockito.internal.configuration.DefaultAnnotationEngine.createMockFor(DefaultAnnotationEngine.java:43)
  at org.mockito.internal.configuration.DefaultAnnotationEngine.process(DefaultAnnotationEngine.java:66)
  at org.mockito.internal.configuration.InjectingAnnotationEngine.processIndependentAnnotations(InjectingAnnotationEngine.java:71)
  at org.mockito.internal.configuration.InjectingAnnotationEngine.process(InjectingAnnotationEngine.java:55)
  at org.mockito.MockitoAnnotations.initMocks(MockitoAnnotations.java:108)
  at org.mockito.internal.runners.JUnit45AndHigherRunnerImpl$1.withBefores(JUnit45AndHigherRunnerImpl.java:27)
  at org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:276)
  at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
  at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
  at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
  at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
  at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
  at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
  at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
  at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
  at org.mockito.internal.runners.JUnit45AndHigherRunnerImpl.run(JUnit45AndHigherRunnerImpl.java:37)
  at org.mockito.runners.MockitoJUnitRunner.run(MockitoJUnitRunner.java:62)
  at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:59)
  at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.executeTestSet(AbstractDirectoryTestSuite.java:115)
  at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.execute(AbstractDirectoryTestSuite.java:102)
  at org.apache.maven.surefire.Surefire.run(Surefire.java:180)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:497)
  at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:350)
  at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1021)
  </code></pre>
  
  <p>I took several consecutive thread dumps, and this thread was pretty much always at the exactly same spot - HashMap.put(HashMap.java:611) - while that JVM was taking up 100% CPU. I haven't had an opportunity to attach a debugger while this happens, but I just wanted to see if anyone recognized this as a known bug. To me, it looks like some sort of a hot-spinning endless loop that is trying to add some item to a hash map over and over again. This may be related to Jenkins, but the other major players that make an appearance in the stack trace are obviously Maven Surefire, JUnit, Mockito and ByteBuddy (sorry for including all those tags).</p>
  
  <p>I can't reliably reproduce this issue, and unfortunately I also don't know which one of our hundreds of builds leaves behind these screwed up JVMs. Just for completeness sake, the environment is Jenkins 2.46.1, Maven 3.3.3, Surefire 2.19.1, JUnit 4.12, but a unfortunately a range of different Mockito (and hence ByteBuddy) versions. I'm hoping that someone recognizes this as a known bug in one of the involved components and can suggest a work-around...</p>