title: Stuck threads in weblogic classloader
body: <p>We have a situation here where more than 100 threads are stuck in a weblogic (10.3.6) managed server. </p>
  
  <p>I am analysing a threaddump and find 150 threads blocked waiting on a lock in the classloader, like :
  <code>"jmsContainer-14" prio=10 tid=0x00007f11485d7000 nid=0x14c8 waiting for monitor entry [0x00007f1023590000]
     java.lang.Thread.State: BLOCKED (on object monitor)
      at weblogic.utils.classloaders.ChangeAwareClassLoader.loadClass(ChangeAwareClassLoader.java:48)
      - waiting to lock &lt;0x0000000782aeb288&gt; (a weblogic.utils.classloaders.ChangeAwareClassLoader)
      at ch.qos.logback.classic.spi.PackagingDataCalculator.loadClass(PackagingDataCalculator.java:207)
  </code></p>
  
  <p>(10 occurences)</p>
  
  <p>or</p>
  
  <p><code>"[STUCK] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f10d002d800 nid=0x5ea5 waiting for monitor entry [0x00007f1136eeb000]
     java.lang.Thread.State: BLOCKED (on object monitor)
      at weblogic.utils.classloaders.ChangeAwareClassLoader.loadClass(ChangeAwareClassLoader.java:48)
      - waiting to lock &lt;0x0000000782aeb288&gt; (a weblogic.utils.classloaders.ChangeAwareClassLoader)
      at javax.xml.datatype.FactoryFinder.getProviderClass(FactoryFinder.java:115)
      at javax.xml.datatype.FactoryFinder.newInstance(FactoryFinder.java:181)
  </code></p>
  
  <p>(137) occurences</p>
  
  <p>No all stuck thread are blocked for the same reason, but we have 150 threads waiting to lock the same lock &lt;0x0000000782aeb288>, wich I find strange because it is in the classLoader. </p>
  
  <p>I don't think it is the root cause of the problem we are experiencing right now, but if someone could explain this behavious it could be of great help.</p>
  
  <p>Thanks</p>
  
  <p>EDIT :</p>
  
  <p>Here is the blocking thread :</p>
  
  <pre><code>    ExecuteThread '136' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f11484af800 nid=0x7640 runnable [0x00007f102c11b000]
     java.lang.Thread.State: RUNNABLE
      at sun.nio.cs.UTF_8$Encoder.encode(UTF_8.java:632)
      at java.util.zip.ZipCoder.getBytes(ZipCoder.java:86)
      at java.util.zip.ZipFile.getEntry(ZipFile.java:306)
      - locked &lt;0x0000000782b2ddf8&gt; (a java.util.jar.JarFile)
      at java.util.jar.JarFile.getEntry(JarFile.java:226)
      at weblogic.utils.classloaders.ZipClassFinder.getSource(ZipClassFinder.java:39)
      at weblogic.utils.classloaders.JarClassFinder.getSource(JarClassFinder.java:50)
      at weblogic.utils.classloaders.AbstractClassFinder.getClassSource(AbstractClassFinder.java:31)
      at weblogic.utils.classloaders.MultiClassFinder.getClassSource(MultiClassFinder.java:58)
      at weblogic.utils.classloaders.MultiClassFinder.getClassSource(MultiClassFinder.java:58)
      at weblogic.utils.classloaders.MultiClassFinder.getClassSource(MultiClassFinder.java:58)
      at weblogic.utils.classloaders.MultiClassFinder.getClassSource(MultiClassFinder.java:58)
      at weblogic.application.utils.CompositeWebAppFinder.getClassSource(CompositeWebAppFinder.java:88)
      at weblogic.utils.classloaders.MultiClassFinder.getClassSource(MultiClassFinder.java:58)
      at weblogic.utils.classloaders.MultiClassFinder.getClassSource(MultiClassFinder.java:58)
      at weblogic.utils.classloaders.CodeGenClassFinder.getClassSource(CodeGenClassFinder.java:25)
      at weblogic.utils.classloaders.GenericClassLoader.findLocalClass(GenericClassLoader.java:291)
      - eliminated &lt;0x0000000782aeb288&gt; (a weblogic.utils.classloaders.ChangeAwareClassLoader)
      at weblogic.utils.classloaders.GenericClassLoader.findClass(GenericClassLoader.java:270)
      at weblogic.utils.classloaders.ChangeAwareClassLoader.findClass(ChangeAwareClassLoader.java:64)
      at weblogic.utils.classloaders.ChangeAwareClassLoader.loadClass(ChangeAwareClassLoader.java:49)
      - locked &lt;0x0000000782aeb288&gt; (a weblogic.utils.classloaders.ChangeAwareClassLoader)
      at javax.xml.datatype.FactoryFinder.getProviderClass(FactoryFinder.java:115)
      at javax.xml.datatype.FactoryFinder.newInstance(FactoryFinder.java:181)
  </code></pre>
  
  <p>sorry, I should have posted it earlier.</p>
  
  <p>Nearly all the threads are bloked in, or blocked by one of two lines of codes containing this piece of code:</p>
  
  <p><code>DatatypeFactory.newInstance().newXMLGregorianCalendar(dateDebCre)
  </code></p>
  
  <p>Apparently Weblogic is trying very hard to load the same jar again and again. Which I do not understand, because it is supposed to be stored in a cache in memory, right ?</p>
  
  <p>First, we will remove this lines of code as they are completely useless (a shame it is). But again, I think understanding the problem would be better.</p>
  
  <p>Thank you</p>