title: Hibernate "APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!"
body: <p>I recently got the following messages in our logs followed by a JVM crash (Due to OOME). I am not sure what to make of this and would really appreciate any guidance.</p>
  
  <pre><code>2015-03-19 21:15:02,457 [Timer-0] WARN  (ThreadPoolAsynchronousRunner.java [run]:608) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@6824f21c -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks! 
  2015-03-19 21:26:29,543 [Timer-0] WARN  (ThreadPoolAsynchronousRunner.java [run]:624) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@6824f21c -- APPARENT DEADLOCK!!! Complete Status: 
      Managed Threads: 3
      Active Threads: 3
      Active Tasks: 
          com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@15da1b6b (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
          com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@b35b08a (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
          com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@51cfdd17 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
      Pending Tasks: 
          com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@19397937
          com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@5c7d3838
          com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@7aea62dd
          com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@55622ff2
          com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@74004a8
  Pool thread stack traces:
      Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
          com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:560)
      Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
          java.net.SocketOutputStream.socketWrite0(Native Method)
          java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
          java.net.SocketOutputStream.write(SocketOutputStream.java:153)
          java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
          java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
          com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3227)
          com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1917)
          com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2060)
          com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2536)
          com.mysql.jdbc.ConnectionImpl.configureClientCharacterSet(ConnectionImpl.java:1751)
          com.mysql.jdbc.ConnectionImpl.initializePropsFromServer(ConnectionImpl.java:3425)
          com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2196)
          com.mysql.jdbc.ConnectionImpl.&lt;init&gt;(ConnectionImpl.java:718)
          com.mysql.jdbc.JDBC4Connection.&lt;init&gt;(JDBC4Connection.java:46)
          sun.reflect.GeneratedConstructorAccessor306.newInstance(Unknown Source)
          sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
          java.lang.reflect.Constructor.newInstance(Constructor.java:408)
          com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
          com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:302)
          com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:282)
          com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
          com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
          com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
          com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
          com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
          com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
          com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
          com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
      Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
          com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:560)
  
  
  2015-03-19 21:56:59,137 [Timer-0] WARN  (ThreadPoolAsynchronousRunner.java [run]:608) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@6824f21c -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks! 
  2015-03-19 21:56:59,143 [eXistThread-18676] ERROR (XQueryServlet.java [process]:566) - Java heap space 
  java.lang.OutOfMemoryError: Java heap space
      at org.exist.storage.btree.BTree$BTreeNode.read(BTree.java:1269)
      at org.exist.storage.btree.BTree$BTreeNode.access$16(BTree.java:1239)
      at org.exist.storage.btree.BTree.getBTreeNode(BTree.java:460)
      at org.exist.storage.btree.BTree.scanSequential(BTree.java:413)
      at org.exist.storage.btree.BTree$BTreeNode.scanNextPage(BTree.java:2039)
      at org.exist.storage.btree.BTree$BTreeNode.query(BTree.java:1835)
      at org.exist.storage.btree.BTree$BTreeNode.query(BTree.java:1759)
      at org.exist.storage.btree.BTree$BTreeNode.query(BTree.java:1759)
      at org.exist.storage.btree.BTree$BTreeNode.query(BTree.java:1759)
      at org.exist.storage.btree.BTree$BTreeNode.access$12(BTree.java:1734)
      at org.exist.storage.btree.BTree.query(BTree.java:379)
      at org.exist.storage.structural.NativeStructuralIndexWorker.scanByType(NativeStructuralIndexWorker.java:259)
      at org.exist.dom.VirtualNodeSet.getNodesFromIndex(VirtualNodeSet.java:457)
      at org.exist.dom.VirtualNodeSet.realize(VirtualNodeSet.java:585)
      at org.exist.dom.VirtualNodeSet.iterator(VirtualNodeSet.java:860)
      at org.exist.dom.AbstractNodeSet.iterator(AbstractNodeSet.java:1)
      at org.exist.storage.structural.NativeStructuralIndexWorker.findDescendantsByTagName(NativeStructuralIndexWorker.java:162)
      at org.exist.xquery.LocationStep.getAttributes(LocationStep.java:645)
      at org.exist.xquery.LocationStep.eval(LocationStep.java:434)
      at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71)
      at org.exist.xquery.PathExpr.eval(PathExpr.java:264)
      at org.exist.xquery.Predicate.selectByNodeSet(Predicate.java:446)
      at org.exist.xquery.Predicate.evalPredicate(Predicate.java:326)
      at org.exist.xquery.LocationStep.processPredicate(LocationStep.java:251)
      at org.exist.xquery.LocationStep.applyPredicate(LocationStep.java:238)
      at org.exist.xquery.LocationStep.eval(LocationStep.java:462)
      at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71)
      at org.exist.xquery.PathExpr.eval(PathExpr.java:264)
      at org.exist.xquery.LetExpr.eval(LetExpr.java:142)
      at org.exist.xquery.LetExpr.eval(LetExpr.java:187)
      at org.exist.xquery.LetExpr.eval(LetExpr.java:187)
      at org.exist.xquery.BindingExpression.eval(BindingExpression.java:164)
  2015-03-19 21:56:59,147 [Timer-0] WARN  (ThreadPoolAsynchronousRunner.java [run]:624) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@6824f21c -- APPARENT DEADLOCK!!! Complete Status: 
      Managed Threads: 3
      Active Threads: 3
      Active Tasks: 
          com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@79180a12 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
          com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@243c6d0c (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
          com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@50191373 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
      Pending Tasks: 
          com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@3a9d08ca
          com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@3ecdd11
          com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@44ff846d
          com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@5ce5850a
          com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@eec1d04
          com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@6b8d4d9d
          com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@53e9706d
          com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@23d472cf
          com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@4dbe4f8c
          com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@4c5e0203
          com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@54ac79fd
          com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@546e2bad
          com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@6b13cc83
          com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@57e185f8
          com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@60357d68
          com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@45231180
          com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@3021aa73
          com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@6bb437ca
          com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@2021c9e9
          com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@7d53637c
          com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@409c2c97
          com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@adc5929
          com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@241ca71a
          com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@42b26866
          com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@636b1c33
          com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@b160466
          com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@4af34669
          com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1b53e609
          com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@2062ebd4
          com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1b6cfe8a
          com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@4b7c2380
          com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@4f9be748
          com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@78108924
          com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@474b002
          com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@2ebee32f
          com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@3e0fe017
          com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@42aa175b
          com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@637f5bac
          com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@3a017b77
          com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@7b4f2b78
  Pool thread stack traces:
      Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
          sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
          java.lang.reflect.Constructor.newInstance(Constructor.java:408)
          com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
          com.mysql.jdbc.ResultSetImpl.getInstance(ResultSetImpl.java:370)
          com.mysql.jdbc.MysqlIO.buildResultSetWithRows(MysqlIO.java:2532)
          com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:477)
          com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2510)
          com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1746)
          com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2135)
          com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2536)
          com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2465)
          com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1383)
          com.mysql.jdbc.DatabaseMetaData$9.forEach(DatabaseMetaData.java:4826)
          com.mysql.jdbc.IterateBlock.doForAll(IterateBlock.java:50)
          com.mysql.jdbc.DatabaseMetaData.getTables(DatabaseMetaData.java:4804)
          com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnectionNoQuery(DefaultConnectionTester.java:185)
          com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:62)
          com.mchange.v2.c3p0.AbstractConnectionTester.activeCheckConnection(AbstractConnectionTester.java:67)
          com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
          com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:310)
          com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.java:1999)
          com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
      Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
          com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2596)
          com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2465)
          com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1383)
          com.mysql.jdbc.DatabaseMetaData$9.forEach(DatabaseMetaData.java:4826)
          com.mysql.jdbc.IterateBlock.doForAll(IterateBlock.java:50)
          com.mysql.jdbc.DatabaseMetaData.getTables(DatabaseMetaData.java:4804)
          com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnectionNoQuery(DefaultConnectionTester.java:185)
          com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:62)
          com.mchange.v2.c3p0.AbstractConnectionTester.activeCheckConnection(AbstractConnectionTester.java:67)
          com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
          com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:310)
          com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.java:1999)
          com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
      Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
          java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1012)
          java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
          com.newrelic.agent.TransactionService.addTransaction(TransactionService.java:142)
          com.newrelic.agent.Transaction.getTransaction(Transaction.java:1104)
          com.newrelic.agent.Transaction.getTransaction(Transaction.java:1087)
          com.newrelic.agent.TracerService$TracerServiceImpl.getTracer(TracerService.java:136)
          com.newrelic.agent.TracerService.getTracer(TracerService.java:41)
          com.newrelic.agent.instrumentation.InvocationPoint.invoke(InvocationPoint.java:55)
          com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java)
          com.mysql.jdbc.DatabaseMetaData$9.forEach(DatabaseMetaData.java:4826)
          com.mysql.jdbc.IterateBlock.doForAll(IterateBlock.java:50)
          com.mysql.jdbc.DatabaseMetaData.getTables(DatabaseMetaData.java:4804)
          com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnectionNoQuery(DefaultConnectionTester.java:185)
          com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:62)
          com.mchange.v2.c3p0.AbstractConnectionTester.activeCheckConnection(AbstractConnectionTester.java:67)
          com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
          com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:310)
          com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.java:1999)
          com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
  </code></pre>
  
  <p>We are using hiberanate and c3p0 with the following c3p0 configuration:</p>
  
  <p>We are using the following maven artifacts for hibernate and c3p0:</p>
  
  <pre><code> &lt;dependency&gt;
              &lt;groupId&gt;org.hibernate&lt;/groupId&gt;
              &lt;artifactId&gt;hibernate-core&lt;/artifactId&gt;
              &lt;version&gt;4.3.6.Final&lt;/version&gt;
          &lt;/dependency&gt;
  
          &lt;dependency&gt;
              &lt;groupId&gt;org.hibernate&lt;/groupId&gt;
              &lt;artifactId&gt;hibernate-c3p0&lt;/artifactId&gt;
              &lt;version&gt;4.3.6.Final&lt;/version&gt;
          &lt;/dependency&gt;
  </code></pre>
  
  <p>with the following c3p0 configuration:</p>
  
  <pre><code>  configuration = new Configuration().setProperty("hibernate.dialect", "org.hibernate.dialect.MySQL5InnoDBDialect")
    .setProperty("hibernate.connection.provider_class", "org.hibernate.connection.C3P0ConnectionProvider")
    .setProperty("hibernate.c3p0.idle_test_period", "1000")
    .setProperty("hibernate.c3p0.min_size", "20")
    .setProperty("hibernate.c3p0.max_size", "50")
    .setProperty("hibernate.c3p0.timeout", "1800")
    .setProperty("hibernate.c3p0.max_statements", "50")
  </code></pre>
  
  <p>The server is under very light load ~5 queries a second. (Java 8)</p>