title: Jboss XA Datasource with Oracle Can't Open Connection
body: <p>We have strange problem with Jboss Wildfly 8.1 with XA Datasource using Oracle database. </p>
  
  <p>After some time we get "Cannot open connection" exception, while we can see 100+ inactive connection in pool but Jboss Datasource is not using any of them from pool but giving exception. </p>
  
  <p>Following are details. </p>
  
  <p><strong>Oracle</strong> Database 11g Enterprise Edition Release 11.2.0.1.0 - Production With the Partitioning, OLAP, Data Mining and Real Application Testing options.</p>
  
  <pre><code>&lt;subsystem xmlns="urn:jboss:domain:datasources:2.0"&gt;
              &lt;datasources&gt;
                  &lt;xa-datasource jndi-name="java:/XAOracleDS" pool-name="XAOracleDS" enabled="true" use-java-context="true" use-ccm="true"&gt;
  
                      &lt;xa-datasource-property name="URL"&gt;
                          jdbc:oracle:thin:@10.0.1.55:1521:sid
                      &lt;/xa-datasource-property&gt;
  
                      &lt;driver&gt;oracle&lt;/driver&gt;
  
                      &lt;transaction-isolation&gt;TRANSACTION_READ_COMMITTED&lt;/transaction-isolation&gt;
  
                     &lt;xa-pool&gt;
                          &lt;min-pool-size&gt;10&lt;/min-pool-size&gt;
                          &lt;max-pool-size&gt;50&lt;/max-pool-size&gt;
                          &lt;prefill&gt;true&lt;/prefill&gt;
                          &lt;use-strict-min&gt;false&lt;/use-strict-min&gt;
                          &lt;flush-strategy&gt;FailingConnectionOnly&lt;/flush-strategy&gt;
                          &lt;pad-xid&gt;true&lt;/pad-xid&gt;
                          &lt;wrap-xa-resource&gt;true&lt;/wrap-xa-resource&gt;
                      &lt;/xa-pool&gt;
  
                     &lt;security&gt;
                          &lt;user-name&gt;scott&lt;/user-name&gt;
                          &lt;password&gt;tigger&lt;/password&gt;
                      &lt;/security&gt;
  
                      &lt;validation&gt;
                          &lt;check-valid-connection-sql&gt;select 1 from dual&lt;/check-valid-connection-sql&gt;
                          &lt;background-validation&gt;true&lt;/background-validation&gt;
                          &lt;use-fast-fail&gt;false&lt;/use-fast-fail&gt;
                          &lt;exception-sorter class-name="org.jboss.jca.adapters.jdbc.extensions.oracle.OracleExceptionSorter"/&gt;
                      &lt;/validation&gt;
  
                      &lt;timeout&gt;
                          &lt;blocking-timeout-millis&gt;120000&lt;/blocking-timeout-millis&gt;
                          &lt;idle-timeout-minutes&gt;5&lt;/idle-timeout-minutes&gt;
                          &lt;allocation-retry&gt;3&lt;/allocation-retry&gt;
                          &lt;allocation-retry-wait-millis&gt;1000&lt;/allocation-retry-wait-millis&gt;
                      &lt;/timeout&gt;
  
                      &lt;statement&gt;
                          &lt;prepared-statement-cache-size&gt;50&lt;/prepared-statement-cache-size&gt;
                      &lt;/statement&gt;
  
                  &lt;/xa-datasource&gt;
  
                  &lt;drivers&gt;
                      &lt;driver name="oracle" module="com.oracle.ojdbc6"&gt;
                          &lt;xa-datasource-class&gt;oracle.jdbc.xa.client.OracleXADataSource&lt;/xa-datasource-class&gt;
                      &lt;/driver&gt;
                  &lt;/drivers&gt;
  
              &lt;/datasources&gt;
          &lt;/subsystem&gt;
  </code></pre>
  
  <p>Linux version 2.6.32-431.20.3.el6.x86_64 (mockbuild@c6b9.bsys.dev.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC) ) #1 SMP Thu Jun 19 21:14:45 UTC 2014</p>
  
  <p>JDK Version: jdk1.7.0_65</p>
  
  <p><strong>Exception Stack trace:</strong> </p>
  
  <pre><code> ERROR [org.springframework.scheduling.quartz.SchedulerFactoryBean#1_Worker-1] | Hibernate operation: Cannot open connection; uncategorized SQLException for SQL [???]; SQL state [null]; error code [0]; javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@506f65ce[state=DESTROYED managed connection=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@46c87b4b connection handles=0 lastUse=1410399087408 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@6b8913c mcp=SemaphoreArrayListManagedConnectionPool@984da7b[pool=XAOracleDS] xaResource=XAResourceWrapperImpl@10cab9b8[xaResource=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@46c87b4b pad=true overrideRmValue=null productName=Oracle productVersion=Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
   With the Partitioning, OLAP, Data Mining and Real Application Testing options jndiName=java:/XAOracleDS] txSync=null]; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@506f65ce[state=DESTROYED managed connection=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@46c87b4b connection handles=0 lastUse=1410399087408 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@6b8913c mcp=SemaphoreArrayListManagedConnectionPool@984da7b[pool=XAOracleDS] xaResource=XAResourceWrapperImpl@10cab9b8[xaResource=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@46c87b4b pad=true overrideRmValue=null productName=Oracle productVersion=Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
   With the Partitioning, OLAP, Data Mining and Real Application Testing options jndiName=java:/XAOracleDS] txSync=null]
   org.springframework.jdbc.UncategorizedSQLException: Hibernate operation: Cannot open connection; uncategorized SQLException for SQL [???]; SQL state [null]; error code [0]; javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@506f65ce[state=DESTROYED managed connection=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@46c87b4b connection handles=0 lastUse=1410399087408 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@6b8913c mcp=SemaphoreArrayListManagedConnectionPool@984da7b[pool=XAOracleDS] xaResource=XAResourceWrapperImpl@10cab9b8[xaResource=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@46c87b4b pad=true overrideRmValue=null productName=Oracle productVersion=Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
   With the Partitioning, OLAP, Data Mining and Real Application Testing options jndiName=java:/XAOracleDS] txSync=null]; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@506f65ce[state=DESTROYED managed connection=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@46c87b4b connection handles=0 lastUse=1410399087408 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@6b8913c mcp=SemaphoreArrayListManagedConnectionPool@984da7b[pool=XAOracleDS] xaResource=XAResourceWrapperImpl@10cab9b8[xaResource=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@46c87b4b pad=true overrideRmValue=null productName=Oracle productVersion=Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
   With the Partitioning, OLAP, Data Mining and Real Application Testing options jndiName=java:/XAOracleDS] txSync=null]
      at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:83)
      at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80)
      at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80)
      at org.springframework.orm.hibernate3.HibernateAccessor.convertJdbcAccessException(HibernateAccessor.java:424)
      at org.springframework.orm.hibernate3.HibernateAccessor.convertHibernateAccessException(HibernateAccessor.java:410)
      at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:411)
      at org.springframework.orm.hibernate3.HibernateTemplate.executeWithNativeSession(HibernateTemplate.java:374)
      at org.springframework.orm.hibernate3.HibernateTemplate.saveOrUpdate(HibernateTemplate.java:737)
      at com.inov8.framework.server.dao.framework.hibernate.BaseHibernateDAO.saveOrUpdate(BaseHibernateDAO.java:590)
      at com.inov8.microbank.server.service.transactionmodule.TransactionManagerImpl.saveTransactionCodeModel(TransactionManagerImpl.java:144)
      at com.inov8.microbank.server.service.transactionmodule.TransactionManagerImpl.generateTransactionCodeRequiresNewTransaction(TransactionManagerImpl.java:99)
      at sun.reflect.GeneratedMethodAccessor1031.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
      at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
      at com.sun.proxy.$Proxy59.generateTransactionCodeRequiresNewTransaction(Unknown Source)
      at com.inov8.microbank.server.service.bulkdisbursements.BulkDisbursementsManagerImpl.creditOLABulkDisbursementSundryAccount(BulkDisbursementsManagerImpl.java:555)
      at com.inov8.microbank.server.service.bulkdisbursements.BulkDisbursementsManagerImpl.makeDebitCreditAccount(BulkDisbursementsManagerImpl.java:143)
      at sun.reflect.GeneratedMethodAccessor1030.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
      at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
      at com.sun.proxy.$Proxy136.makeDebitCreditAccount(Unknown Source)
      at com.inov8.microbank.server.facade.BulkDisbursementsFacadeImpl.makeDebitCreditAccount(BulkDisbursementsFacadeImpl.java:67)
      at com.inov8.microbank.server.service.dailyjob.BulkDisbursementsScheduler.activateDailyBulkDisbursements(BulkDisbursementsScheduler.java:178)
      at com.inov8.microbank.server.service.dailyjob.BulkDisbursementsScheduler.init(BulkDisbursementsScheduler.java:84)
      at com.inov8.microbank.server.service.dailyjob.BulkDisbursementsScheduler$$FastClassByCGLIB$$e344897e.invoke(&lt;generated&gt;)
      at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)
      at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:618)
      at com.inov8.microbank.server.service.dailyjob.BulkDisbursementsScheduler$$EnhancerByCGLIB$$a63ce042.init(&lt;generated&gt;)
      at sun.reflect.GeneratedMethodAccessor1029.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:273)
      at org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:311)
      at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:113)
      at org.quartz.core.JobRunShell.run(JobRunShell.java:223)
      at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
   Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@506f65ce[state=DESTROYED managed connection=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@46c87b4b connection handles=0 lastUse=1410399087408 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@6b8913c mcp=SemaphoreArrayListManagedConnectionPool@984da7b[pool=XAOracleDS] xaResource=XAResourceWrapperImpl@10cab9b8[xaResource=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@46c87b4b pad=true overrideRmValue=null productName=Oracle productVersion=Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
   With the Partitioning, OLAP, Data Mining and Real Application Testing options jndiName=java:/XAOracleDS] txSync=null]
      at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:154)
      at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:81)
      at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423)
      at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
      at org.hibernate.jdbc.AbstractBatcher.prepareSelectStatement(AbstractBatcher.java:123)
      at org.hibernate.id.SequenceGenerator.generate(SequenceGenerator.java:73)
      at org.hibernate.id.SequenceHiLoGenerator.generate(SequenceHiLoGenerator.java:53)
      at org.hibernate.event.def.AbstractSaveEventListener.saveWithGeneratedId(AbstractSaveEventListener.java:99)
      at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.saveWithGeneratedOrRequestedId(DefaultSaveOrUpdateEventListener.java:187)
      at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.entityIsTransient(DefaultSaveOrUpdateEventListener.java:172)
      at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.performSaveOrUpdate(DefaultSaveOrUpdateEventListener.java:94)
      at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.onSaveOrUpdate(DefaultSaveOrUpdateEventListener.java:70)
      at org.hibernate.impl.SessionImpl.fireSaveOrUpdate(SessionImpl.java:507)
      at org.hibernate.impl.SessionImpl.saveOrUpdate(SessionImpl.java:499)
      at org.hibernate.impl.SessionImpl.saveOrUpdate(SessionImpl.java:495)
      at org.springframework.orm.hibernate3.HibernateTemplate$16.doInHibernate(HibernateTemplate.java:740)
      at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:406)
      ... 42 more
   Caused by: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@506f65ce[state=DESTROYED managed connection=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@46c87b4b connection handles=0 lastUse=1410399087408 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@6b8913c mcp=SemaphoreArrayListManagedConnectionPool@984da7b[pool=XAOracleDS] xaResource=XAResourceWrapperImpl@10cab9b8[xaResource=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@46c87b4b pad=true overrideRmValue=null productName=Oracle productVersion=Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
   With the Partitioning, OLAP, Data Mining and Real Application Testing options jndiName=java:/XAOracleDS] txSync=null]
      at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractConnectionManager.java:773)
      at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:516)
      at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:146)
      ... 58 more
   Caused by: javax.resource.ResourceException: IJ000461: Could not enlist in transaction on entering meta-aware object
      at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.managedConnectionReconnected(TxConnectionManagerImpl.java:546)
      at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractConnectionManager.java:768)
      ... 60 more
   Caused by: javax.transaction.SystemException: IJ000356: Failed to enlist: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple &lt; ac, BasicAction: 0:ffff7f000001:751c2dee:54107a6d:1db4a7 status: ActionStatus.ABORT_ONLY &gt;
      at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener$TransactionSynchronization.checkEnlisted(TxConnectionListener.java:812)
      at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener.enlist(TxConnectionListener.java:362)
      at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.managedConnectionReconnected(TxConnectionManagerImpl.java:539)
      ... 61 more
  ramework.scheduling.quartz.SchedulerFactoryBean#1_Worker-1) com.inov8.framework.common.exception.FrameworkCheckedException: Hibernate operation: Cannot open connection; uncategorized SQLException for SQL [???]; SQL state [null]; error code [0]; javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@506f65ce[state=DESTROYED managed connection=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@46c87b4b connection handles=0 lastUse=1410399087408 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@6b8913c mcp=SemaphoreArrayListManagedConnectionPool@984da7b[pool=XAOracleDS] xaResource=XAResourceWrapperImpl@10cab9b8[xaResource=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@46c87b4b pad=true overrideRmValue=null productName=Oracle productVersion=Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
  c=null]; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@506f65ce[state=DESTROYED managed connection=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@46c87b4b connection handles=0 lastUse=1410399087408 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@6b8913c mcp=SemaphoreArrayListManagedConnectionPool@984da7b[pool=XAOracleDS] xaResource=XAResourceWrapperImpl@10cab9b8[xaResource=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@46c87b4b pad=true overrideRmValue=null productName=Oracle productVersion=Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
  c=null]
  </code></pre>
  
  <p>Any expert opinion regarding this issue?</p>