title: HikariCP connection pools hangs in spring-batch 4.0
body: <p>We have updated our Spring Batch based application to the newest version  of Spring Batch 4.x and Boot 2.0.x and having some problems with the HikariCP MySQL connection pool. </p>
  
  <p>If I use the default maxPoolSize of 10 connections and I'm using 5 threads which each starts a simple dummy job this works ok. (Using SimpleJobLauncher). All Jobs are started immediately and after that the HikariCP pool has all used connections returned.</p>
  
  <p>The problem starts when I do the same but then with 11 threads. Then I see that the application hangs until a connection timeout, and then the jobs are started. I expected that the first 10 jobs will start and after the first one finished the 11th job...</p>
  
  <p>Before the jobs start I see this:</p>
  
  <pre><code>HikariPool-1 - Pool stats (total=10, active=0, idle=10, waiting=0)
  </code></pre>
  
  <p>Now I start 11 jobs and the application will hang for 30 seconds...</p>
  
  <pre><code>2019-01-21 11:25:01.256  INFO 10194 --- [ver (CliServer)] c.n.r.i.batch.cli.CliClientHandler       : Start 11 jobs
  2019-01-21 11:25:30.928 DEBUG 10194 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Pool stats (total=10, active=10, idle=0, waiting=2)
  2019-01-21 11:25:35.990 DEBUG 10194 --- [      Thread-10] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Timeout failure stats (total=10, active=10, idle=0, waiting=1)
  2019-01-21 11:25:35.992 DEBUG 10194 --- [       Thread-6] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Timeout failure stats (total=10, active=10, idle=0, waiting=1)
  org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
      at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:305)
      at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:378)
      at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:474)
      at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:289)
      at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
      at org.springframework.batch.core.repository.support.AbstractJobRepositoryFactoryBean$1.invoke(AbstractJobRepositoryFactoryBean.java:181)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
      at com.sun.proxy.$Proxy74.createJobExecution(Unknown Source)
      at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:131)
      at com.xx.yy.zz.batch.cli.commands.DummyCliCommand$1.run(DummyCliCommand.java:54)
      at java.lang.Thread.run(Thread.java:748)
  Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
      at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:697)
      at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:196)
      at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:161)
      at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:128)
      at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:262)
      ... 12 more
  </code></pre>
  
  <p>And after this timeout the first jobs are started...</p>
  
  <pre><code>2019-01-21 11:25:36.061  INFO 10194 --- [      Thread-11] c.n.r.i.b.u.c.BlockingThreadPoolExecutor : [sftp-executor] 3 running jobs, 0 queued jobs, 2000 jobs allowed
  2019-01-21 11:25:36.062  INFO 10194 --- [       Thread-5] c.n.r.i.b.u.c.BlockingThreadPoolExecutor : [sftp-executor] 3 running jobs, 0 queued jobs, 2000 jobs allowed
  2019-01-21 11:25:36.061  INFO 10194 --- [       Thread-3] c.n.r.i.b.u.c.BlockingThreadPoolExecutor : [sftp-executor] 3 running jobs, 0 queued jobs, 2000 jobs allowed
  2019-01-21 11:25:36.063  INFO 10194 --- [pool-1-thread-2] o.s.b.c.l.support.SimpleJobLauncher      : Job: [SimpleJob: [name=dummyJob]] launched with the following parameters: [{dummy=f5c864d9-8a74-4d51-9a53-e3fc5c0b731d}]
  </code></pre>
  
  <p>Version:
  - HikariCP 2.7.9
  - spring-batch-core 4.0.1
  - Hibernate Core 5.2.17.Final
  - spring-boot 2.0.5</p>
  
  <p>I don't understand why the pool hangs? (The Dummy Jobs do only sleep a second)</p>
  
  <h3>HikariCP config</h3>
  
  <pre><code>allowPoolSuspension.............false
  autoCommit......................true
  catalog.........................none
  connectionInitSql...............none
  connectionTestQuery.............none
  connectionTimeout...............30000
  dataSource......................none
  dataSourceClassName.............none
  dataSourceJNDI..................none
  dataSourceProperties............{usePipelineAuth=false, password=&lt;masked&gt;, prepStmtCacheSqlLimit=2048, rewriteBatchedStatements=false, useSSL=false, cachePrepStmts=true, useServerPrepStmts=true, prepStmtCacheSize=250}
  driverClassName................."com.mysql.jdbc.Driver"
  healthCheckProperties...........{}
  healthCheckRegistry.............none
  idleTimeout.....................600000
  initializationFailTimeout.......1
  isolateInternalQueries..........false
  jdbcUrl.........................jdbc:mysql://localhost:3306/development
  leakDetectionThreshold..........60000
  maxLifetime.....................1800000
  maximumPoolSize.................10
  metricRegistry..................none
  metricsTrackerFactory...........none
  minimumIdle.....................10
  password........................&lt;masked&gt;
  poolName........................"HikariPool-1"
  readOnly........................false
  registerMbeans..................true
  scheduledExecutor...............none
  schema..........................none
  threadFactory...................internal
  transactionIsolation............default
  username........................"user"
  validationTimeout...............5000
  </code></pre>