title: Tomcat 7 JDBC pool exhausted during failover and won't recover
body: <p>Tomcat 7.0.47
  MySQL Connector/J 5.1.20</p>
  
  <p>I'm testing an HA database setup where a mysql DB (RDS in AWS) is accessible by DNS name. Upon failover the DB will get a new IP.</p>
  
  <p>I have a test web app with the following database connection defined in context.xml:</p>
  
  <pre><code>&lt;Resource auth="Container" driverClassName="com.mysql.jdbc.Driver"
    maxActive="3" maxIdle="2" maxWait="1000" name="jdbc/db"
    password=“xxx” type="javax.sql.DataSource"
    url="jdbc:mysql://xxx:3306/xxx?autoReconnect=true" username="root"
    factory="org.apache.tomcat.jdbc.pool.DataSourceFactory" 
    removeAbandoned="true" logAbandoned="true" removeAbandonedTimeout="60"
  /&gt;
  </code></pre>
  
  <p>As you can see there are a very small number of connections defined so this problem can more easily be reproduced. I have a web client that is making an AJAX REST call every second, this call simply makes a single select and a single insert into the DB. Web framework is Spring MVC using Spring JDBC.</p>
  
  <p>I spin up enough clients to max out the concurrent DB usage and then I fail over the database. the database recovers and DNS fails over to the new IP. The behavior I'm seeing in Tomcat is that the connections are never released or abandoned. </p>
  
  <pre><code>SEVERE: Servlet.service() for servlet [appServlet] in context with path [/test] threw exception [Request processing failed; nested exception
    is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; 
    nested exception is org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-8080-exec-11] Timeout: Pool empty. Unable to fetch a connection in 1 seconds, none available[size:3; busy:2; idle:0; lastwait:1000].] with root cause
    org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-8080-exec-11] Timeout: Pool empty. Unable to fetch a connection in 1 seconds, none available[size:3; busy:2; idle:0; lastwait:1000].
  at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:674)
  </code></pre>
  
  <p>... and the app never recovers. I'm looking for a way for Tomcat to recover these connections without having to restart Tomcat. Surely there must be away for Tomcat to recognize that these connections are dead and eject them from the pool? I believe that's the intent of removeAbandoned="true" but it doesn't seem to be helping here.</p>
  
  <p>Any ideas?</p>
  
  <p>--- UPDATE ---</p>
  
  <p>In a debugger I've discovered the PoolCleaner thread that cleans up abandoned connections is hung up on JDBC4Connection(ConnectionImpl).close():1641. The PoolCleaner stopped and connections aren't getting cleaned up. Not sure why close() is timing out...</p>