title: Tomcat JDBC pool connections all stuck at Socket Read, and no new connection is created
body: <p>I have a Tomcat 7 application hosted on Amazon ElasticBeanstalk, and a MySQL 5.5 database hosted on Amazon RDS. The database server serves only one Tomcat application and has max connection limit set to 10,000.</p>
  
  <p>However strange things are happening to the DB connections after few hours of uptime.</p>
  
  <p>MySQL server reports that only 3 connections were created by Tomcat JDBC connection pool, they were all "sleeping" (example output):</p>
  
  <pre><code>| 228 | root     | ip-10-240-xx-xxx.ap-southeast-2.compute.internal:33270 | xxxxx | Sleep   |   13 |       | NULL             |
  </code></pre>
  
  <p>Tomcat thread dump suggested that all 3 connections are reading (blocking) from network IO sockets:</p>
  
  <pre><code>at java.net.SocketInputStream.socketRead0(Native Method)
          at java.net.SocketInputStream.read(SocketInputStream.java:150)
          at java.net.SocketInputStream.read(SocketInputStream.java:121)
          at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:114)
          at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:161)
          at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:189)
          - locked &lt;0x00000000bc349cc0&gt; (a com.mysql.jdbc.util.ReadAheadInputStream)
          at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3036)
  </code></pre>
  
  <p>Because there are only 3 DB connections in connection pool and they are all stuck on IO read, Tomcat refuses to serve any more HTTP requests that require DB data.</p>
  
  <p>Tomcat JDBC pool settings have assigned:</p>
  
  <pre><code>maxActive="500"
  maxIdle="100"
  minIdle="50"
  initialSize="50"
  maxWait="15000"
  timeBetweenEvictionRunsMillis="10000"
  minEvictableIdleTimeMillis="30000"
  removeAbandoned="true"
  removeAbandonedTimeout="120000"
  logAbandoned="true"
  testOnBorrow="true"
  testWhileIdle="true"
  validationQuery="select 1"
  driverClassName="com.mysql.jdbc.Driver"
  factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
  </code></pre>
  
  <p>Still unsure why all of 3 DB connections got stuck at network IO read, however given the configuration above, I would expect Tomcat to create additional connections when all existing connections are busy.</p>
  
  <p>I have 8 other Beanstalk applications using identical setup, however only this particular application has such strange pooling behaviour and network IO problem.</p>
  
  <p>What do you suggest?</p>
  
  <p>Thank you very much.</p>