title: Why does the Java VM not recover after "Too many open files" errors?
body: <p>In certain well-understood circumstances, our application will open too many sockets (database connections) and reach the maximum open files that the OS allows. We understand this; we are fixing the issue and also bumping up the limit.</p>
  
  <p>What we can't explain is why parts of our application don't recover even after the number of connections abates and we're well within the limit.</p>
  
  <p>In this case, it's an application running under Tomcat. </p>
  
  <p>When this happens, we first start seeing "Too many open files" errors:</p>
  
  <pre><code>SEVERE: Socket accept failed
  java.net.SocketException: Too many open files
          at java.net.PlainSocketImpl.socketAccept(Native Method)
          at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
          at java.net.ServerSocket.implAccept(ServerSocket.java:453)
          at java.net.ServerSocket.accept(ServerSocket.java:421)
          at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61)
          at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:310)
          at java.lang.Thread.run(Thread.java:619)
  </code></pre>
  
  <p>Eventually, we start seeing <code>NoClassDefFoundError</code>s inside an application thread that's trying to open HTTP connections:</p>
  
  <pre><code>java.lang.NoClassDefFoundError: org/apache/commons/httpclient/protocol/ControllerThreadSocketFactory
          at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:128)
          at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
          at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1349)
         [...]
  Caused by: java.lang.ClassNotFoundException: org.apache.commons.httpclient.protocol.ControllerThreadSocketFactory
          at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1387)
          at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1233)
          at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
          ... 8 more
  </code></pre>
  
  <p>When the errant connections go away, the server starts accepting connections again, and everything seems ok, but we're left with the latter error constantly being spewed to stderr. </p>
  
  <p>Although the application typically logs unloaded classes to stdout, I don't see any such logs just before, during or after the "Too many open files" errors. </p>
  
  <p>My initial theory was that the Hotspot JVM would unload seemingly unused classes when it encounters "Too many open files," but if so, it doesn't log the fact. </p>
  
  <p><strong>Edit:</strong> As Stephen C indicates below, if it is unloading the class, and encounters an error the first time it reloads, that could explain why it never recovers. I think that's a good working theory. Is it documented in the Sun docs? Why would it not log that the class is being unloaded the way unloading a class usually is?</p>
  
  <p>Platform details:</p>
  
  <pre><code>Java(TM) SE Runtime Environment (build 1.6.0_14-b08)
  Java HotSpot(TM) 64-Bit Server VM (build 14.0-b16, mixed mode)
  
  Apache Tomcat Version 6.0.18
  </code></pre>