title: Tomcat startup fails due to 'java.net.SocketException Invalid argument' on Mac OS X
body: <p>We have an application that runs on Tomcat 6 (6.0.35.0 to be precise), and most of our engineers on Mac OS are having problems starting Tomcat due to the socketAccept call in the Catalina.await method throwing a SocketException:</p>
  
  <pre><code>SEVERE: StandardServer.await: accept:
  java.net.SocketException: Invalid argument
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.socketAccept(PlainSocketImpl.java)
        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
        at java.net.ServerSocket.implAccept(ServerSocket.java:522)
        at java.net.ServerSocket.accept(ServerSocket.java:490)
        at org.apache.catalina.core.StandardServer.await(StandardServer.java:431)
        at org.apache.catalina.startup.Catalina.await(Catalina.java:676)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:628)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
        at mycompany.tomcat.startup.ThreadDumpWrapper.main(ThreadDumpWrapper.java:260)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.tanukisoftware.wrapper.WrapperStartStopApp.run(WrapperStartStopApp.java:238)
        at java.lang.Thread.run(Thread.java:722)
  </code></pre>
  
  <p>This causes Tomcat to shut down immediately after startup (and no small amount of rage). 
  We think this has been with us for the duration on Mac OS w/ Java 1.7, in the last several months a lot of us have switched to Macbook Pros. Up until now, the only symptom was occasional zero byte responses from Tomcat, due to this exception also being thrown on a socketRead. Errors don't hit the logs and we'd individually shrugged it off as an isolated problem, and only found the cause when the startup problem started and I set a SocketException breakpoint:</p>
  
  <pre><code>Daemon Thread [http-8080-1] (Suspended (breakpoint at line 47 in SocketException))  
    SocketException.&lt;init&gt;(String) line: 47 
    SocketInputStream.socketRead0(FileDescriptor, byte[], int, int, int) line: not available [native method] 
    SocketInputStream.socketRead0(FileDescriptor, byte[], int, int, int) line: not available  
    SocketInputStream.read(byte[], int, int, int) line: 150 
    SocketInputStream.read(byte[], int, int) line: 121  
    InternalInputBuffer.fill() line: 735  
    InternalInputBuffer.parseRequestLine() line: 366  
    Http11Processor.process(Socket) line: 814 
    Http11Protocol$Http11ConnectionHandler.process(Socket) line: 602  
    JIoEndpoint$Worker.run() line: 489  
    Thread.run() line: 722  
  </code></pre>
  
  <p>For arguments:</p>
  
  <pre><code>arg0  FileDescriptor  (id=499)  
    fd  1097  
    useCount  AtomicInteger  (id=503) 
      value 2 
  arg1  (id=502)
  arg2  0 
  arg3  8192  
  arg4  20000 
  </code></pre>
  
  <p>The problem is time sensitive. Increasing startup time due to application changes (lots more Spring introspection/singleton overhead) seems to be the factor that causes this to affect Tomcat startup; the tipping point being about 160 seconds. We can mitigate the problem by disabling some of the non-mandatory contexts we don't need during development to reduce startup time, but I'd prefer to find the root cause.</p>
  
  <h2>Application configuration</h2>
  
  <p>The specifics of the application are far too complex to go into too much detail, but I have a hunch that this might relate to an earlier bind, so I'll at least list the listening ports on my machine:</p>
  
  <pre><code>localhost:32000 - Java service wrapper port
  *:10001         - RMI registry
  *:2322          - Java debug
  *:56566         - RMI
  *:8180          - Tomcat HTTP connector
  *:8543          - Tomcat HTTPS connector
  *:2223          - Tomcat Internal HTTP connector (used for cross-server requests)
  *:14131         - 'Locking' port to determine if an internal service is running
  *:56571         - EhCache RMI
  *:56573         - RMI
  *:62616         - ActiveMQ broker
  *:5001          - SOAPMonitorService
  *:8109          - Tomcat shutdown port
  </code></pre>
  
  <h2>Items ruled out</h2>
  
  <ul>
  <li>The most obvious solution: <code>-Djava.net.preferIPv4Stack=true</code>. I've always had that option configured</li>
  <li>Any recent configuration change to our base application configuration, libraries, JVM options (there aren't any)</li>
  <li>A JDK regression. I've tested JDK 1.7.0_09, 11, 15, 17 and 21 (the JDKs I've had installed on my machine for the duration)</li>
  <li>Mac OS update. Mac OS 10.7.x and 10.8.0 through 1.8.3 are affected</li>
  <li>File descriptor limits - increased from <code>5000</code> to <code>10000</code></li>
  <li>Disabling IPv6 completely on the main ethernet interface</li>
  <li>Setting breakpoints, and removing the first contexts to be affected by the SocketException (they're outgoing HTTP calls to web services). No change</li>
  <li>Configuring <code>/etc/hosts</code> so the machine hostname resolves to localhost, and configuring JVM options to prefer IPv4 and to <em>not</em> prefer IPv6 addresses (This answer: <a href="https://stackoverflow.com/a/16318860/364206">https://stackoverflow.com/a/16318860/364206</a>)</li>
  </ul>
  
  <p>For those interested in hosts configuration, it's the same as default. I can reproduce this on a Fusion VM w/ a clean install of 10.8:</p>
  
  <pre><code>##
  # Host Database
  #
  # localhost is used to configure the loopback interface
  # when the system is booting.  Do not change this entry.
  ##
  127.0.0.1   localhost
  255.255.255.255 broadcasthost
  ::1             localhost
  fe80::1%lo0 localhost
  </code></pre>
  
  <h2>Java code investigation</h2>
  
  <p>Due to the apparent time sensitive nature of the issue, setting breakpoints to troubleshoot the issue causes it to not occur. As requested in the comments, I also captured <code>arg0</code> for <code>SocksSocketImpl(PlainSocketImpl).socketAccept(SocketImpl)</code>, nothing seems out of the ordinary. </p>
  
  <pre><code>arg0  SocksSocketImpl  (id=460) 
    address InetAddress  (id=465) 
      canonicalHostName null  
      holder  InetAddress$InetAddressHolder  (id=475) 
        address 0 
        family  0 
        hostName  null  
    applicationSetProxy false 
    closePending  false 
    cmdIn null  
    cmdOut  null  
    cmdsock null  
    CONNECTION_NOT_RESET  0 
    CONNECTION_RESET  2 
    CONNECTION_RESET_PENDING  1 
    external_address  null  
    fd  FileDescriptor  (id=713)  
      fd  -1  
      useCount  AtomicInteger  (id=771) 
        value 0 
    fdLock  Object  (id=714)  
    fdUseCount  0 
    localport 0 
    port  0 
    resetLock Object  (id=716)  
    resetState  0 
    server  null  
    serverPort  1080  
    serverSocket  null  
    shut_rd false 
    shut_wr false 
    socket  Socket  (id=718)  
      bound false 
      closed  false 
      closeLock Object  (id=848)  
      connected false 
      created false 
      impl  null  
      oldImpl false 
      shutIn  false 
      shutOut false 
    socketInputStream null  
    stream  false 
    timeout 0 
    trafficClass  0 
    useV4 false 
  </code></pre>
  
  <p>I think all of the threads where the exceptions are thrown are victims of an earlier call, one that doesn't result in a SocketException so I haven't been able to catch it. Being able to start Tomcat by reducing startup times convinces me that the trigger is probably some scheduled task that performs a socket based operation, which then affects other socket operations.</p>
  
  <p>That doesn't explain how and why this could affect several threads, no matter what we're doing to cause this condition, mysterious SocketExceptions shouldn't bubble up from native code and cause these exceptions simultaneously on multiple threads - that is, two threads making outgoing web service calls, the Tomcat await call, and several TP processor threads repeatedly.</p>
  
  <h2>JNI code investigation</h2>
  
  <p>Given the generic message, I assumed that an <code>EINVAL</code> error must be returned from one of the system calls in the socketAccept JNI code, so I traced the system calls leading up to the exception; there's no <code>EINVAL</code> returned from any system call. So, I went to the OpenJDK sources looking for conditions in the socketAccept code that would set and then throw an <code>EINVAL</code>, but I also couldn't find any code that sets <code>errno</code> to <code>EINVAL</code>, or calls <code>NET_ThrowByNameWithLastError</code>, <code>NET_ThrowCurrent</code> or <code>NET_ThrowNew</code> in a way that would throw a SocketException with this default error message. </p>
  
  <p>As far as the system calls, we don't seem to get as far as the accept system call:</p>
  
  <pre><code> PID/THRD        RELATIVE   ELAPSD    CPU SYSCALL(args)    = return
   6606/0x2c750d:  221538243       5      0 sigprocmask(0x1, 0x0, 0x14D8BE100)    = 0x0 0
   6606/0x2c750d:  221538244       3      0 sigaltstack(0x0, 0x14D8BE0F0, 0x0)     = 0 0
   6606/0x2c750d:  221538836      14     10 socket(0x2, 0x1, 0x0)    = 1170 0
   6606/0x2c750d:  221538837       3      0 fcntl(0x492, 0x3, 0x4)     = 2 0
   6606/0x2c750d:  221538839       3      1 fcntl(0x492, 0x4, 0x6)     = 0 0
   6606/0x2c750d:  221538842       5      2 setsockopt(0x492, 0xFFFF, 0x4)     = 0 0
   6606/0x2c750d:  221538852       7      4 bind(0x492, 0x14D8BE5D8, 0x10)     = 0 0
   6606/0x2c750d:  221538857       5      2 listen(0x492, 0x1, 0x4)    = 0 0
   6606/0x2c750d:  221539625       6      2 psynch_cvsignal(0x7FEFBFE00868, 0x10000000200, 0x100)    = 257 0
   6606/0x2c750d:  221539633       4      1 write(0x2, "Apr 18, 2013 11:05:35 AM org.apache.catalina.core.StandardServer await\nSEVERE: StandardServer.await: accept: \njava.net.SocketException: Invalid argument\n\tat java.net.PlainSocketImpl.socketAccept(Native Method)\n\tat java.net.PlainSocketImpl.socketAcce", 0x644)    = 1604 0
  </code></pre>
  
  <p>So, I <em>think</em> the problem occurs in the timeout handling code at the top of the accept loop in <code>socketAccept</code>, but I couldn't find any case where <code>NET_Timeout</code> would set <code>errno</code> to <code>EINVAL</code>, and result in this SocketException being thrown. I'm referring to this code; I assume the jdk7u branch is for the most part what ships in the Oracle JDK:</p>
  
  <ul>
  <li><a href="http://hg.openjdk.java.net/jdk7u/jdk7u/jdk/file/d4bf5c15837c/src/solaris/native/java/net/PlainSocketImpl.c" rel="nofollow noreferrer">http://hg.openjdk.java.net/jdk7u/jdk7u/jdk/file/d4bf5c15837c/src/solaris/native/java/net/PlainSocketImpl.c</a></li>
  <li><a href="http://hg.openjdk.java.net/jdk7u/jdk7u/jdk/file/d4bf5c15837c/src/solaris/native/java/net/bsd_close.c" rel="nofollow noreferrer">http://hg.openjdk.java.net/jdk7u/jdk7u/jdk/file/d4bf5c15837c/src/solaris/native/java/net/bsd_close.c</a></li>
  <li><a href="http://hg.openjdk.java.net/jdk7u/jdk7u/jdk/file/d4bf5c15837c/src/solaris/native/java/net/net_util_md.c" rel="nofollow noreferrer">http://hg.openjdk.java.net/jdk7u/jdk7u/jdk/file/d4bf5c15837c/src/solaris/native/java/net/net_util_md.c</a></li>
  </ul>
  
  <h2>Help!</h2>
  
  <p>I can't find anyone in the outside world affected by this particular problem on Mac OS, but almost everyone here is affected. There must be some application configuration that contributes, but I've exhausted every avenue I can think of to find the root cause.</p>
  
  <p>Pointers on troubleshooting or insight on a possible cause would be much appreciated.</p>