title: Weblogic EJB calls start to fail under moderate load with OptionalDataException
body: <p>Our system setup consists of two Weblogic 10.3 servers: one hosts the presentation layer and the other hosts the EJBs. The system runs fine under moderate load for some time (one to several days) after which EJB method calls from the presentation server to the EJB server start to fail with the following error:</p>
  
  <pre><code>java.rmi.RemoteException: java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is: java.io.OptionalDataException
  </code></pre>
  
  <p>Stack trace:</p>
  
  <pre><code>java.io.OptionalDataException
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
      at weblogic.utils.io.ChunkedObjectInputStream.readObject(ChunkedObjectInputStream.java:197)
      at weblogic.rjvm.MsgAbbrevInputStream.readObject(MsgAbbrevInputStream.java:564)
      at weblogic.utils.io.ChunkedObjectInputStream.readObject(ChunkedObjectInputStream.java:193)
      at weblogic.jndi.internal.RootNamingNode_WLSkel.invoke(Unknown Source)
      at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:589)
      at weblogic.rmi.cluster.ClusterableServerRef.invoke(ClusterableServerRef.java:230)
      at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:477)
      at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
      at weblogic.security.service.SecurityManager.runAs(Unknown Source)
      at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:473)
      at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118)
  </code></pre>
  
  <p>Once the first OptionalDataException is encountered all subsequent calls fail with the same result. Some sources suggest that this might be related to cluster multicast port being misconfigured. However, these servers do not belong to a cluster.</p>
  
  <p>Booting the EJB server always temporarily resolves the issue, but the issue seems to occur again after some time.</p>
  
  <p><em>Update</em>: it seems that the problem is <em>not</em> related to an overflow in the number of socket connections after all (see my own answer below). After disallowing network classloading we ran very steadily for a week after which we started receiving OptionalDataExceptions on the presentation server again (stack trace below). It is very strange that the system works fine for a week and then starts to fail.</p>
  
  <pre><code>javax.naming.CommunicationException [Root exception is java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
      java.io.OptionalDataException]
      at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:74)
      at weblogic.jndi.internal.WLContextImpl.translateException(WLContextImpl.java:439)
      at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:395)
      at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:380)
      at javax.naming.InitialContext.lookup(InitialContext.java:392)
      ...
  Caused by: java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:
  
      java.io.OptionalDataException
      at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:234)
      at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:348)
      at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:259)
      at weblogic.jndi.internal.ServerNamingNode_1030_WLStub.lookup(Unknown Source)
      at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:392)  
      ... 38 more
  Caused by: java.io.OptionalDataException
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
      at     
      weblogic.utils.io.ChunkedObjectInputStream.readObject(ChunkedObjectInputStream.java:197)
      at weblogic.rjvm.MsgAbbrevInputStream.readObject(MsgAbbrevInputStream.java:564)
      at     
  weblogic.utils.io.ChunkedObjectInputStream.readObject(ChunkedObjectInputStream.java:193)
      at weblogic.jndi.internal.RootNamingNode_WLSkel.invoke(Unknown Source)
      at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:589)
      at weblogic.rmi.cluster.ClusterableServerRef.invoke(ClusterableServerRef.java:230)
      at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:477)
      at        
  weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
      at weblogic.security.service.SecurityManager.runAs(Unknown Source)
      at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:473)
      at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118)
      ... 2 more
  </code></pre>
  
  <p>We obtain the initial context quite the standard way:</p>
  
  <pre><code>Properties p = new Properties();
  p.put(Context.INITIAL_CONTEXT_FACTORY, "weblogic.jndi.WLInitialContextFactory");
  p.put(Context.PROVIDER_URL, serverPath);
  Context context = new InitialContext(p);
  </code></pre>
  
  <p>Also calls to any obtained references fail with a similar OptionalDataException. Booting the presentation server alone resolves the issue temporarily.</p>