title: High response times for rmi calls to weblogic
body: <p>We're integrating towards a weblogic server using RMI. We're running the client in a webapp on a tomcat server. We're having weird issues with the response times. Some of the request will respond in ~20ms, but some will take 10-12 seconds. There are no responses in between ~20ms and 10-12 seconds. </p>
  
  <p>A few things are worth mentioning</p>
  
  <ol>
  <li><p>We are using wlfullclient 10.3.2. I had to remove some classes in it to be able to make it run on Tomcat. This was a package with Servlet.class. Ugly, I know, but I wasn't able to take my webapp work with wlthinclient or other packages. Also, we are using a custom rmi client made by the middleware guys and it works well on weblogic server we have been running for ages.</p></li>
  <li><p>When a call takes 10-12 seconds, it seems as if there is no tcp traffic before until the very end of this time span. I have only monitored port 7461 here.</p></li>
  <li><p>I have the host names in my /etc/hosts file so it shouldn't be a DNS problem</p></li>
  <li><p>I am experiencing this only on our test/prod server, not when I run the webapp on Tomcat on my work station</p></li>
  <li><p>From debugging the issue it seems that the hang happens in weblogic.security.Security.runAs(). </p></li>
  </ol>
  
  <p>Edit: 
  I dumped the thread, and it seems I have some kind of locking issue with the socket.</p>
  
  <pre><code>"http-bio-8082-exec-45" daemon prio=10 tid=0x0000000002a96000 nid=0x318c runnable [0x00007f09ca8e3000]
     java.lang.Thread.State: RUNNABLE
      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
      - locked &lt;0x000000076586d0e0&gt; (a java.net.SocksSocketImpl)
      at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
      at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
      at java.net.Socket.connect(Socket.java:579)
      at weblogic.socket.SocketMuxer.newSocket(SocketMuxer.java:342)
      at weblogic.socket.ChannelSocketFactory.createSocket(ChannelSocketFactory.java:79)
      at weblogic.socket.BaseAbstractMuxableSocket.createSocket(BaseAbstractMuxableSocket.java:133)
      at weblogic.rjvm.t3.MuxableSocketT3.newSocketWithRetry(MuxableSocketT3.java:206)
      at weblogic.rjvm.t3.MuxableSocketT3.connect(MuxableSocketT3.java:375)
      at weblogic.rjvm.t3.ConnectionFactoryT3.createConnection(ConnectionFactoryT3.java:33)
      at weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1773)
      at weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1416)
      - locked &lt;0x000000076586c7f8&gt; (a weblogic.rjvm.ConnectionManagerClient)
      - locked &lt;0x000000076586c7f8&gt; (a weblogic.rjvm.ConnectionManagerClient)
      at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:437)
      at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:396)
      at weblogic.rjvm.RJVMImpl.ensureConnectionEstablished(RJVMImpl.java:303)
      at weblogic.rjvm.RJVMImpl.getOutputStream(RJVMImpl.java:347)
      at weblogic.rjvm.RJVMImpl.getRequestStreamInternal(RJVMImpl.java:609)
      at weblogic.rjvm.RJVMImpl.getRequestStream(RJVMImpl.java:560)
      at weblogic.rjvm.RJVMImpl.getOutboundRequest(RJVMImpl.java:786)
      at weblogic.rmi.internal.BasicRemoteRef.getOutboundRequest(BasicRemoteRef.java:159)
      at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:341)
      at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:259)
  </code></pre>
  
  <p>Any ideas? </p>