title: web service client encounters a 'java.net.SocketException: Connection reset'
body: <p><strong>problem</strong></p>
  
  <p>We have a java ee app deployed on Weblogic server 10.3.1 which provides HTTP web service. Let's just call it 'server'. Another java app deployed on tomcat on another machine, which will make web service calls to the server, Let's just call it 'client'. Both the client and the server app are using Axis.</p>
  
  <p>There are chances that the client will fail with an exception </p>
  
  <pre><code>"java.net.SocketException: Connection reset
      at java.net.SocketInputStream.read(SocketInputStream.java:168)", 
  </code></pre>
  
  <p>and 1 minute later the a server app log shows:</p>
  
  <pre><code>org.apache.axis.Message ERROR  - 
  java.io.IOException: java.net.SocketException: Socket closed
       at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:99). 
  </code></pre>
  
  <p>It failed no more than 5 times a day. The total web service request on the server is less, like, several hundreds. And it seems it tend to fail after the server has a long time 'rest': The first call on every morning will probably fail (we don't have business services at night).</p>
  
  <p><strong>The exceptions stacks are:</strong></p>
  
  <p>Client side:</p>
  
  <pre><code>Caused by: java.net.SocketException: Connection reset
  at java.net.SocketInputStream.read(SocketInputStream.java:168)
  at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
  at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
  at java.io.FilterInputStream.read(FilterInputStream.java:66)
  at org.apache.xerces.impl.XMLEntityManager$RewindableInputStream.read(Unknown Source)
  at org.apache.xerces.impl.XMLEntityManager.setupCurrentEntity(Unknown Source)
  at org.apache.xerces.impl.XMLVersionDetector.determineDocVersion(Unknown Source)
  at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
  at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
  at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
  at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
  at javax.xml.parsers.SAXParser.parse(SAXParser.java:375)
  at org.apache.axis.encoding.DeserializationContext.parse(DeserializationContext.java:227)
  at org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:696)
  ... 12 more
  </code></pre>
  
  <p>Server side:</p>
  
  <pre><code>2013-03-06 08:37:29,491 org.apache.axis.Message ERROR  - java.io.IOException:
  java.net.SocketException: Socket closed
      at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:99)
      at java.net.SocketOutputStream.write(SocketOutputStream.java:137)
      at weblogic.servlet.internal.ChunkOutput.writeChunkNoTransfer(ChunkOutput.java:530)
      at weblogic.servlet.internal.ChunkOutput.writeChunks(ChunkOutput.java:487)
      at weblogic.servlet.internal.ChunkOutput.flush(ChunkOutput.java:382)
      at weblogic.servlet.internal.CharsetChunkOutput.flush(CharsetChunkOutput.java:315)
      at weblogic.servlet.internal.ChunkOutput$2.checkForFlush(ChunkOutput.java:580)
      at weblogic.servlet.internal.CharsetChunkOutput.write(CharsetChunkOutput.java:222)
      at weblogic.servlet.internal.ChunkOutputWrapper.write(ChunkOutputWrapper.java:146)
      at weblogic.servlet.internal.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:138)
      at org.apache.axis.utils.ByteArray.writeTo(ByteArray.java:375)
      at org.apache.axis.SOAPPart.writeTo(SOAPPart.java:265)
      at org.apache.axis.Message.writeTo(Message.java:539)
      at org.apache.axis.transport.http.AxisServlet.sendResponse(AxisServlet.java:902)
      at org.apache.axis.transport.http.AxisServlet.doPost(AxisServlet.java:777)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
      at org.apache.axis.transport.http.AxisServletBase.service(AxisServletBase.java:327)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:821)
      at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
      at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
      at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
      at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:27)
      at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
      at org.springframework.security.util.FilterChainProxy.doFilter(FilterChainProxy.java:170)
      at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
      at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:168)
      at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
      at org.springframework.orm.jpa.support.OpenEntityManagerInViewFilter.doFilterInternal(OpenEntityManagerInViewFilter.java:112)
      at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
      at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
      at com.ulic.ucia.framework.util.AppFilter.doFilter(AppFilter.java:48)
      at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
      at com.ebao.pub.framework.AppFilter.doFilter(AppFilter.java:101)
      at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:57)
      at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3588)
      at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
      at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
      at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2200)
      at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2106)
      at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1428)
      at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
      at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
  </code></pre>
  
  <p>When the client fails, we check the app log and the database, and it turns out that the business transaction is always success. Weblogic server somehow fails to write the HTTP response. And Weblogic server log file says:</p>
  
  <p>BEA 101366 The server could not send the HTTP message during the configured timeout value. The socket has been closed.</p>
  
  <p><strong>Sorted logs and some analyses</strong></p>
  
  <p>Both the servers have ntp service. I simplified and sorted the server logs and the client logs by time to get a clearer view,</p>
  
  <pre><code>2013-03-06 08:36:18,755  Web Service Client      began it's work
  2013-03-06 08:36:18,758  Web Service Client  make http calls and wait for response
  2013-03-06 08:36:19,039  APP on WLS         received request
  2013-03-06 08:36:24,553  APP on WLS         app log says transaction finished with a success.
  2013-03-06 08:36:24,575  Web Service Client received java.net.SocketException: Connection reset” 
  2013-03-06 08:37:29,491  APP on WLS      org.apache.axis.Message ERROR  - java.io.IOException:java.net.SocketException: Socket closed
  </code></pre>
  
  <p>Most notably, the client received a Connection Reset after 6 seconds of waiting when the server app just finished proceeding. So I think it must has something to do with the server. The server machine sended a 'TCP RST' when weblogic was trying to send http response due to some reason, but what is that reason?
      For all I know , there are two scenarios (may be more) the 'RST' will be send in an java application. 
      Java thread ends without closing the socket, the tcp stack will send a 'RST' to the socket on the other hand to indicate an error
      Tcp stack failed to send all data after 'linger' time. </p>
  
  <p>And now my brain has stucked, don't know what to do next. Any suggestions will be appreciated.</p>