title: IO errors integrating two Java webapps via http
body: <p>I have two java webapps, each one deployed on a separate tomcat server (and in a separate machine), one of them makes a lot of consecutive (but sequential, not parallel) http requests to the other one (+1000). It works most of the time, however when the load on the "client server" (the one that generates the requests) is high, I get random IO errors I have been unable to track down.</p>
  
  <p>Here's a simplified version of the client</p>
  
  <pre class="lang-java prettyprint-override"><code>public class BPXProxy {
      private static HttpClient client;
      public static void init(){
          client = new HttpClient();
      }
      public static InvoiceCreationInfo registerInvoice(InvoiceData invoice) throws SQLException, HttpException, IOException {
          JSONObject invoiceJSON = new JSONObject();
          invoiceJSON.put("invoicer", invoice.getInvoicerIdentification().getDocument());
          invoiceJSON.put("buyer", invoice.getBuyerIdentification().getDocument());
          String serviceURL = ConfigurationManager.getBPXServicesPath() + "Invoice?operation=registerExternalInvoice&amp;branchCode="+ branchCode;
          PostMethod updatePage = new PostMethod(serviceURL);
          updatePage.addParameter("invoice", invoiceJSON.toJSONString());
          updatePage.addParameter("invoiceSubject", invoice.getInvoiceSubject());
          client.executeMethod(updatePage); // java.io.IOException: Stream closed, java.net.SocketException: Socket closed
          String response = updatePage.getResponseBodyAsString();  // java.io.IOException: chunked stream ended unexpectedly, java.io.IOException: CRLF expected at end of chunk: -1/-1
          updatePage.releaseConnection();
          JSONObject jsonResponse = JSONSimpleHelper.parseJSON(response);
          return jsonResponse;
      }
  }
  </code></pre>
  
  <p>Examples of these exceptions:</p>
  
  <pre><code>java.io.IOException: chunked stream ended unexpectedly
          at org.apache.commons.httpclient.ChunkedInputStream.getChunkSizeFromInputStream(ChunkedInputStream.java:251)
          at org.apache.commons.httpclient.ChunkedInputStream.nextChunk(ChunkedInputStream.java:220)
          at org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.java:175)
          at java.io.FilterInputStream.read(FilterInputStream.java:133)
          at org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:107)
          at java.io.FilterInputStream.read(FilterInputStream.java:107)
          at org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:126)
          at org.apache.commons.httpclient.HttpMethodBase.getResponseBody(HttpMethodBase.java:684)
          at org.apache.commons.httpclient.HttpMethodBase.getResponseBodyAsString(HttpMethodBase.java:735)
          at net.slashware.mobilia.slashbillingClient.proxy.BPXProxy.registerInvoice(BPXProxy.java:208)
  
  java.io.IOException: Stream closed
      at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:162)
      at java.io.BufferedInputStream.fill(BufferedInputStream.java:206)
      at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
      at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:77)
      at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:105)
      at org.apache.commons.httpclient.HttpParser.parseHeaders(HttpParser.java:165)
      at org.apache.commons.httpclient.HttpMethodBase.readResponseHeaders(HttpMethodBase.java:1790)
      at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1592)
      at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:995)
      at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:397)
      at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
      at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
      at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)
      at net.slashware.mobilia.slashbillingClient.proxy.BPXProxy.registerInvoice(BPXProxy.java:207)
  
  java.net.SocketException: Socket closed
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:146)
      at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
      at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
      at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:77)
      at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:105)
      at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1115)
      at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1832)
      at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1590)
      at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:995)
      at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:397)
      at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
      at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
      at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)
      at net.slashware.mobilia.slashbillingClient.proxy.BPXProxy.getAccountStatusInvoices(BPXProxy.java:2905)
  
  java.io.IOException: CRLF expected at end of chunk: -1/-1
      at org.apache.commons.httpclient.ChunkedInputStream.readCRLF(ChunkedInputStream.java:206)
      at org.apache.commons.httpclient.ChunkedInputStream.nextChunk(ChunkedInputStream.java:218)
      at org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.java:175)
      at java.io.FilterInputStream.read(FilterInputStream.java:133)
      at org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:107)
      at java.io.FilterInputStream.read(FilterInputStream.java:107)
      at org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:126)
      at org.apache.commons.httpclient.HttpMethodBase.getResponseBody(HttpMethodBase.java:684)
      at org.apache.commons.httpclient.HttpMethodBase.getResponseBodyAsString(HttpMethodBase.java:735)
      at net.slashware.mobilia.slashbillingClient.proxy.BPXProxy.getPersonOpenReceivables(BPXProxy.java:3132)
  
  org.apache.commons.httpclient.ProtocolException: The server ec2-54-196-218-148.compute-1.amazonaws.com failed to respond with a valid HTTP response
      at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1846)
      at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1590)
      at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:995)
      at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:397)
      at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
      at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
      at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)
      at net.slashware.mobilia.slashbillingClient.proxy.BPXProxy.registerInvoice(BPXProxy.java:207)
  </code></pre>
  
  <p>Any idea why these could happen? does it look like the fix should be for the "client server" or the other server?</p>
  
  <p>I am doing this on aws, each server is on a separate ec2 node, each request is taking about 4 seconds to complete. I'm using apache commons httpclient to make the calls; I have monitored the servers and there are no issues with memory consumption, and I don't see peaks with network in and network out on each servers.</p>