title: Java AmazonS3Client getObject hanging, thread state stuck in IN_NATIVE during socketRead0
body: <p>I have a Java application which runs a series of parallel threads that download chunks of objects from AmazonS3. I noticed, after running for a few hours without any issues, eventually getting a specific chunk will hang and cause the program to freeze. </p>
  
  <p>The Java process is still running, but CPU usage and network IO are minimal. Doing a thread dump, there is one thread that is never leaving the IN_NATIVE state and socketRead0() seems to never return during an SSL handshake. Here is the paraphrased stack trace:</p>
  
  <pre>Thread 17260: (state = IN_NATIVE)
   - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) @bci=0 (Compiled frame; information may be imprecise)
   - java.net.SocketInputStream.socketRead(java.io.FileDescriptor, byte[], int, int, int) @bci=8, line=116 (Compiled frame)
   - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79, line=170 (Compiled frame)
   - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=141 (Compiled frame)
   - sun.security.ssl.InputRecord.readFully(java.io.InputStream, byte[], int, int) @bci=21, line=465 (Compiled frame)
   - sun.security.ssl.InputRecord.read(java.io.InputStream, java.io.OutputStream) @bci=32, line=503 (Compiled frame)
   - sun.security.ssl.SSLSocketImpl.readRecord(sun.security.ssl.InputRecord, boolean) @bci=44, line=973 (Compiled frame)
   - sun.security.ssl.SSLSocketImpl.performInitialHandshake() @bci=84, line=1375 (Compiled frame)
   - sun.security.ssl.SSLSocketImpl.startHandshake(boolean) @bci=13, line=1403 (Compiled frame)
   - sun.security.ssl.SSLSocketImpl.startHandshake() @bci=2, line=1387 (Compiled frame)
   - org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(int, java.net.Socket, org.apache.http.HttpHost, java.net.InetSocketAddress, java.net.InetSocketAddress, org.apache.http.protocol.HttpContext) @bci=87, line=533 (Compiled frame)
   - org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(java.net.Socket, java.net.InetSocketAddress, java.net.InetSocketAddress, org.apache.http.params.HttpParams) @bci=69, line=401 (Compiled frame)
   - com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(java.net.Socket, java.net.InetSocketAddress, java.net.InetSocketAddress, org.apache.http.params.HttpParams) @bci=60, line=128 (Compiled frame)
   - org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(org.apache.http.conn.OperatedClientConnection, org.apache.http.HttpHost, java.net.InetAddress, org.apache.http.protocol.HttpContext, org.apache.http.params.HttpParams) @bci=226, line=177 (Compiled frame)
   - org.apache.http.impl.conn.ManagedClientConnectionImpl.open(org.apache.http.conn.routing.HttpRoute, org.apache.http.protocol.HttpContext, org.apache.http.params.HttpParams) @bci=126, line=304 (Compiled frame)
   - org.apache.http.impl.client.DefaultRequestDirector.tryConnect(org.apache.http.impl.client.RoutedRequest, org.apache.http.protocol.HttpContext) @bci=50, line=610 (Compiled frame)
   - org.apache.http.impl.client.DefaultRequestDirector.execute(org.apache.http.HttpHost, org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext) @bci=389, line=445 (Compiled frame)
   - org.apache.http.impl.client.AbstractHttpClient.doExecute(org.apache.http.HttpHost, org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext) @bci=365, line=863 (Compiled frame)
   - org.apache.http.impl.client.CloseableHttpClient.execute(org.apache.http.client.methods.HttpUriRequest, org.apache.http.protocol.HttpContext) @bci=14, line=82 (Compiled frame)
   - org.apache.http.impl.client.CloseableHttpClient.execute(org.apache.http.client.methods.HttpUriRequest, org.apache.http.protocol.HttpContext) @bci=3, line=57 (Compiled frame)
   - com.amazonaws.http.AmazonHttpClient.executeOneRequest(com.amazonaws.Request, com.amazonaws.http.HttpResponseHandler, com.amazonaws.http.HttpResponseHandler, com.amazonaws.http.ExecutionContext, com.amazonaws.util.AWSRequestMetrics, com.amazonaws.http.AmazonHttpClient$ExecOneRequestParams) @bci=369, line=728 (Compiled frame)
   - com.amazonaws.http.AmazonHttpClient.executeHelper(com.amazonaws.Request, com.amazonaws.http.HttpResponseHandler, com.amazonaws.http.HttpResponseHandler, com.amazonaws.http.ExecutionContext) @bci=312, line=489 (Compiled frame)
   - com.amazonaws.http.AmazonHttpClient.execute(com.amazonaws.Request, com.amazonaws.http.HttpResponseHandler, com.amazonaws.http.HttpResponseHandler, com.amazonaws.http.ExecutionContext) @bci=150, line=310 (Compiled frame)
   - com.amazonaws.services.s3.AmazonS3Client.invoke(com.amazonaws.Request, com.amazonaws.http.HttpResponseHandler, java.lang.String, java.lang.String) @bci=168, line=3796 (Compiled frame)
   - com.amazonaws.services.s3.AmazonS3Client.getObject(com.amazonaws.services.s3.model.GetObjectRequest) @bci=235, line=1201 (Compiled frame)
  ...</pre>
  
  <p>And here is a code snippet for how this is being executed:</p>
  
  <pre>
  protected static byte[] getChunk(AmazonS3Client client, long start, int offset, String bucket, String key) {
      if(offset &lt; 1){
          return null;
      }
      S3Object obj = null;
      S3ObjectInputStream is = null;
      byte[] buffer = new byte[offset];
  
      try {
          GetObjectRequest request = new GetObjectRequest(bucket, key);
          request.setRange(start, start + offset);
  
          obj = client.getObject(request);
          is = obj.getObjectContent();
          int copied = 0;
          int length;
          while ((length = is.read(buffer,copied,offset-copied)) > 0){
              copied += length;
          }
  
          buffer = (copied &lt; 1) ? null : Arrays.copyOf(buffer, copied);
      } catch (Exception e) {
          buffer = null;
      } finally {
          Util.close(is);
      }
      return buffer;
  }
  </pre>
  
  <p>As a side note, in case this can be resolved with a different AmazonS3Client configuration, here is how the client is instantiated:</p>
  
  <pre>
  private static final String S3_ENDPOINT = "https://s3.amazonaws.com/";
  
  public static AmazonS3Client getAmazonClient(String accessKey, String secretKey){
      AmazonS3Client client = null;
      try {
          ClientConfiguration clientConfig = new ClientConfiguration();
          clientConfig.setProtocol(Protocol.HTTP);
  
          AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
  
          client = new AmazonS3Client(credentials, clientConfig);
          client.setEndpoint(S3_ENDPOINT);
      } catch (Exception e){
          e.printStackTrace();
          client = null;
      }
      return client;
  }
  </pre>
  
  <p>(This is also with aws-java-sdk version 1.10.9). Here are the JVM specs:</p>
  
  <pre>
  java version "1.8.0_51"
  Java(TM) SE Runtime Environment (build 1.8.0_51-b16)
  Java HotSpot(TM) 64-Bit Server VM (build 25.51-b03, mixed mode)
  </pre>
  
  <p>Any ideas how to overcome this hang on the socket initialization?</p>