title: OperationTimeoutException Cassandra cluster AWS / EMR
body: <p>I have an Java app running on Amazon on a cluster of Cassandra managed by Priam.</p>
  
  <p>We use the Elastic Map/Reduce service of Amazon, and at certain moment at time, when I running EMR and I try get some data inserted on Cassandra I got  an Exception: OperationTimeoutException.</p>
  
  <p>These are the configuration parameters passed when I create my Cassandra pool over Astyanax:</p>
  
  <pre><code>`ConnectionPoolConfigurationImpl conPool = new` `ConnectionPoolConfigurationImpl(getConecPoolName())`
      .setMaxConnsPerHost(20)
          .setSeeds("ec2-xx-xxx-xx-xx.compute-1.amazonaws.com")
      .setMaxOperationsPerConnection(100)                       .setMaxPendingConnectionsPerHost(20) 
      .setConnectionLimiterMaxPendingCount(20) 
      .setTimeoutWindow(10000) 
      .setConnectionLimiterWindowSize(1000) 
      .setMaxTimeoutCount(3) 
      .setConnectTimeout(5000) 
      .setMaxFailoverCount(-1) 
      .setLatencyAwareBadnessThreshold(20)
          .setLatencyAwareUpdateInterval(1000)
      .setLatencyAwareResetInterval(10000) 
          .setLatencyAwareWindowSize(100) 
      .setLatencyAwareSentinelCompare(100f) 
  
  
  AstyanaxContext&lt;Keyspace&gt; context = new AstyanaxContext.Builder()
          .forCluster("clusterName")
          .forKeyspace("keyspaceName")
      .withAstyanaxConfiguration(
             new AstyanaxConfigurationImpl().setDiscoveryType(NodeDiscoveryType.NONE))
      .withConnectionPoolConfiguration(conPool)
      .withConnectionPoolMonitor(new CountingConnectionPoolMonitor())
      .buildKeyspace(ThriftFamilyFactory.getInstance());
  </code></pre>
  
  <p>full stack trace:</p>
  
  <pre><code>ERROR com.s1mbi0se.dg.input.service.InputService (main): EXCEPTION:OperationTimeoutException: [host=ec2-xx-xxx-xx-xx.compute-1.amazonaws.com(10.100.6.242):9160, latency=10004(10004), attempts=1]TimedOutException()
  
  com.netflix.astyanax.connectionpool.exceptions.OperationTimeoutException: OperationTimeoutException: [host=ec2-54-224-65-18.compute-1.amazonaws.com(10.100.6.242):9160, latency=10004(10004), attempts=1]TimedOutException()
      at com.netflix.astyanax.thrift.ThriftConverter.ToConnectionPoolException(ThriftConverter.java:171)
      at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:61)
      at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1$2.execute(ThriftColumnFamilyQueryImpl.java:206)
      at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1$2.execute(ThriftColumnFamilyQueryImpl.java:198)
      at com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.execute(ThriftSyncConnectionFactoryImpl.java:151)
      at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:69)
      at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:253)
      at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1.execute(ThriftColumnFamilyQueryImpl.java:196)
      at com.s1mbi0se.dg.input.service.InputService.searchUserByKey(InputService.java:833)
      at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
      at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:771)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:375)
      at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
      at org.apache.hadoop.mapred.Child.main(Child.java:249)
  Caused by: TimedOutException()
      at org.apache.cassandra.thrift.Cassandra$get_slice_result.read(Cassandra.java:7874)
      at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
      at org.apache.cassandra.thrift.Cassandra$Client.recv_get_slice(Cassandra.java:594)
      at org.apache.cassandra.thrift.Cassandra$Client.get_slice(Cassandra.java:578)
      at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1$2.internalExecute(ThriftColumnFamilyQueryImpl.java:211)
      at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1$2.internalExecute(ThriftColumnFamilyQueryImpl.java:198)
      at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:56)
  </code></pre>
  
  <p>So I don't know what direction I get to go to solve this problem, because the problem may be in Astyanax pool configuration, EC2 machine configuration (memory increase  ?), Priam configuration or some another configuration necessary for Cassandra or EMR service on AWS in my code ... any hint ?</p>
  
  <hr>
  
  <p>follows the stack trace:</p>
  
  <pre><code>INFO org.apache.hadoop.mapred.TaskLogsTruncater (main): Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
  WARN org.apache.hadoop.mapred.Child (main): Error running child
  java.lang.RuntimeException: InvalidRequestException(why:Start key's token sorts after end token)
      at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$WideRowIterator.maybeInit(ColumnFamilyRecordReader.java:453)
      at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$WideRowIterator.computeNext(ColumnFamilyRecordReader.java:459)
      at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$WideRowIterator.computeNext(ColumnFamilyRecordReader.java:406)
      at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
      at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
      at org.apache.cassandra.hadoop.ColumnFamilyRecordReader.getProgress(ColumnFamilyRecordReader.java:103)
      at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.getProgress(MapTask.java:522)
      at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:547)
      at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
      at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
      at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:771)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:375)
      at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
      at org.apache.hadoop.mapred.Child.main(Child.java:249)
  Caused by: InvalidRequestException(why:Start key's token sorts after end token)
      at org.apache.cassandra.thrift.Cassandra$get_paged_slice_result.read(Cassandra.java:14168)
      at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
      at org.apache.cassandra.thrift.Cassandra$Client.recv_get_paged_slice(Cassandra.java:769)
      at org.apache.cassandra.thrift.Cassandra$Client.get_paged_slice(Cassandra.java:753)
      at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$WideRowIterator.maybeInit(ColumnFamilyRecordReader.java:438)
      ... 
  INFO org.apache.hadoop.mapred.Task (main): Runnning cleanup for the task
  </code></pre>