title: Hazelcast - OperationTimeoutException
body: <p>I am using Hazelcast version 3.3.1.<BR>
  I have a 9 node cluster running on aws using c3.2xlarge servers.<BR>
  I am using a distributed executor service and a distributed map.<BR>
  Distributed executor service uses a single thread.
  Distributed map is configured with no replication and no near-cache and stores about 1 million objects of size 1-2kb using Kryo serializer.<BR>
  My use case goes as follow:<BR></p>
  
  <ul>
  <li>All 9 nodes constantly execute a synchronous remote operation on the distributed executor service and generate about 20k hits per second (about ~2k per node).</li>
  <li>Invocations are executed using Hazelcast API: com.hazelcast.core.IExecutorService#executeOnKeyOwner.</li>
  <li>Each operation accesses the distributed map on the node owning the partition, does some calculation using the stored object and stores the object in to the map. (for that I use the get and set API of the IMap object).</li>
  </ul>
  
  <p>Every once in a while Hazelcast encounters a timeout exceptions such as:<BR>
  <em>com.hazelcast.core.OperationTimeoutException: No response for 120000 ms. Aborting invocation! BasicInvocationFuture{invocation=BasicInvocation{ serviceName='hz:impl:mapService', op=GetOperation{}, partitionId=212, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[172.31.44.2]:5701, backupsExpected=0, backupsCompleted=0}, response=null, done=false} No response has been received!  backups-expected:0 backups-completed: 0</em> </p>
  
  <p>In some cases I see map partitions start to migrate which makes thing even worse, nodes constantly leave and re-join the cluster and the only way I can overcome the problem is by restarting the entire cluster.</p>
  
  <p>I am wondering what may cause Hazelcast to block a map-get operation for 120 seconds?<br>
  I am pretty sure it's not network related since other services on the same servers operate just fine.
  Also note that the servers are mostly idle (~70%).</p>
  
  <p>Any feedbacks on my use case will be highly appreciated.</p>