title: Cassandra High client read request latency compared to local read latency
body: <p>We have a 20 nodes Cassandra cluster running a lot of read requests (~900k/sec at peak). Our dataset is fairly small, so everything is served directly from memory (OS Page Cache). Our datamodel is quite simple (just a key/value) and all of our reads are performed with consistency level one (RF 3).</p>
  
  <p>We use the Java Datastax driver with TokenAwarePolicy, so all of our reads should go directly to one node that has the requested data.</p>
  
  <p>These are some metrics extracted from one of the nodes regarding client read request latency and local read latency.</p>
  
  <pre><code>org_apache_cassandra_metrics_ClientRequest_50thPercentile{scope="Read",name="Latency",} 105.778
  org_apache_cassandra_metrics_ClientRequest_95thPercentile{scope="Read",name="Latency",} 1131.752
  org_apache_cassandra_metrics_ClientRequest_99thPercentile{scope="Read",name="Latency",} 3379.391
  org_apache_cassandra_metrics_ClientRequest_999thPercentile{scope="Read",name="Latency",} 25109.16
  
  org_apache_cassandra_metrics_Keyspace_50thPercentile{keyspace=“&lt;keyspace&gt;”,name="ReadLatency",} 61.214
  org_apache_cassandra_metrics_Keyspace_95thPercentile{keyspace="&lt;keyspace&gt;",name="ReadLatency",} 126.934
  org_apache_cassandra_metrics_Keyspace_99thPercentile{keyspace="&lt;keyspace&gt;",name="ReadLatency",} 182.785
  org_apache_cassandra_metrics_Keyspace_999thPercentile{keyspace="&lt;keyspace&gt;",name="ReadLatency",} 454.826
  
  org_apache_cassandra_metrics_Table_50thPercentile{keyspace="&lt;keyspace&gt;",scope="&lt;table&gt;",name="CoordinatorReadLatency",} 105.778
  org_apache_cassandra_metrics_Table_95thPercentile{keyspace="&lt;keyspace&gt;",scope="&lt;table&gt;",name="CoordinatorReadLatency",} 1131.752
  org_apache_cassandra_metrics_Table_99thPercentile{keyspace="&lt;keyspace&gt;",scope="&lt;table&gt;",name="CoordinatorReadLatency",} 3379.391
  org_apache_cassandra_metrics_Table_999thPercentile{keyspace="&lt;keyspace&gt;",scope="&lt;table&gt;",name="CoordinatorReadLatency",} 25109.16
  </code></pre>
  
  <p>Another important detail is that most of our queries (~70%) don't return anything, i.e., they are for records not found. So, bloom filters play an important role here and they seem to be fine:</p>
  
  <pre><code>Bloom filter false positives: 27574
  Bloom filter false ratio: 0.00000
  Bloom filter space used: 
  Bloom filter off heap memory used: 6760992
  </code></pre>
  
  <p>As it can be seen, the reads in each one of the nodes are really fast, the 99.9% is less than 0.5 ms. However, the client request latency is way higher, going above 4ms on the 99%. If I'm reading with CL ONE and using TokenAwarePolicy, shouldn't both values be similar to each other, since no coordination is required? Am I missing something? Is there anything else I could check to see what's going on?</p>
  
  <p>Thanks in advance.</p>