title: What exactly happens when tombstone limit is reached
body: <p>According to cassandra's log (see below) queries are getting aborted due to too many <code>tombstones</code> being present. This is happening because once a week I cleanup (delete) rows with a counter that is too low. This 'deletes' <em>hundreds of thousands</em> of rows (marks them as such with a <code>tombstone</code>.)</p>
  
  <p>It is not at all a problem if, in this table, a deleted row re-appears because a node was down during the cleanup process, so I set the <code>gc grace time</code> for the single affected table to 10 hours (down from default 10 days) so the tombstoned rows can get permanently deleted relatively fast.</p>
  
  <p>Regardless, I had to set the <code>tombstone_failure_threshold</code> extremely high to avoid the below exception. (one hundred million, up from one hundred thousand.) My question is, is this necessary? I have absolutely no idea what type of queries get aborted; inserts, selects, deletes?</p>
  
  <p>If it's merely some selects being aborted, it's not that big a deal. But that's assuming abort means 'capped' in that the query stops prematurely and returns whatever live data it managed to gather before too many tombstones were found.</p>
  
  <p>Well, to ask it simpler; what happens when the <code>tombstone_failure_threshold</code> is exceeded?</p>
  
  <pre><code>INFO [HintedHandoff:36] 2014-02-12 17:44:22,355 HintedHandOffManager.java (line 323) Started hinted handoff for host: fb04ad4c-xxxx-4516-8569-xxxxxxxxx with IP: /XX.XX.XXX.XX
  ERROR [HintedHandoff:36] 2014-02-12 17:44:22,667 SliceQueryFilter.java (line 200) Scanned over 100000 tombstones; query aborted (see tombstone_fail_threshold)
  ERROR [HintedHandoff:36] 2014-02-12 17:44:22,668 CassandraDaemon.java (line 187) Exception in thread Thread[HintedHandoff:36,1,main]
  org.apache.cassandra.db.filter.TombstoneOverwhelmingException
      at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:201)
      at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122)
      at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
      at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
      at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
      at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
      at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1516)
      at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1335)
      at org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:351)
      at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:309)
      at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:92)
      at org.apache.cassandra.db.HintedHandOffManager$4.run(HintedHandOffManager.java:530)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:744)
  </code></pre>
  
  <p>Forgot to mention; running Cassandra version <code>2.0.4</code></p>