title: Hector is unable to read Cassandra data when nodes reboot or terminate
body: <p>We are trying to run a cassandra cluster on AWS/EC2 within a standard VPC footprint (cassandra nodes on private subnets). Because this is AWS there is always a chance that an EC2 instance will terminate or reboot with no warning. I have been simulating this case on a test cluster and I am seeing things with the cluster that I thought a cluster was suppose to prevent. Specifically if a node reboots some data will go temporarily missing until the node completes its reboot. If a node terminates it appears that some data is lost forever.</p>
  
  <p>For my test I just did a bunch of writes (using QUORUM consistency) to some keyspaces then interrogate the contents of those keyspaces as I bring down nodes (either through reboot or terminate). I'm just using cqlsh SELECT to do the keyspace/column family interrogation of the cluster using ONE consistency level.</p>
  
  <p>Note, even though I am performing no writes to the cluster while I am doing the SELECTs rows temporarily disappear when rebooting and can permanently go missing during termination.</p>
  
  <p>I thought Netflix Priam might be able to help, but sadly it doesn't work in a VPC the last time I checked.</p>
  
  <p>Also, because we are using ephemeral storage instances there is no equivalent of 'shutdown' so I cannot run any scripts during reboot/terminate of an instance to perform a <code>nodetool decommission</code> or <code>nodetool removenode</code> before an instance goes away. Terminate is the equivalent of kicking the plug out of the wall.</p>
  
  <p>Since I am using a replication factor of 3 and quorum/write that should mean that all data is written to at least 2 nodes. So, unless I am totally misunderstanding things (which is possible), losing one node should not mean that I lose any data for any period of time when I am using consistency level ONE for the read.</p>
  
  <h1>Questions</h1>
  
  <p>Why wouldn't a 6 node cluster with a replication factor of 3 work?</p>
  
  <p>Do I need to run something like a 12 node cluster with a replication factor of 7? Don't bother telling me that will fix the problem, because it doesn't.</p>
  
  <p>Do I need to use consistency level of ALL on the writes then use ONE or QUORUM on the reads?</p>
  
  <p>Is there something not quite right with virtual nodes? unlikely</p>
  
  <p>Are there nodetool commands besides removenode that I need to run when a node terminates to recover missing data? As mentioned earlier, when a reboot occurs, eventually the missing data reappears.</p>
  
  <p>Is there some cassandra savant who can look at my cassandra.yaml file below and send me on the path to salvation?</p>
  
  <h1>More Info added 7/19</h1>
  
  <p>I don't think this is a QUORUM vs ONE vs ALL is the issue. The test I set up performs no writes to the keyspaces after the initial population of the column families. So the data has had plenty of time (hours) to make it to all the nodes as required by the replication factor. Plus the test dataset is REALLY small (2 column families with about 300-1000 values each). So in other words, the data is completely static.</p>
  
  <p>The behavior I am seeing seems to be tied to the fact that the ec2 instance is no longer on the network. The reason I say this is because if I log on to a node and just do a <code>cassandra stop</code> I see no loss of data. But if I do the reboot or terminate I start getting the following in a stack trace.</p>
  
  <pre><code>CassandraHostRetryService  - Downed Host Retry service started with queue size -1 and retry delay 10s
  CassandraHostRetryService  - Downed Host retry shutdown complete
  CassandraHostRetryService  - Downed Host retry shutdown hook called
  Caused by: TimedOutException()
  Caused by: TimedOutException()
  </code></pre>
  
  <p>So it seems to be more of a networking communication issue in that the cluster is expecting, for example 10.0.12.74, to be on the network after it has joined the cluster. If that ip is suddenly unreachable either due to reboot or termination the timeouts start happening.</p>
  
  <p>When I do a <code>nodetool status</code> under all three scenarios (<code>cassandra stop</code>, reboot or terminate) the status of the node shows up as DN. Which is what you would expect. Eventually <code>nodetool status</code> will return to UN with <code>cassandra start</code> or reboot, but obviously termination always stays DN.</p>
  
  <h1>Details of my Configuration</h1>
  
  <p>Here are some details of my configuration (cassandra.yaml is at the bottom of this posting):</p>
  
  <p>Nodes are running in private subnets of a VPC.</p>
  
  <p>Cassandra 1.2.5 with num_tokens: 256 (virtual nodes). initial_token: (blank). I am really hoping this works because all of our nodes run in autoscaling groups so the thought that redistribution could be handle dynamically is appealing.</p>
  
  <p>EC2 m1.large one seed and one non-seed node in each availability zone. (so 6 total nodes in the cluster). </p>
  
  <p>Ephemeral storage, not EBS.</p>
  
  <p>Ec2Snitch with NetworkTopologyStrategy and all keyspaces have replication factor of 3.</p>
  
  <p>Non-seed nodes are auto_bootstraped, seed nodes are not.</p>
  
  <p>sample cassandra.yaml file</p>
  
  <pre><code> cluster_name: 'TestCluster'
   num_tokens: 256
   initial_token:
   hinted_handoff_enabled: true
   max_hint_window_in_ms: 10800000
   hinted_handoff_throttle_in_kb: 1024
   max_hints_delivery_threads: 2
   authenticator: org.apache.cassandra.auth.AllowAllAuthenticator
   authorizer: org.apache.cassandra.auth.AllowAllAuthorizer
   partitioner: org.apache.cassandra.dht.Murmur3Partitioner
   disk_failure_policy: stop
   key_cache_size_in_mb:
   key_cache_save_period: 14400
   row_cache_size_in_mb: 0
   row_cache_save_period: 0
   row_cache_provider: SerializingCacheProvider
   saved_caches_directory: /opt/company/dbserver/caches
   commitlog_sync: periodic
   commitlog_sync_period_in_ms: 10000
   commitlog_segment_size_in_mb: 32
   seed_provider:
   - class_name: org.apache.cassandra.locator.SimpleSeedProvider
   parameters:
   - seeds: "SEED_IP_LIST"
   flush_largest_memtables_at: 0.75
   reduce_cache_sizes_at: 0.85
   reduce_cache_capacity_to: 0.6
   concurrent_reads: 32
   concurrent_writes: 8
   memtable_flush_queue_size: 4
   trickle_fsync: false
   trickle_fsync_interval_in_kb: 10240
   storage_port: 7000
   ssl_storage_port: 7001
   listen_address: LISTEN_ADDRESS
   start_native_transport: false
   native_transport_port: 9042
   start_rpc: true
   rpc_address: 0.0.0.0
   rpc_port: 9160
   rpc_keepalive: true
   rpc_server_type: sync
   thrift_framed_transport_size_in_mb: 15
   thrift_max_message_length_in_mb: 16
   incremental_backups: true
   snapshot_before_compaction: false
   auto_bootstrap: AUTO_BOOTSTRAP
   column_index_size_in_kb: 64
   in_memory_compaction_limit_in_mb: 64
   multithreaded_compaction: false
   compaction_throughput_mb_per_sec: 16
   compaction_preheat_key_cache: true
   read_request_timeout_in_ms: 10000
   range_request_timeout_in_ms: 10000
   write_request_timeout_in_ms: 10000
   truncate_request_timeout_in_ms: 60000
   request_timeout_in_ms: 10000
   cross_node_timeout: false
   endpoint_snitch: Ec2Snitch
   dynamic_snitch_update_interval_in_ms: 100
   dynamic_snitch_reset_interval_in_ms: 600000
   dynamic_snitch_badness_threshold: 0.1
   request_scheduler: org.apache.cassandra.scheduler.NoScheduler
   index_interval: 128
   server_encryption_options:
   internode_encryption: none
   keystore: conf/.keystore
   keystore_password: cassandra
   truststore: conf/.truststore
   truststore_password: cassandra
   client_encryption_options:
   enabled: false
   keystore: conf/.keystore
   keystore_password: cassandra
   internode_compression: all
  </code></pre>