title: UnavailableException: Not enough replicas available for query at consistency LOCAL_ONE Cassandra Stress
body: <p>I try to make Materialized View tests on our cassandra structure..</p>
  
  <p>I installed Cassandra 3.5 on 6 Node Debian 8 servers from apache repos.</p>
  
  <p>Then I try to run basic stress on it, then I came across with these errors. I can't configure that why I gets this errors.</p>
  
  <p>Here is every node's <code>cassandra.yaml</code> </p>
  
  <pre><code>cluster_name: 'Cluster'
  num_tokens: 256
  hinted_handoff_enabled: true
  max_hint_window_in_ms: 10800000 # 3 hours
  hinted_handoff_throttle_in_kb: 1024
  max_hints_delivery_threads: 2
  hints_flush_period_in_ms: 10000
  max_hints_file_size_in_mb: 128
  batchlog_replay_throttle_in_kb: 1024
  authenticator: PasswordAuthenticator 
  authorizer: AllowAllAuthorizer
  role_manager: CassandraRoleManager
  roles_validity_in_ms: 2000
  permissions_validity_in_ms: 2000
  credentials_validity_in_ms: 2000
  partitioner: org.apache.cassandra.dht.Murmur3Partitioner
  data_file_directories:
      - /var/lib/cassandra/data
  commitlog_directory: /var/lib/cassandra/commitlog
  disk_failure_policy: stop
  commit_failure_policy: stop
  key_cache_size_in_mb:
  key_cache_save_period: 14400
  row_cache_size_in_mb: 0
  row_cache_save_period: 0
  counter_cache_size_in_mb:
  counter_cache_save_period: 7200
  saved_caches_directory: /var/lib/cassandra/saved_caches
  commitlog_sync: periodic
  commitlog_sync_period_in_ms: 10000
  commitlog_segment_size_in_mb: 32
  seed_provider:
      - class_name: org.apache.cassandra.locator.SimpleSeedProvider
        parameters:
            - seeds: "195.x.x.192,195.x.x.195"
  concurrent_reads: 32
  concurrent_writes: 32
  concurrent_counter_writes: 32
  concurrent_materialized_view_writes: 32
  memtable_allocation_type: heap_buffers
  index_summary_capacity_in_mb:
  index_summary_resize_interval_in_minutes: 60
  trickle_fsync: false
  trickle_fsync_interval_in_kb: 10240
  storage_port: 7000
  ssl_storage_port: 7001
  listen_address: 195.x.x.192 # node ip adress
  start_native_transport: true
  native_transport_port: 9042
  start_rpc: true
  rpc_address: 0.0.0.0
  rpc_port: 9160
  broadcast_rpc_address: 195.x.x.255
  rpc_keepalive: true
  rpc_server_type: sync
  thrift_framed_transport_size_in_mb: 15
  incremental_backups: false
  snapshot_before_compaction: false
  auto_snapshot: true
  tombstone_warn_threshold: 1000
  tombstone_failure_threshold: 100000
  column_index_size_in_kb: 64
  batch_size_warn_threshold_in_kb: 5
  batch_size_fail_threshold_in_kb: 50
  compaction_throughput_mb_per_sec: 16
  compaction_large_partition_warning_threshold_mb: 100
  sstable_preemptive_open_interval_in_mb: 50
  read_request_timeout_in_ms: 5000
  range_request_timeout_in_ms: 10000
  write_request_timeout_in_ms: 2000
  counter_write_request_timeout_in_ms: 5000
  cas_contention_timeout_in_ms: 1000
  truncate_request_timeout_in_ms: 60000
  request_timeout_in_ms: 10000
  cross_node_timeout: false
  endpoint_snitch: GossipingPropertyFileSnitch 
  dynamic_snitch_update_interval_in_ms: 100 
  dynamic_snitch_reset_interval_in_ms: 600000
  dynamic_snitch_badness_threshold: 0.1
  request_scheduler: org.apache.cassandra.scheduler.NoScheduler
  server_encryption_options:
      internode_encryption: none
      keystore: conf/.keystore
      keystore_password: cassandra
      truststore: conf/.truststore
      truststore_password: cassandra
  client_encryption_options:
      enabled: false
      optional: false
      keystore: conf/.keystore
      keystore_password: cassandra
  internode_compression: all
  inter_dc_tcp_nodelay: false
  tracetype_query_ttl: 86400
  tracetype_repair_ttl: 604800
  gc_warn_threshold_in_ms: 1000
  enable_user_defined_functions: false
  enable_scripted_user_defined_functions: false
  windows_timer_interval: 1
  transparent_data_encryption_options:
      enabled: false
      chunk_length_kb: 64
      cipher: AES/CBC/PKCS5Padding
      key_alias: testing:1
      key_provider: 
        - class_name: org.apache.cassandra.security.JKSKeyProvider
          parameters: 
            - keystore: conf/.keystore
              keystore_password: cassandra
              store_type: JCEKS
              key_password: cassandra
  </code></pre>
  
  <p><strong>nodetool status :</strong></p>
  
  <pre><code>Datacenter: DC1
  ===============
  Status=Up/Down
  |/ State=Normal/Leaving/Joining/Moving
  --  Address          Load       Tokens       Owns (effective)  Host ID                               Rack
  UN  195.x.x.194  291.61 KB  256          16.7%             0da9c3e0-a0de-4d39-8f94-34f4717c7280  rack1
  UN  195.x.x.192  338.54 KB  256          16.2%             ff966a71-f76b-4106-9132-fbfe840fa5e1  rack1
  UN  195.x.x.193  293.04 KB  256          17.0%             7cf3488a-d9bb-4b92-87ab-3b98fc6887a1  rack1
  Datacenter: DC2
  ===============
  Status=Up/Down
  |/ State=Normal/Leaving/Joining/Moving
  --  Address          Load       Tokens       Owns (effective)  Host ID                               Rack
  UN  195.x.x.195  264.81 KB  256          16.8%             ca8cf24c-7c1a-4b66-be77-a7c02fdb54ea  rack1
  UN  195.x.x.198  288.28 KB  256          17.3%             ed02b226-5830-4aa7-88dd-d6d5f757ed88  rack1
  UN  195.x.x.196  285.21 KB  256          15.9%             4c3944c4-1657-48fe-be3e-737fe545b86b  rack1
  </code></pre>
  
  <p><strong>netstat :</strong></p>
  
  <pre><code>Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
  tcp        0      0 195.x.x.192:7000    0.0.0.0:*               LISTEN      6358/java       
  tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      471/exim4       
  tcp        0      0 127.0.0.1:7199          0.0.0.0:*               LISTEN      6358/java       
  tcp        0      0 0.0.0.0:9160            0.0.0.0:*               LISTEN      6358/java       
  tcp        0      0 127.0.0.1:35945         0.0.0.0:*               LISTEN      6358/java       
  tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      92/rpcbind      
  tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      97/sshd         
  tcp6       0      0 :::111                  :::*                    LISTEN      92/rpcbind      
  tcp6       0      0 0.0.0.0:9042            :::*                    LISTEN      6358/java       
  tcp6       0      0 :::22                   :::*                    LISTEN      97/sshd   
  </code></pre>
  
  <p><strong>Cassanra-stress with thrift mode:</strong></p>
  
  <pre><code>cassandra-stress write n=50000 -rate threads=40 -mode thrift user=x password=x -node {node hostnames}
  </code></pre>
  
  <p>Output:</p>
  
  <pre><code>Unable to create stress keyspace: Keyspace names must be case-insensitively unique ("keyspace1" conflicts with "keyspace1")
  Sleeping 2s...
  Warming up WRITE with 62500 iterations...
  Failed to connect over JMX; not collecting these stats
  UnavailableException()
  UnavailableException()
  UnavailableException()
  UnavailableException()
  UnavailableException()
  UnavailableException()
  UnavailableException()
  UnavailableException()
  UnavailableException()
  UnavailableException()
  java.io.IOException: Operation x10 on key(s) [394e35344e4b34383631]: Error executing: (UnavailableException)
  
      at org.apache.cassandra.stress.Operation.error(Operation.java:138)
      at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:116)
      at org.apache.cassandra.stress.operations.predefined.ThriftInserter.run(ThriftInserter.java:66)
      at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:329)
  Running WRITE with 40 threads for 50000 iteration
  Failed to connect over JMX; not collecting these stats
  type       total ops,    op/s,    pk/s,   row/s,    mean,     med,     .95,     .99,    .999,     max,   time,   stderr, errors,  gc: #,  max ms,  sum ms,  sdv ms,      mb
  UnavailableException()
  UnavailableException()
  UnavailableException()
  UnavailableException()
  UnavailableException()
  UnavailableException()
  UnavailableException()
  UnavailableException()
  UnavailableException()
  UnavailableException()
  java.io.IOException: Operation x10 on key(s) [394e35344e4b34383631]: Error executing: (UnavailableException)
  
      at org.apache.cassandra.stress.Operation.error(Operation.java:138)
      at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:116)
      at org.apache.cassandra.stress.operations.predefined.ThriftInserter.run(ThriftInserter.java:66)
      at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:329)
  FAILURE
  </code></pre>
  
  <p>Cassanra-stress with native cql3 mode:</p>
  
  <pre><code>cassandra-stress write n=50000 -rate threads=40 -mode native cql3 user=x password=x -node {hostnames}
  </code></pre>
  
  <p>Output:</p>
  
  <pre><code>WARN  14:16:12 Error while computing token map for keyspace system_auth with datacenter DC2: could not achieve replication factor 3 (found 1 replicas only), check your keyspace replication settings.
  WARN  14:16:12 Error while computing token map for keyspace system_auth with datacenter DC1: could not achieve replication factor 3 (found 1 replicas only), check your keyspace replication settings.
  Connected to cluster: COMECluster, max pending requests per connection 128, max connections per host 8
  Datatacenter: DC1; Host: /195.x.x.255; Rack: rack1
  Datatacenter: DC2; Host: intrinsitymetalsbanka/195.x.x.198; Rack: rack1
  Created keyspaces. Sleeping 5s for propagation.
  Sleeping 2s...
  Warming up WRITE with 62500 iterations...
  Failed to connect over JMX; not collecting these stats
  com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: intrinsitymetalsbanka/195.x.x.198:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_ONE (1 required but only 0 alive)))
  com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: intrinsitymetalsbanka/195.x.x.198:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_ONE (1 required but only 0 alive)))
  com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: intrinsitymetalsbanka/195.x.x.198:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_ONE (1 required but only 0 alive)))
  com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: intrinsitymetalsbanka/195.x.x.198:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_ONE (1 required but only 0 alive)))
  com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: intrinsitymetalsbanka/195.x.x.198:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_ONE (1 required but only 0 alive)))
  com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: intrinsitymetalsbanka/195.x.x.198:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_ONE (1 required but only 0 alive)))
  at org.apache.cassandra.stress.Operation.error(Operation.java:138)
  at org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:116)
  at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:101)
  at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:109)
  at org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:261)
  at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:321)
  </code></pre>