After high traffic Cassandra WriteTimeoutException <p>I'm experimenting with cassandra to use it for storing tracking data. I process get requests (300 req/sec) with tracking information through django with uwsgi on nginx. The django application writes to cassandra (single cluster with 3 RF) directly. For this im taking the cassandra-driver (2.5.0).</p> <p>After restarting all relevant services the whole thing works for about two hours. Then the server goes down and produces a server error 500.</p> <p>Currently i don't know where the bottleneck is. Syslog show normal I/O activity on my system also CPU is just used by 32%.</p> <p>I'm working primarily with counters. My tables look like this:</p> <pre><code>CREATE TABLE rollup_hour_counter ( name text, player_id text, hour timestamp, "count" counter, PRIMARY KEY ((name, player_id), hour) ) </code></pre> <p>My code to write into cassandra looks like this:</p> <pre><code>rows = session.execute('UPDATE rollup_hour_counter ' 'SET count = count + 1' 'WHERE player_id=\'%s\' AND hour=\'%s:00:00\' AND name = \'name\'' % (player_uuid, date_now_hour)) </code></pre> <p>I did some research on the errors in the log files. </p> <p>uwsgi logs show:</p> <blockquote> <p>Thu Apr 16 07:44:04 2015 - uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 296] during GET /t/i/7bcdd5e185fd4608b0d3c3451f5ec56a/ (146.58.126.229)</p> </blockquote> <p>cassandra log shows:</p> <blockquote> <p>WARN [SharedPool-Worker-12] 2015-04-16 07:44:50,424 AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-12,5,main]: {} java.lang.RuntimeException: org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out - received only 0 responses. at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2174) ~[apache-cassandra-2.1.4.jar:2.1.4] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_76] at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-2.1.4.jar:2.1.4] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.4.jar:2.1.4] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_76] Caused by: org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out - received only 0 responses. at org.apache.cassandra.db.CounterMutation.grabCounterLocks(CounterMutation.java:146) ~[apache-cassandra-2.1.4.jar:2.1.4] at org.apache.cassandra.db.CounterMutation.apply(CounterMutation.java:122) ~[apache-cassandra-2.1.4.jar:2.1.4] at org.apache.cassandra.db.CounterMutation.apply(CounterMutation.java:122) ~[apache-cassandra-2.1.4.jar:2.1.4] at org.apache.cassandra.service.StorageProxy$8.runMayThrow(StorageProxy.java:1147) ~[apache-cassandra-2.1.4.jar:2.1.4] at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2171) ~[apache-cassandra-2.1.4.jar:2.1.4] ... 4 common frames omitted</p> </blockquote> <p>nginx logs show: </p> <blockquote> <p>2015/04/16 07:43:57 [error] 2111#0: *268738 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 82.115.124.102, server: server.someservice.com, request: "GET [...] HTTP/1.1", upstream: "uwsgi://unix:///tmp/track.sock", host: "server2.someotherservice.com", referrer: "[..]"</p> </blockquote> <p>So first I thought that the uwsgi processes and sockets were responsible. I added some worker processes increased buffer size but no help.</p> <p>Second i tuned some cassandra settings but no help:</p> <p>My cassandra settings look like this:</p> <pre><code>num_tokens: 256 hinted_handoff_enabled: true max_hint_window_in_ms: 10800000 # 3 hours hinted_handoff_throttle_in_kb: 1024 max_hints_delivery_threads: 2 batchlog_replay_throttle_in_kb: 1024 authenticator: AllowAllAuthenticator authorizer: AllowAllAuthorizer permissions_validity_in_ms: 2000 partitioner: org.apache.cassandra.dht.Murmur3Partitioner data_file_directories: - /var/lib/cassandra/data commitlog_directory: /var/lib/cassandra/commitlog disk_failure_policy: stop commit_failure_policy: stop key_cache_size_in_mb: key_cache_save_period: 14400 row_cache_size_in_mb: 0 row_cache_save_period: 0 counter_cache_size_in_mb: counter_cache_save_period: 7200 saved_caches_directory: /var/lib/cassandra/saved_caches commitlog_sync: periodic commitlog_sync_period_in_ms: 10000 commitlog_segment_size_in_mb: 32 seed_provider: - class_name: org.apache.cassandra.locator.SimpleSeedProvider parameters: # seeds is actually a comma-delimited list of addresses. # Ex: "&lt;ip1&gt;,&lt;ip2&gt;,&lt;ip3&gt;" - seeds: "127.0.0.1" concurrent_reads: 96 concurrent_writes: 96 concurrent_counter_writes: 96 file_cache_size_in_mb: 1512 memtable_allocation_type: heap_buffers memtable_flush_writers: 96 index_summary_capacity_in_mb: index_summary_resize_interval_in_minutes: 60 trickle_fsync: false trickle_fsync_interval_in_kb: 10240 storage_port: 7000 ssl_storage_port: 7001 listen_address: localhost start_native_transport: true native_transport_port: 9042 start_rpc: true rpc_address: localhost rpc_port: 9160 rpc_keepalive: true rpc_server_type: sync rpc_min_threads: 60 rpc_max_threads: 96 thrift_framed_transport_size_in_mb: 15 memtable_flush_after_mins: 60 incremental_backups: false snapshot_before_compaction: false auto_snapshot: true tombstone_warn_threshold: 1000 tombstone_failure_threshold: 100000 column_index_size_in_kb: 64 batch_size_warn_threshold_in_kb: 5 compaction_throughput_mb_per_sec: 16 sstable_preemptive_open_interval_in_mb: 50 read_request_timeout_in_ms: 5000 range_request_timeout_in_ms: 10000 write_request_timeout_in_ms: 2000 counter_write_request_timeout_in_ms: 5000 cas_contention_timeout_in_ms: 1000 truncate_request_timeout_in_ms: 60000 request_timeout_in_ms: 10000 cross_node_timeout: false endpoint_snitch: SimpleSnitch dynamic_snitch_update_interval_in_ms: 100 dynamic_snitch_reset_interval_in_ms: 600000 dynamic_snitch_badness_threshold: 0.1 request_scheduler: org.apache.cassandra.scheduler.NoScheduler server_encryption_options: internode_encryption: none keystore: conf/.keystore keystore_password: cassandra truststore: conf/.truststore truststore_password: cassandra # More advanced defaults below: # protocol: TLS # algorithm: SunX509 # store_type: JKS # cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA] # require_client_auth: false client_encryption_options: enabled: false keystore: conf/.keystore keystore_password: cassandra # require_client_auth: false # Set trustore and truststore_password if require_client_auth is true # truststore: conf/.truststore # truststore_password: cassandra # More advanced defaults below: # protocol: TLS # algorithm: SunX509 # store_type: JKS # cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA] internode_compression: all inter_dc_tcp_nodelay: false </code></pre> <p>Does someone have an idea where the bottleneck is?</p>