{"action":"closed","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/1353","repository_url":"https://api.github.com/repos/scylladb/scylla","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/1353/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/1353/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/1353/events","html_url":"https://github.com/scylladb/scylla/issues/1353","id":160591673,"number":1353,"title":"Node running out of disk space does not disconnect from the cluster","user":{"login":"slivne","id":3465480,"avatar_url":"https://avatars.githubusercontent.com/u/3465480?v=3","gravatar_id":"","url":"https://api.github.com/users/slivne","html_url":"https://github.com/slivne","followers_url":"https://api.github.com/users/slivne/followers","following_url":"https://api.github.com/users/slivne/following{/other_user}","gists_url":"https://api.github.com/users/slivne/gists{/gist_id}","starred_url":"https://api.github.com/users/slivne/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slivne/subscriptions","organizations_url":"https://api.github.com/users/slivne/orgs","repos_url":"https://api.github.com/users/slivne/repos","events_url":"https://api.github.com/users/slivne/events{/privacy}","received_events_url":"https://api.github.com/users/slivne/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/scylladb/scylla/labels/bug","name":"bug","color":"eb6420"},{"url":"https://api.github.com/repos/scylladb/scylla/labels/showstopper","name":"showstopper","color":"e11d21"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2016-06-16T07:03:48Z","updated_at":"2016-06-16T10:34:27Z","closed_at":"2016-06-16T10:34:27Z","body":"Scylla version: 1.1.3-rc1\r\n\r\nthe node does close down the cql connection\r\n\r\n```\r\nData.db:level=1, /var/lib/scylla/data/keyspace1/standard1-620372d0338c11e68b39000000000001/keyspace1-standard1-ka-274-Data.db:level=0, ]\r\nJun 16 06:56:13 ip-10-37-213-60 scylla[19227]:  [shard 0] storage_service - Shutting down communications due to I/O errors until operator intervention\r\nJun 16 06:56:13 ip-10-37-213-60 scylla[19227]:  [shard 0] database - failed to write sstable: storage_io_error (Storage io error errno: 28)\r\nJun 16 06:56:13 ip-10-37-213-60 scylla[19227]:  [shard 0] compaction - Compaction found an exception when writing sstable /var/lib/scylla/data/keyspace1/standard1-620372d0338c11e68b39000000000001/keyspace1-standard1-ka-748-Data.db : storage_io_error (Storage io error errno: 28)\r\nJun 16 06:56:13 ip-10-37-213-60 scylla[19227]:  [shard 0] compaction_manager - compaction failed: read exception: seastar::broken_pipe_exception (Broken pipe), write_exception: storage_io_error (Storage io error errno: 28)\r\nJun 16 06:56:13 ip-10-37-213-60 scylla[19227]:  [shard 0] compaction_manager - compaction task handler sleeping for 5 seconds\r\nJun 16 06:56:14 ip-10-37-213-60 scylla[19227]:  [shard 0] storage_service - CQL server stopped\r\n```\r\n\r\nthe node does not close down communication with other nodes\r\n\r\n```\r\non the other node in the cluster\r\n\r\nubuntu@ip-10-63-76-132:~$ nodetool status\r\nDatacenter: datacenter1\r\n=======================\r\nStatus=Up/Down\r\n|/ State=Normal/Leaving/Joining/Moving\r\n--  Address       Load       Tokens  Owns    Host ID                               Rack\r\nUN  10.37.213.60  69.09 GB   256     ?       7ed5a3cd-1ea5-4280-b108-f3c88fafda6a  rack1\r\nUN  10.63.76.132  74.33 GB   256     ?       5816f8fd-d595-42b4-82c6-e02ebf5cc1fd  rack1\r\n```\r\n\r\n\r\nhow to reproduce\r\n\r\ncreate a cluster of 2 servers\r\none of them with a smaller disk\r\nrun a load using\r\n\r\n``` cassandra-stress write n=50000000 -mode cql3 native -schema compaction\\(strategy=LeveledCompactionStrategy\\) compression=LZ4Compressor \"replication(factor=2)\" -col size=FIXED\\(2048\\) n=FIXED\\(5\\) -pop seq=1..10000000 -node `hostname -i`\r\n```\r\n\r\nwhen the node with the lower disk space runs out of disk - try to connect with cql - its not possible\r\n\r\ncheck with netstat that connection to other nodes is down \r\n\r\ncheck with nodetool status that other nodes think the node is down\r\n\r\n```\r\nubuntu@ip-10-37-213-60:~$ netstat -a -n | grep 7000\r\ntcp        0      0 10.37.213.60:7000       0.0.0.0:*               LISTEN     \r\ntcp        0      0 10.37.213.60:7000       10.63.76.132:48245      ESTABLISHED\r\ntcp        0      0 10.37.213.60:7000       10.63.76.132:45150      ESTABLISHED\r\ntcp        0      0 10.37.213.60:40047      10.63.76.132:7000       ESTABLISHED\r\ntcp        0      0 10.37.213.60:39236      10.63.76.132:7000       ESTABLISHED\r\ntcp        0      0 10.37.213.60:7000       10.63.76.132:48610      ESTABLISHED\r\ntcp        0      0 10.37.213.60:7000       10.63.76.132:58978      ESTABLISHED\r\ntcp        0      0 10.37.213.60:41012      10.63.76.132:7000       ESTABLISHED\r\ntcp        0      0 10.37.213.60:7000       10.63.76.132:52343      ESTABLISHED\r\ntcp        0      0 10.37.213.60:7000       10.63.76.132:43436      ESTABLISHED\r\ntcp        0      0 10.37.213.60:7000       10.63.76.132:56520      ESTABLISHED\r\ntcp        0      0 10.37.213.60:7000       10.63.76.132:59013      ESTABLISHED\r\ntcp        0      0 10.37.213.60:38189      10.63.76.132:7000       ESTABLISHED\r\ntcp        0      0 10.37.213.60:7000       10.63.76.132:50757      ESTABLISHED\r\ntcp        0      0 10.37.213.60:60218      10.63.76.132:7000       ESTABLISHED\r\ntcp        0      0 10.37.213.60:7000       10.63.76.132:51041      ESTABLISHED\r\ntcp        0      0 10.37.213.60:7000       10.63.76.132:33913      ESTABLISHED\r\ntcp        0      0 10.37.213.60:7000       10.63.76.132:60524      ESTABLISHED\r\nubuntu@ip-10-37-213-60:~$ \r\n```"}}
{"actor":{"display_login":"penberg"}}