{"action":"created","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/1874","repository_url":"https://api.github.com/repos/scylladb/scylla","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/1874/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/1874/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/1874/events","html_url":"https://github.com/scylladb/scylla/issues/1874","id":191051524,"number":1874,"title":"[1.4.1 AMI] Longevity in > 1TB data cluster fails","user":{"login":"lmr","id":296807,"avatar_url":"https://avatars.githubusercontent.com/u/296807?v=3","gravatar_id":"","url":"https://api.github.com/users/lmr","html_url":"https://github.com/lmr","followers_url":"https://api.github.com/users/lmr/followers","following_url":"https://api.github.com/users/lmr/following{/other_user}","gists_url":"https://api.github.com/users/lmr/gists{/gist_id}","starred_url":"https://api.github.com/users/lmr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lmr/subscriptions","organizations_url":"https://api.github.com/users/lmr/orgs","repos_url":"https://api.github.com/users/lmr/repos","events_url":"https://api.github.com/users/lmr/events{/privacy}","received_events_url":"https://api.github.com/users/lmr/received_events","type":"User","site_admin":false},"labels":[{"id":160570956,"url":"https://api.github.com/repos/scylladb/scylla/labels/bug","name":"bug","color":"eb6420","default":true}],"state":"open","locked":false,"assignee":{"login":"tgrabiec","id":283695,"avatar_url":"https://avatars.githubusercontent.com/u/283695?v=3","gravatar_id":"","url":"https://api.github.com/users/tgrabiec","html_url":"https://github.com/tgrabiec","followers_url":"https://api.github.com/users/tgrabiec/followers","following_url":"https://api.github.com/users/tgrabiec/following{/other_user}","gists_url":"https://api.github.com/users/tgrabiec/gists{/gist_id}","starred_url":"https://api.github.com/users/tgrabiec/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tgrabiec/subscriptions","organizations_url":"https://api.github.com/users/tgrabiec/orgs","repos_url":"https://api.github.com/users/tgrabiec/repos","events_url":"https://api.github.com/users/tgrabiec/events{/privacy}","received_events_url":"https://api.github.com/users/tgrabiec/received_events","type":"User","site_admin":false},"assignees":[{"login":"tgrabiec","id":283695,"avatar_url":"https://avatars.githubusercontent.com/u/283695?v=3","gravatar_id":"","url":"https://api.github.com/users/tgrabiec","html_url":"https://github.com/tgrabiec","followers_url":"https://api.github.com/users/tgrabiec/followers","following_url":"https://api.github.com/users/tgrabiec/following{/other_user}","gists_url":"https://api.github.com/users/tgrabiec/gists{/gist_id}","starred_url":"https://api.github.com/users/tgrabiec/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tgrabiec/subscriptions","organizations_url":"https://api.github.com/users/tgrabiec/orgs","repos_url":"https://api.github.com/users/tgrabiec/repos","events_url":"https://api.github.com/users/tgrabiec/events{/privacy}","received_events_url":"https://api.github.com/users/tgrabiec/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/scylladb/scylla/milestones/16","html_url":"https://github.com/scylladb/scylla/milestones/1.5","labels_url":"https://api.github.com/repos/scylladb/scylla/milestones/16/labels","id":1794298,"number":16,"title":"1.5","description":"","creator":{"login":"tzach","id":170200,"avatar_url":"https://avatars.githubusercontent.com/u/170200?v=3","gravatar_id":"","url":"https://api.github.com/users/tzach","html_url":"https://github.com/tzach","followers_url":"https://api.github.com/users/tzach/followers","following_url":"https://api.github.com/users/tzach/following{/other_user}","gists_url":"https://api.github.com/users/tzach/gists{/gist_id}","starred_url":"https://api.github.com/users/tzach/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tzach/subscriptions","organizations_url":"https://api.github.com/users/tzach/orgs","repos_url":"https://api.github.com/users/tzach/repos","events_url":"https://api.github.com/users/tzach/events{/privacy}","received_events_url":"https://api.github.com/users/tzach/received_events","type":"User","site_admin":false},"open_issues":4,"closed_issues":29,"state":"open","created_at":"2016-05-29T08:33:14Z","updated_at":"2016-11-29T10:20:56Z","due_on":"2016-10-31T00:00:00Z","closed_at":null},"comments":9,"created_at":"2016-11-22T16:16:12Z","updated_at":"2016-11-30T10:58:38Z","closed_at":null,"body":"*Installation details*\r\nScylla version (or git commit hash): 1.4.1 AMI\r\nCluster size: 4 Nodes, i2.4xlarge\r\nOS (RHEL/CentOS/Ubuntu/AWS AMI): AWS AMI\r\n\r\nThis is a bug report for an issue observed when running a longevity test on a 4 Node\r\ncluster i2.2xlarge, with the intention of having a test on a cluster\r\nholding more than 1 TB of data per node.\r\n\r\nThe data size on nodes is indeed > 1 TB, as reported by nodetool:\r\n\r\n    [centos@ip-172-31-23-71 ~]$ nodetool status\r\n    Datacenter: datacenter1\r\n    =======================\r\n    Status=Up/Down\r\n    |/ State=Normal/Leaving/Joining/Moving\r\n    --  Address       Load       Tokens  Owns    Host ID\r\n                Rack\r\n    DN  172.31.23.74  2.21 TB    256     ?\r\n    898006be-01fc-49fc-8be3-f5d1648f4de6  rack1\r\n    DN  172.31.23.73  2.27 TB    256     ?\r\n    234aa709-386e-4d81-9651-7a46dcb9f961  rack1\r\n    UN  172.31.23.71  1.24 TB    256     ?\r\n    fe0dc69d-9b36-4670-ad45-2345fa0aa446  rack1\r\n\r\nWe ran 2 loaders, both running the following c-s commands:\r\n\r\n    2016-11-18 14:09:03,501 job              L0430 INFO |       |    -> stress_cmd: cassandra-stress write cl=QUORUM duration=10080m -schema\r\n    'replication(factor=3)' -port jmx=6868 -mode cql3 native -rate\r\n    threads=200 -col 'size=FIXED(1000) n=FIXED(1)' -pop seq=1..1250000000\r\n\r\nAfter about 18 hours, both c-s loads died:\r\n\r\n    2016-11-18 14:28:33,901 remote           L0194 DEBUG| [54.183.249.56]\r\n    [stdout] type,      total ops,    op/s,    pk/s,   row/s,    mean,\r\n    med,     .95,     .99,    .999,     max,   time,   stderr, errors,\r\n    gc: #,  max ms,  sum ms,  sdv ms,      mb\r\n    2016-11-18 14:28:35,070 remote           L0194 DEBUG| [54.183.249.56]\r\n    [stdout] total,        123236,  123192,  123192,  123192,     1.6,\r\n    1.4,     2.2,     3.2,    24.1,    47.3,    1.0,  0.00000,      0,\r\n     0,       0,       0,       0,       0\r\n    ...\r\n    2016-11-19 08:22:51,920 remote           L0194 DEBUG| [54.183.249.56]\r\n    [stdout] java.io.IOException: Operation x10 on key(s)\r\n    [374d32324e374c304b30]: Error executing: (NoHostAvailableException):\r\n    All host(s) tried for query failed (tried:\r\n    ip-172-31-23-71.us-west-1.compute.internal/172.31.23.71:9042\r\n    (com.datastax.driver.core.exceptions.UnavailableException: Not enough\r\n    replicas available for query at consistency QUORUM (2 required but\r\n    only 1 alive)))\r\n\r\nChecking the last nemesis to be executed:\r\n\r\n    2016-11-19 07:05:57,282 nemesis          L0272 DEBUG|\r\n    sdcm.nemesis.ChaosMonkey: CorruptThenRepair Node\r\n    scylla-1-4-ld-chaos-scylla-db-node-828d19cd-4 [54.193.122.180 |\r\n    172.31.23.74] (seed: False) duration -> 57382 s\r\n    2016-11-19 07:20:57,380 nemesis          L0054 INFO |\r\n    sdcm.nemesis.ChaosMonkey: Current Target: Node\r\n    scylla-1-4-ld-chaos-scylla-db-node-828d19cd-4 [54.193.122.180 |\r\n    172.31.23.74] (seed: False)\r\n    2016-11-19 07:20:57,381 nemesis          L0257 DEBUG|\r\n    sdcm.nemesis.ChaosMonkey: Start -> CorruptThenRepair Node\r\n    scylla-1-4-ld-chaos-scylla-db-node-828d19cd-4 [54.193.122.180 |\r\n    172.31.23.74] (seed: False)\r\n    2016-11-19 07:20:57,381 nemesis          L0160 INFO |\r\n    sdcm.nemesis.ChaosMonkey: CorruptThenRepair Node\r\n    scylla-1-4-ld-chaos-scylla-db-node-828d19cd-4 [54.193.122.180 |\r\n    172.31.23.74] (seed: False)\r\n    2016-11-19 07:20:58,242 nemesis          L0117 INFO |\r\n    sdcm.nemesis.ChaosMonkey: Kill all scylla processes in Node\r\n    scylla-1-4-ld-chaos-scylla-db-node-828d19cd-4 [54.193.122.180 |\r\n    172.31.23.74] (seed: False)\r\n    2016-11-19 07:20:58,468 nemesis          L0122 INFO |\r\n    sdcm.nemesis.ChaosMonkey: Waiting scylla services to be restarted\r\n    after we killed them...\r\n\r\nWe see that systemd attempted to start scylla but the services never\r\ncame up back online. Looking at systemd, it appears that the node is\r\nstill replaying commitlog after 3 days (72 hours):\r\n\r\n    ● scylla-server.service - Scylla Server\r\n       Loaded: loaded (/usr/lib/systemd/system/scylla-server.service;\r\n    enabled; vendor preset: disabled)\r\n       Active: activating (start) since Mon 2016-11-21 16:03:36 UTC; 5min ago\r\n      Process: 74983 ExecStopPost=/usr/lib/scylla/scylla_stop\r\n    (code=exited, status=0/SUCCESS)\r\n      Process: 27316 ExecStartPre=/usr/lib/scylla/scylla_prepare\r\n    (code=exited, status=0/SUCCESS)\r\n     Main PID: 27355 (scylla)\r\n       Status: \"replaying commit log\"\r\n       CGroup: /system.slice/scylla-server.service\r\n               └─27355 /usr/bin/scylla --log-to-syslog 1 --log-to-stdout 0\r\n    --default-log-level info --collectd-address=127.0.0.1:25826\r\n    --collectd=1 --collectd-poll-period 3000 --network-stack posix\r\n    --num-io-queues 16 --max-io-requests 128 --cpuset\r\n    1,2,3,4,5,6,7,9,10,11,12,13,14,15\r\n\r\n    Nov 21 16:03:49 ip-172-31-23-74 scylla[27355]:  [shard 13] compaction\r\n    - Compacted 4 sstables to\r\n    [/var/lib/scylla/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/system-compaction_history-ka-4381-Data.db:level=0,\r\n    ]. 30075 bytes to 30075 (~100% of original) in 479ms = 0.0598784MB/s.\r\n    ~1024 total partitions merged to 90.\r\n    Nov 21 16:03:49 ip-172-31-23-74 scylla[27355]:  [shard 12] compaction\r\n    - Compacted 4 sstables to\r\n    [/var/lib/scylla/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/system-compaction_history-ka-4380-Data.db:level=0,\r\n    ]. 27045 bytes to 27045 (~100% of original) in 598ms = 0.0431306MB/s.\r\n    ~1024 total partitions merged to 81.\r\n    Nov 21 16:03:49 ip-172-31-23-74 scylla[27355]:  [shard 8] compaction -\r\n    Compacted 4 sstables to\r\n    [/var/lib/scylla/data/system/schema_columns-296e9c049bec3085827dc17d3df2122a/system-schema_columns-ka-4306-Data.db:level=0,\r\n    ]. 112208 bytes to 28052 (~25% of original) in 741ms = 0.0361032MB/s.\r\n    ~1024 total partitions merged to 1.\r\n    Nov 21 16:03:49 ip-172-31-23-74 scylla[27355]:  [shard 8] compaction -\r\n    Compacted 4 sstables to\r\n    [/var/lib/scylla/data/system/schema_triggers-0359bc7171233ee19a4ab9dfb11fc125/system-schema_triggers-ka-2766-Data.db:level=0,\r\n    ]. 88 bytes to 22 (~25% of original) in 744ms = 2.82e-05MB/s. ~1024\r\n    total partitions merged to 1.\r\n    Nov 21 16:03:51 ip-172-31-23-74 scylla[27355]:  [shard 1] compaction -\r\n    Compacted 4 sstables to\r\n    [/var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-29989-Data.db:level=0,\r\n    ]. 102860660 bytes to 29716113 (~28% of original) in 2205ms =\r\n    12.8524MB/s. ~99456 total partitions merged to 28491.\r\n    Nov 21 16:03:51 ip-172-31-23-74 scylla[27355]:  [shard 1] compaction -\r\n    Compacting [/var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-5139-Data.db:level=0,\r\n    /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-10109-Data.db:level=0,\r\n    /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-14743-Data.db:level=0,\r\n    /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-19559-Data.db:level=0,\r\n    ]\r\n    Nov 21 16:03:54 ip-172-31-23-74 scylla[27355]:  [shard 13] compaction\r\n    - Compacted 4 sstables to\r\n    [/var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-30001-Data.db:level=0,\r\n    ]. 345148517 bytes to 95335415 (~27% of original) in 5412ms =\r\n    16.7995MB/s. ~331648 total partitions merged to 91405.\r\n    Nov 21 16:03:54 ip-172-31-23-74 scylla[27355]:  [shard 13] compaction\r\n    - Compacting [/var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-5081-Data.db:level=0,\r\n    /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-9869-Data.db:level=0,\r\n    /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-14629-Data.db:level=0,\r\n    /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-19403-Data.db:level=0,\r\n    ]\r\n    Nov 21 16:03:55 ip-172-31-23-74 scylla[27355]:  [shard 5] compaction -\r\n    Compacted 4 sstables to\r\n    [/var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-29993-Data.db:level=0,\r\n    ]. 437358061 bytes to 120532209 (~27% of original) in 6353ms =\r\n    18.0936MB/s. ~420096 total partitions merged to 115563.\r\n    Nov 21 16:03:55 ip-172-31-23-74 scylla[27355]:  [shard 5] compaction -\r\n    Compacting [/var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-5059-Data.db:level=0,\r\n    /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-9847-Data.db:level=0,\r\n    /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-14579-Data.db:level=0,\r\n    /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-19395-Data.db:level=0,\r\n    ]\r\n\r\nWe also have 2 coredumps generated, the backtraces are not very helpful:\r\n\r\n               PID: 114624 (scylla)\r\n               UID: 994 (scylla)\r\n               GID: 1001 (scylla)\r\n            Signal: 11 (SEGV)\r\n         Timestamp: Sat 2016-11-19 17:52:51 UTC (2min 22s ago)\r\n      Command Line: /usr/bin/scylla --log-to-syslog 1 --log-to-stdout 0\r\n    --default-log-level info --collectd-address=127.0.0.1:25826\r\n    --collectd=1 --collectd-poll-period 3000 --network-stack posix\r\n    --num-io-queues 16 --max-io-requests 128 --cpuset\r\n    1,2,3,4,5,6,7,9,10,11,12,13,14,15\r\n        Executable: /usr/bin/scylla\r\n     Control Group: /system.slice/scylla-server.service\r\n              Unit: scylla-server.service\r\n             Slice: system.slice\r\n           Boot ID: bb6117d9b57c4da88f20eeb9b672dc5a\r\n        Machine ID: 8ea3e6ea03b64d018847fcdf669935d4\r\n          Hostname: ip-172-31-23-74\r\n          Coredump:\r\n    /var/lib/systemd/coredump/core.scylla.994.bb6117d9b57c4da88f20eeb9b672dc5a.114624.1479577971000000\r\n           Message: Process 114624 (scylla) of user 994 dumped core.\r\n\r\n                    Stack trace of thread 114624:\r\n                    #0  0x000000000054c10d _ZN12future_stateIJEE3setEv\r\n    (/usr/bin/scylla)\r\n               PID: 122299 (scylla)\r\n               UID: 994 (scylla)\r\n               GID: 1001 (scylla)\r\n            Signal: 6 (ABRT)\r\n         Timestamp: Sat 2016-11-19 19:54:41 UTC (1min 57s ago)\r\n      Command Line: /usr/bin/scylla --log-to-syslog 1 --log-to-stdout 0\r\n    --default-log-level info --collectd-address=127.0.0.1:25826\r\n    --collectd=1 --collectd-poll-period 3000 --network-stack posix\r\n    --num-io-queues 16 --max-io-requests 128 --cpuset\r\n    1,2,3,4,5,6,7,9,10,11,12,13,14,15\r\n        Executable: /usr/bin/scylla\r\n     Control Group: /system.slice/scylla-server.service\r\n              Unit: scylla-server.service\r\n             Slice: system.slice\r\n           Boot ID: bb6117d9b57c4da88f20eeb9b672dc5a\r\n        Machine ID: 8ea3e6ea03b64d018847fcdf669935d4\r\n          Hostname: ip-172-31-23-74\r\n          Coredump:\r\n    /var/lib/systemd/coredump/core.scylla.994.bb6117d9b57c4da88f20eeb9b672dc5a.122299.1479585281000000\r\n           Message: Process 122299 (scylla) of user 994 dumped core.\r\n\r\n                    Stack trace of thread 122299:\r\n                    #0  0x00007f6b167a05f7 n/a (n/a)\r\n\r\nBoth core dumps happened on Node 4 (the node that had data corrupted\r\nand had services restarted).\r\n\r\nTo download the core dump, you may use\r\n\r\n    curl --user-agent [user-agent] scylladb-users-upload.s3.amazonaws.com core.scylla.994.bb6117d9b57c4da88f20eeb9b672dc5a.122299.1479585281000/core.scylla.994.bb6117d9b57c4da88f20eeb9b672dc5a.122299.1479585281000000.bz2 >  core.scylla.994.bb6117d9b57c4da88f20eeb9b672dc5a.122299.1479585281000000.bz2\r\n"},"comment":{"url":"https://api.github.com/repos/scylladb/scylla/issues/comments/263845051","html_url":"https://github.com/scylladb/scylla/issues/1874#issuecomment-263845051","issue_url":"https://api.github.com/repos/scylladb/scylla/issues/1874","id":263845051,"user":{"login":"tgrabiec","id":283695,"avatar_url":"https://avatars.githubusercontent.com/u/283695?v=3","gravatar_id":"","url":"https://api.github.com/users/tgrabiec","html_url":"https://github.com/tgrabiec","followers_url":"https://api.github.com/users/tgrabiec/followers","following_url":"https://api.github.com/users/tgrabiec/following{/other_user}","gists_url":"https://api.github.com/users/tgrabiec/gists{/gist_id}","starred_url":"https://api.github.com/users/tgrabiec/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tgrabiec/subscriptions","organizations_url":"https://api.github.com/users/tgrabiec/orgs","repos_url":"https://api.github.com/users/tgrabiec/repos","events_url":"https://api.github.com/users/tgrabiec/events{/privacy}","received_events_url":"https://api.github.com/users/tgrabiec/received_events","type":"User","site_admin":false},"created_at":"2016-11-30T10:58:38Z","updated_at":"2016-11-30T10:58:38Z","body":"Before the node startup times out, we can see that compaction is started before commitlog replay:\r\n\r\n```\r\nNov 19 17:37:57 info   |  [shard 8] database - Keyspace keyspace1: Reading CF standard1\r\nNov 19 17:37:57 info   |  [shard 13] database - Populating Keyspace keyspace1\r\nNov 19 17:37:57 info   |  [shard 13] database - Keyspace keyspace1: Reading CF standard1\r\nNov 19 17:38:03 info   |  [shard 0] compaction - Compacting [/var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-5194-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-9982-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace\r\n1-standard1-ka-14588-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-19628-Data.db:level=0, ]\r\nNov 19 17:38:03 info   |  [shard 1] compaction - Compacting [/var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-22163-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-22191-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspa\r\nce1-standard1-ka-22219-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-22261-Data.db:level=0, ]\r\nNov 19 17:38:03 info   |  [shard 5] compaction - Compacting [/var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-22167-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-22195-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspa\r\nce1-standard1-ka-22223-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-22265-Data.db:level=0, ]\r\nNov 19 17:38:03 info   |  [shard 3] compaction - Compacting [/var/lib/scylla/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/system-compaction_history-ka-465-Data.db:level=0, /var/lib/scylla/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/system-compaction_history-ka-479-Data.db:level=0, /var/lib/scylla/data/system/compaction_history-b4dbb7b4dc493\r\nfb5b3bfce6e434832ca/system-compaction_history-ka-535-Data.db:level=0, /var/lib/scylla/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/system-compaction_history-ka-731-Data.db:level=0, ]\r\nNov 19 17:38:03 info   |  [shard 12] compaction - Compacting [/var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-5094-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-9826-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspac\r\ne1-standard1-ka-14656-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-19444-Data.db:level=0, ]\r\nNov 19 17:38:03 info   |  [shard 7] compaction - Compacting [/var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-5075-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-9835-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace\r\n1-standard1-ka-14595-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-19439-Data.db:level=0, ]\r\nNov 19 17:38:03 info   |  [shard 13] compaction - Compacting [/var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-5081-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-9869-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspac\r\ne1-standard1-ka-14629-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-19403-Data.db:level=0, ]\r\nNov 19 17:38:03 info   |  [shard 8] compaction - Compacting [/var/lib/scylla/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/system-compaction_history-ka-414-Data.db:level=0, /var/lib/scylla/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/system-compaction_history-ka-470-Data.db:level=0, /var/lib/scylla/data/system/compaction_history-b4dbb7b4dc493\r\nfb5b3bfce6e434832ca/system-compaction_history-ka-708-Data.db:level=0, /var/lib/scylla/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/system-compaction_history-ka-736-Data.db:level=0, ]\r\nNov 19 17:38:03 info   |  [shard 6] compaction - Compacting [/var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-15728-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-16904-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspa\r\nce1-standard1-ka-18122-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-19298-Data.db:level=0, ]\r\nNov 19 17:38:03 info   |  [shard 11] compaction - Compacting [/var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-5093-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-9867-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspac\r\ne1-standard1-ka-14599-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-19443-Data.db:level=0, ]\r\nNov 19 17:38:03 info   |  [shard 10] compaction - Compacting [/var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-15690-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-16880-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keysp\r\nace1-standard1-ka-18070-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-19246-Data.db:level=0, ]\r\nNov 19 17:38:03 info   |  [shard 2] compaction - Compacting [/var/lib/scylla/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/system-compaction_history-ka-590-Data.db:level=0, /var/lib/scylla/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/system-compaction_history-ka-646-Data.db:level=0, /var/lib/scylla/data/system/compaction_history-b4dbb7b4dc493\r\nfb5b3bfce6e434832ca/system-compaction_history-ka-688-Data.db:level=0, /var/lib/scylla/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/system-compaction_history-ka-730-Data.db:level=0, ]\r\nNov 19 17:38:03 info   |  [shard 4] compaction - Compacting [/var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-21564-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-21802-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspa\r\nce1-standard1-ka-21844-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-22264-Data.db:level=0, ]\r\nNov 19 17:38:03 info   |  [shard 3] compaction - Compacting [/var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-15711-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-16915-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspa\r\nce1-standard1-ka-18091-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-19309-Data.db:level=0, ]\r\nNov 19 17:38:03 info   |  [shard 9] compaction - Compacting [/var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-5063-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-9795-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace\r\n1-standard1-ka-14597-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-19427-Data.db:level=0, ]\r\nNov 19 17:38:03 info   |  [shard 8] compaction - Compacting [/var/lib/scylla/data/system/schema_columns-296e9c049bec3085827dc17d3df2122a/system-schema_columns-ka-834-Data.db:level=0, /var/lib/scylla/data/system/schema_columns-296e9c049bec3085827dc17d3df2122a/system-schema_columns-ka-848-Data.db:level=0, /var/lib/scylla/data/system/schema_columns-296e9c049bec3085827dc17d3df2122a\r\n/system-schema_columns-ka-862-Data.db:level=0, /var/lib/scylla/data/system/schema_columns-296e9c049bec3085827dc17d3df2122a/system-schema_columns-ka-876-Data.db:level=0, ]\r\nNov 19 17:38:03 info   |  [shard 2] compaction - Compacting [/var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-15696-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-16886-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-18076-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-19266-Data.db:level=0, ]\r\nNov 19 17:38:03 info   |  [shard 8] compaction - Compacting [/var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-15744-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-16934-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-18124-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-19328-Data.db:level=0, ]\r\nNov 19 17:38:03 info   |  [shard 4] compaction - Compacted 4 sstables to [/var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-22292-Data.db:level=0, ]. 1356943 bytes to 671692 (~49% of original) in 146ms = 4.3875MB/s. ~2048 total partitions merged to 644.\r\nNov 19 17:38:03 info   |  [shard 4] compaction - Compacting [/var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-15712-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-16916-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-18092-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-19296-Data.db:level=0, ]\r\nNov 19 17:38:03 info   |  [shard 0] commitlog_replayer - Replaying /var/lib/scylla/commitlog/CommitLog-1-234187180626851859.log, /var/lib/scylla/commitlog/CommitLog-1-18014398513068057.log, /var/lib/scylla/commitlog/CommitLog-1-216172782117369874.log, /var/lib/scylla/commitlog/CommitLog-1-3586071.log, /var/lib/scylla/commitlog/CommitLog-1-216172782117369875.log, /var/lib/scylla/commitlog/CommitLog-1-108086391060477971.log, /var/lib/scylla/commitlog/CommitLog-1-198158383607887895.log, /var/lib/scylla/commitlog/CommitLog-1-3586073.log, /var/lib/scylla/commitlog/CommitLog-1-108086391060477979.log, /var/lib/scylla/commitlog/CommitLog-1-126100789569959956.log, /var/lib/scylla/commitlog/CommitLog-1-162129586588923933.log, /var/lib/scylla/commitlog/CommitLog-1-36028797022550040.log, /var/lib/scylla/commitlog/CommitLog-1-108086391060477975.log, /var/lib/scylla/commitlog/CommitLog-1-198158383607887894.log, /var/lib/scylla/commitlog/CommitLog-1-54043195532032022.log, /var/lib/scylla/commitlog/CommitLog-1-198158383607887898.log, /var/lib/scylla/commitlog/CommitLog-1-54043195532032023.log, /var/lib/scylla/commitlog/CommitLog-1-3586074.log, /var/lib/scylla/commitlog/CommitLog-1-180143985098405905.log, /var/lib/scylla/commitlog/CommitLog-1-36028797022550039.log, /var/lib/scylla/commitlog/CommitLog-1-126100789569959957.log, /var/lib/scylla/commitlog/CommitLog-1-234187180626851864.log, /var/lib/scylla/commitlog/CommitLog-1-126100789569959963.log, /var/lib/scylla/commitlog/CommitLog-1-126100789569959955.log, /var/lib/scylla/commitlog/CommitLog-1-108086391060477973.log, /var/lib/scylla/commitlog/CommitLog-1-162129586588923926.log, /var/lib/scylla/commitlog/CommitLog-1-54043195532032026.log, /var/lib/scylla/commitlog/CommitLog-1-108086391060477972.log, /var/lib/scylla/commitlog/CommitLog-1-18014398513068058.log, /var/lib/scylla/commitlog/CommitLog-1-108086391060477976.log, /var/lib/scylla/commitlog/CommitLog-1-18014398513068056.log, /var/lib/scylla/commitlog/CommitLog-1-162129586588923927.log, /var/lib/scylla/commitlog/CommitLog-1-126100789569959962.log, /var/lib/scylla/commitlog/CommitLog-1-3586072.log, /var/lib/scylla/commitlog/CommitLog-1-36028797022550042.log, /var/lib/scylla/commitlog/CommitLog-1-180143985098405901.log, /var/lib/scylla/commitlog/CommitLog-1-180143985098405900.log, /var/lib/scylla/commitlog/CommitLog-1-234187180626851860.log, /var/lib/scylla/commitlog/CommitLog-1-18014398513068059.log, /var/lib/scylla/commitlog/CommitLog-1-180143985098405904.log, /var/lib/scylla/commitlog/CommitLog-1-234187180626851861.log, /var/lib/scylla/commitlog/CommitLog-1-18014398513068060.log, /var/lib/scylla/commitlog/CommitLog-1-162129586588923931.log, /var/lib/scylla/commitlog/CommitLog-1-234187180626851865.log, /var/lib/scylla/commitlog/CommitLog-1-108086391060477974.log, /var/lib/scylla/commitlog/CommitLog-1-108086391060477977.log, /var/lib/scylla/commitlog/CommitLog-1-162129586588923932.log, /var/lib/scylla/commitlog/CommitLog-1-162129586588923928.log, /var/lib/scylla/commitlog/CommitLog-1-72057594041514010.log, /var/lib/scylla/commitlog/CommitLog-1-234187180626851858.log, /var/lib/scylla/commitlog/CommitLog-1-36028797022550041.log, /var/lib/scylla/commitlog/CommitLog-1-108086391060477978.log, /var/lib/scylla/commitlog/CommitLog-1-198158383607887897.log, /var/lib/scylla/commitlog/CommitLog-1-126100789569959961.log, /var/lib/scylla/commitlog/CommitLog-1-3586069.log, /var/lib/scylla/commitlog/CommitLog-1-54043195532032025.log, /var/lib/scylla/commitlog/CommitLog-1-36028797022550036.log, /var/lib/scylla/commitlog/CommitLog-1-90071992550995991.log, /var/lib/scylla/commitlog/CommitLog-1-90071992550995992.log, /var/lib/scylla/commitlog/CommitLog-1-3586070.log, /var/lib/scylla/commitlog/CommitLog-1-180143985098405907.log, /var/lib/scylla/commitlog/CommitLog-1-234187180626851862.log, /var/lib/scylla/commitlog/CommitLog-1-198158383607887896.log, /var/lib/scylla/commitlog/CommitLog-1-36028797022550033.log, /var/lib/scylla/commitlog/CommitLog-1-144115188079441944.log, /var/lib/scylla/commitlog/CommitLog-1-198158383607887893.log, /var/lib/scylla/commitlog/CommitLog-1-\r\nNov 19 17:38:04 info   |  [shard 3] compaction - Compacted 4 sstables to [/var/lib/scylla/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/system-compaction_history-ka-745-Data.db:level=0, ]. 6689 bytes to 6689 (~100% of original) in 632ms = 0.0100936MB/s. ~1024 total partitions merged to 20.\r\nNov 19 17:38:04 info   |  [shard 2] compaction - Compacted 4 sstables to [/var/lib/scylla/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/system-compaction_history-ka-744-Data.db:level=0, ]. 7654 bytes to 7654 (~100% of original) in 783ms = 0.00932238MB/s. ~1024 total partitions merged to 23.\r\nNov 19 17:38:04 info   |  [shard 8] compaction - Compacted 4 sstables to [/var/lib/scylla/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/system-compaction_history-ka-750-Data.db:level=0, ]. 3679 bytes to 3679 (~100% of original) in 1048ms = 0.00334787MB/s. ~1024 total partitions merged to 11.\r\nNov 19 17:38:04 info   |  [shard 8] compaction - Compacted 4 sstables to [/var/lib/scylla/data/system/schema_columns-296e9c049bec3085827dc17d3df2122a/system-schema_columns-ka-890-Data.db:level=0, ]. 112208 bytes to 28052 (~25% of original) in 1047ms = 0.0255515MB/s. ~1024 total partitions merged to 1.\r\nNov 19 17:38:05 info   |  [shard 1] compaction - Compacted 4 sstables to [/var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-22289-Data.db:level=0, ]. 82576396 bytes to 22737400 (~27% of original) in 1630ms = 13.3031MB/s. ~80000 total partitions merged to 21800.\r\nNov 19 17:38:05 info   |  [shard 1] compaction - Compacting [/var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-5139-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-10109-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-14743-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-19559-Data.db:level=0, ]\r\nNov 19 17:38:09 info   |  [shard 5] compaction - Compacted 4 sstables to [/var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-22293-Data.db:level=0, ]. 361684239 bytes to 100930067 (~27% of original) in 5599ms = 17.1914MB/s. ~347648 total partitions merged to 96769.\r\nNov 19 17:38:09 info   |  [shard 5] compaction - Compacting [/var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-5059-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-9847-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-14579-Data.db:level=0, /var/lib/scylla/data/keyspace1/standard1-acec3970ad9a11e683a1000000000002/keyspace1-standard1-ka-19395-Data.db:level=0, ]\r\nERROR 17:45:55 [Control connection] Cannot connect to any host, scheduling retry in 600000 milliseconds\r\nERROR 17:45:55 [Control connection] Cannot connect to any host, scheduling retry in 600000 milliseconds\r\nNov 19 17:52:51 warning| scylla-server.service start operation timed out. Terminating.\r\nNov 19 17:52:51 info   |  [shard 0] compaction_manager - Asked to stop\r\nNov 19 17:52:51 info   |  [shard 5] compaction_manager - Asked to stop\r\nNov 19 17:52:51 info   |  [shard 8] compaction_manager - Asked to stop\r\nNov 19 17:52:51 info   |  [shard 2] compaction_manager - Asked to stop\r\nNov 19 17:52:51 info   |  [shard 11] compaction_manager - Asked to stop\r\nNov 19 17:52:51 info   |  [shard 6] compaction_manager - Asked to stop\r\nNov 19 17:52:51 info   |  [shard 4] compaction_manager - Asked to stop\r\nNov 19 17:52:51 info   |  [shard 7] compaction_manager - Asked to stop\r\nNov 19 17:52:51 info   |  [shard 9] compaction_manager - Asked to stop\r\nNov 19 17:52:51 info   |  [shard 3] compaction_manager - Asked to stop\r\nNov 19 17:52:51 info   |  [shard 12] compaction_manager - Asked to stop\r\nNov 19 17:52:51 info   |  [shard 1] compaction_manager - Asked to stop\r\nNov 19 17:52:51 info   |  [shard 13] compaction_manager - Asked to stop\r\nNov 19 17:52:51 info   |  [shard 10] compaction_manager - Asked to stop\r\nNov 19 17:52:51 info   |  [shard 11] compaction_manager - Stopped\r\nNov 19 17:52:51 info   |  [shard 4] compaction_manager - Stopped\r\nNov 19 17:52:51 info   |  [shard 9] compaction_manager - Stopped\r\nNov 19 17:52:51 info   |  [shard 7] compaction_manager - Stopped\r\nNov 19 17:52:51 info   |  [shard 0] compaction_manager - Stopped\r\nNov 19 17:52:51 info   |  [shard 12] compaction_manager - Stopped\r\nNov 19 17:52:51 info   |  [shard 13] compaction_manager - Stopped\r\nNov 19 17:52:51 info   |  [shard 10] compaction_manager - Stopped\r\nNov 19 17:52:51 info   |  [shard 6] compaction_manager - Stopped\r\nNov 19 17:52:51 info   |  [shard 1] compaction_manager - Stopped\r\nNov 19 17:52:51 info   |  [shard 3] compaction_manager - Stopped\r\nNov 19 17:52:51 info   |  [shard 8] compaction_manager - Stopped\r\nNov 19 17:52:51 info   |  [shard 5] compaction_manager - Stopped\r\nNov 19 17:52:51 info   | Segmentation fault on shard 0.\r\nNov 19 17:52:51 info   | Backtrace:\r\nNov 19 17:52:51 info   |  [shard 2] compaction_manager - Stopped\r\nNov 19 17:52:51 info   | 0x00000000004e2e6a\r\nNov 19 17:52:51 info   | 0x00000000004e2f13\r\nNov 19 17:52:51 info   | 0x00007f5b91a2e0ff\r\nNov 19 17:52:51 info   | 0x000000000054c10c\r\nNov 19 17:52:51 info   | 0x000000000055e00e\r\nNov 19 17:52:51 info   | 0x000000000056d8ed\r\nNov 19 17:52:51 info   | 0x00000000004e4ace\r\nNov 19 17:52:51 info   | 0x000000000052f872\r\nNov 19 17:52:51 info   | 0x000000000059b26c\r\nNov 19 17:52:51 info   | 0x000000000047637c\r\nNov 19 17:52:51 info   | 0x00007f5b9167eb14\r\nNov 19 17:52:51 info   | 0x00000000004dffa0\r\n\r\n```"}}
{"actor":{"display_login":"tgrabiec"}}