{"action":"created","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/3524","repository_url":"https://api.github.com/repos/scylladb/scylla","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/3524/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/3524/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/3524/events","html_url":"https://github.com/scylladb/scylla/issues/3524","id":333057099,"node_id":"MDU6SXNzdWUzMzMwNTcwOTk=","number":3524,"title":"Add disk pressure and disk eviction","user":{"login":"nyh","id":584227,"node_id":"MDQ6VXNlcjU4NDIyNw==","avatar_url":"https://avatars3.githubusercontent.com/u/584227?v=4","gravatar_id":"","url":"https://api.github.com/users/nyh","html_url":"https://github.com/nyh","followers_url":"https://api.github.com/users/nyh/followers","following_url":"https://api.github.com/users/nyh/following{/other_user}","gists_url":"https://api.github.com/users/nyh/gists{/gist_id}","starred_url":"https://api.github.com/users/nyh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nyh/subscriptions","organizations_url":"https://api.github.com/users/nyh/orgs","repos_url":"https://api.github.com/users/nyh/repos","events_url":"https://api.github.com/users/nyh/events{/privacy}","received_events_url":"https://api.github.com/users/nyh/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-06-17T11:55:00Z","updated_at":"2018-06-18T13:59:14Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"We have a concept of memory pressure, and memory eviction. In this issue I propose to have a similar concept of **disk space pressure** and disk eviction: When we are getting close to running out of space on a disk, we ask the various disk evictors to start saving space. For example, in a disk shared by data, commitlog and hints, the data, commitlog and hint evictors can:\r\n\r\n1.  Data: Tell the compaction controller to increase the compaction cpu and disk share, to allow compaction to save space. May also apply other emergency tweaks to the compaction strategy - e.g., starting compaction to see which sstables are most redundant (see issue #2132), compacting together even fewer sstables than usual, compacting even single sstables, etc. Watch out not to do this in the classic STCS because this strategy may need huge temporary space during compaction, and just make the disk pressure worse. \r\n\r\n2. Commitlog: Increase the memory flush shares, so more memory can be flushed to disk and commitlog data removed. Note that the benefit of this is not obvious, because commitlog files are deleted but sstables are written. So it depends what is the relative size of commitlog vs. sstable. However, another benefit of replacing commitlog with sstable is that it might allow a compaction (with older data) to save some more space.\r\n\r\n3. Hints: we can drop the oldest hint files. This may sound risky (we will need repair when a node is down for a very long time) but it may be the opposite: 1. This feature will allow us to keep hints for a very long time when disk space is *plenty* and not artificially limit it to say 10% of the disk. 2. This feature in the case of a real disk-space emergency will cause hints to be lost quickly (we can warn the db admin about it) but that's better than losing data when we run out of disk!\r\n\r\nThe idea here is to replace ad-hoc heuristics like \"hints shouldn't take more than 10% of the disk\" that we now have (see issue #3516 ) by something more robust."},"comment":{"url":"https://api.github.com/repos/scylladb/scylla/issues/comments/398063604","html_url":"https://github.com/scylladb/scylla/issues/3524#issuecomment-398063604","issue_url":"https://api.github.com/repos/scylladb/scylla/issues/3524","id":398063604,"node_id":"MDEyOklzc3VlQ29tbWVudDM5ODA2MzYwNA==","user":{"login":"vladzcloudius","id":6103981,"node_id":"MDQ6VXNlcjYxMDM5ODE=","avatar_url":"https://avatars3.githubusercontent.com/u/6103981?v=4","gravatar_id":"","url":"https://api.github.com/users/vladzcloudius","html_url":"https://github.com/vladzcloudius","followers_url":"https://api.github.com/users/vladzcloudius/followers","following_url":"https://api.github.com/users/vladzcloudius/following{/other_user}","gists_url":"https://api.github.com/users/vladzcloudius/gists{/gist_id}","starred_url":"https://api.github.com/users/vladzcloudius/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vladzcloudius/subscriptions","organizations_url":"https://api.github.com/users/vladzcloudius/orgs","repos_url":"https://api.github.com/users/vladzcloudius/repos","events_url":"https://api.github.com/users/vladzcloudius/events{/privacy}","received_events_url":"https://api.github.com/users/vladzcloudius/received_events","type":"User","site_admin":false},"created_at":"2018-06-18T13:59:14Z","updated_at":"2018-06-18T13:59:14Z","author_association":"CONTRIBUTOR","body":"@nyh I like the idea in general but I'd like to clarify a few points.\r\n\r\nIn the context of HH this idea (of a back-pressure/eviction) has been implemented from day 1: if we run out of space we stop the producer and leave the consumer (sender) open. Once hints are sent out the corresponding segments are deleted thereby freeing the disk space.\r\n\r\nThe only difference between this and what you are suggesting is that you want to delete the oldest segments unconditionally while in the current solution we virtually delete the newest segment (we simply don't let them to be created by blocking the producer).\r\n\r\nIn the context of HH it's not obvious to me which strategy is preferable since segments are not per-KS or per-CF - they are per destination node and may contain mutations to different tables. Therefore it's not necessarily that the newest segments are going to contain the latest mutations to all KS-es. While from the codding perspective the current approach is a lot easier to implement.\r\n\r\nAs for the indication of hints being dropped - there is a corresponding metrics that will be bumped up for every hint that we refused to store (we don't differentiate between different classes reasons for a hint drop at the moment but we can easily add this if needed).\r\n\r\nOne more thing that we will definitely have to implement if/when we have this is the \"prioritization\" of different disk space consumers. For instance, you really don't want to let HH consume as much space as it wants simply because it may eat up all your disk space very fast:  hints are stored in a commitlog-like structured files that are not compacted, and if some actively written replica goes down then you'll get a storm of hints to that node. This is exactly where this 10% limitation came from (the scenario above has been pointed out by Avi back then).\r\n\r\nThis means that you'll have to come up with something similar to that \"10% voodoo\", and it may end up to be very similar to what we have right now."}}
{"actor":{"display_login":"vladzcloudius"}}