{"action":"created","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/1295","repository_url":"https://api.github.com/repos/scylladb/scylla","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/1295/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/1295/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/1295/events","html_url":"https://github.com/scylladb/scylla/issues/1295","id":157379180,"number":1295,"title":"Limit number of parallel compactions of different CFs","user":{"login":"nyh","id":584227,"avatar_url":"https://avatars.githubusercontent.com/u/584227?v=3","gravatar_id":"","url":"https://api.github.com/users/nyh","html_url":"https://github.com/nyh","followers_url":"https://api.github.com/users/nyh/followers","following_url":"https://api.github.com/users/nyh/following{/other_user}","gists_url":"https://api.github.com/users/nyh/gists{/gist_id}","starred_url":"https://api.github.com/users/nyh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nyh/subscriptions","organizations_url":"https://api.github.com/users/nyh/orgs","repos_url":"https://api.github.com/users/nyh/repos","events_url":"https://api.github.com/users/nyh/events{/privacy}","received_events_url":"https://api.github.com/users/nyh/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2016-05-29T10:14:20Z","updated_at":"2016-05-31T12:50:25Z","closed_at":null,"body":"I I understand correctly, starting in commit 5aeeb0b3e8859a9687bc6fc3b41d4d46fd4ac93f, parallel compaction of different CFs is no longer limited: If you have 10 different CFs, you might have 10 compactions active in parallel (even more if we decide to do parallel compactions on on CF).\r\n\r\nThe downside of parallel compaction is that it requires quite a bit of memory, such as buffers for reading all the sstables, and when mutations are very large (before we support wide partitions properly), holding multiple huge mutations in memory.\r\n\r\nIn some cases, parallel compaction is important (e.g., to not block small compactions while a very large compaction is in progress), but I don't understand why it might be important to do an large number of compactions of same size CFs in parallel. As an example, consider two CFs that want to start a large compaction which would each take (separately) 10 minutes. The current code may start both together, and they will run 20 minutes. I'm suggesting it's better to start just one, have it finish in 10 minutes, and then start the second, that will finish after an additional 10 minutes.\r\nJust don't forget that even with my suggestion, while a long compaction is running on any CF, we should allow a short compaction to start (on any CF)."},"comment":{"url":"https://api.github.com/repos/scylladb/scylla/issues/comments/222678492","html_url":"https://github.com/scylladb/scylla/issues/1295#issuecomment-222678492","issue_url":"https://api.github.com/repos/scylladb/scylla/issues/1295","id":222678492,"user":{"login":"nyh","id":584227,"avatar_url":"https://avatars.githubusercontent.com/u/584227?v=3","gravatar_id":"","url":"https://api.github.com/users/nyh","html_url":"https://github.com/nyh","followers_url":"https://api.github.com/users/nyh/followers","following_url":"https://api.github.com/users/nyh/following{/other_user}","gists_url":"https://api.github.com/users/nyh/gists{/gist_id}","starred_url":"https://api.github.com/users/nyh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nyh/subscriptions","organizations_url":"https://api.github.com/users/nyh/orgs","repos_url":"https://api.github.com/users/nyh/repos","events_url":"https://api.github.com/users/nyh/events{/privacy}","received_events_url":"https://api.github.com/users/nyh/received_events","type":"User","site_admin":false},"created_at":"2016-05-31T12:50:24Z","updated_at":"2016-05-31T12:50:24Z","body":"@avikivity regarding multi-tenancy, if we do not overcommit resources at all, and promise each tenant a specific slice of the CPU and memory, then you are right.\r\n\r\nBut if we do overcommit, and bill each tenant for the resources they got, then I think we'll get better efficiency by serializing the compaction - even if it means one tenant's compaction gets delayed while another tenant's compaction is being done. One way to think about it: Imagine that we can bill a client for the amount of memory he actually uses - e.g., 1 cent per gigabyte*hour (some providers call this unit \"GB Hour\"). Imagine that while a certain compaction is in progress, it uses 100 MB. If this tenant was allowed to run this compaction to completion, it would finish in (say) 30 seconds. But if it is done in parallel with 4 other tenants, it will complete in 4 times longer. So the tenant will need to pay to hold that 100 MB in memory for 4 times longer. He would probably not appreciate it, and prefer that compactions of different tenants are staggered, not done concurrently."}}
{"actor":{"display_login":"nyh"}}