{"action":"created","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/3292","repository_url":"https://api.github.com/repos/scylladb/scylla","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/3292/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/3292/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/3292/events","html_url":"https://github.com/scylladb/scylla/issues/3292","id":305385816,"number":3292,"title":"Expose a large partition table","user":{"login":"duarten","id":501746,"avatar_url":"https://avatars1.githubusercontent.com/u/501746?v=4","gravatar_id":"","url":"https://api.github.com/users/duarten","html_url":"https://github.com/duarten","followers_url":"https://api.github.com/users/duarten/followers","following_url":"https://api.github.com/users/duarten/following{/other_user}","gists_url":"https://api.github.com/users/duarten/gists{/gist_id}","starred_url":"https://api.github.com/users/duarten/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/duarten/subscriptions","organizations_url":"https://api.github.com/users/duarten/orgs","repos_url":"https://api.github.com/users/duarten/repos","events_url":"https://api.github.com/users/duarten/events{/privacy}","received_events_url":"https://api.github.com/users/duarten/received_events","type":"User","site_admin":false},"labels":[{"id":352004870,"url":"https://api.github.com/repos/scylladb/scylla/labels/UX","name":"UX","color":"006b75","default":false},{"id":160570958,"url":"https://api.github.com/repos/scylladb/scylla/labels/enhancement","name":"enhancement","color":"84b6eb","default":true},{"id":612786780,"url":"https://api.github.com/repos/scylladb/scylla/labels/n00b","name":"n00b","color":"d4c5f9","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/scylladb/scylla/milestones/18","html_url":"https://github.com/scylladb/scylla/milestone/18","labels_url":"https://api.github.com/repos/scylladb/scylla/milestones/18/labels","id":1974692,"number":18,"title":"2.x","description":"","creator":{"login":"tzach","id":170200,"avatar_url":"https://avatars1.githubusercontent.com/u/170200?v=4","gravatar_id":"","url":"https://api.github.com/users/tzach","html_url":"https://github.com/tzach","followers_url":"https://api.github.com/users/tzach/followers","following_url":"https://api.github.com/users/tzach/following{/other_user}","gists_url":"https://api.github.com/users/tzach/gists{/gist_id}","starred_url":"https://api.github.com/users/tzach/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tzach/subscriptions","organizations_url":"https://api.github.com/users/tzach/orgs","repos_url":"https://api.github.com/users/tzach/repos","events_url":"https://api.github.com/users/tzach/events{/privacy}","received_events_url":"https://api.github.com/users/tzach/received_events","type":"User","site_admin":false},"open_issues":227,"closed_issues":34,"state":"open","created_at":"2016-08-31T10:02:38Z","updated_at":"2018-03-18T12:16:52Z","due_on":null,"closed_at":null},"comments":2,"created_at":"2018-03-15T01:37:15Z","updated_at":"2018-03-19T13:59:26Z","closed_at":null,"author_association":"MEMBER","body":"*Installation details*\r\nScylla version (or git commit hash): master\r\n\r\nScylla provides a configuration option to warn when large partitions are written - `compaction_large_partition_warning_threshold_mb`. This provides little observability. What we wish to have instead is a set with the top offenders, which we hold in memory and expose via nodetool or through a virtual table (consumable with cqlsh). @avikivity [writes](https://groups.google.com/d/msg/scylladb-dev/q4GTXhhl1zs/mrNn_4_2BQAJ):\r\n\r\n> I thought about publishing a top-10 keys over REST, but the problem is we\r\ncan't age out the keys. Even if we see the same key again with a smaller\r\nsize, it doesn't mean the partition become smaller, it can be a different\r\ncompaction.\r\n\r\n> Perhaps we can maintain a table of { sstable, key, size } for all partitions\r\n> above a given size. Because the sstable is part of the key, we can now\r\n> remove it when it is compacted away (it's easy - just drop all entries for a\r\n> compacted sstable).\r\n> \r\n> We can then access the large partition table via nodetool. Or even cqlsh, if\r\n> the table is a virtual one.\r\n> \r\n> $ cqlsh\r\n> SELECT * FROM system.large_partitions WHERE table = 'ks1.standard1';\r\n> table   |  key        | size\r\n> \\-\\-\\-\\-\\-\\-\\---------------------\r\n> blah blah\r\n> \r\n> (the virtual table's schema would be\r\n> \r\n> CREATE TABLE system.large_partitions (\r\n>      table text,\r\n>      partition_size bigint,\r\n>      partition_key text,\r\n>      PRIMARY KEY (text, partition_size DESC)\r\n> )\r\n> \r\n> so you can easily select the top N) "},"comment":{"url":"https://api.github.com/repos/scylladb/scylla/issues/comments/374221036","html_url":"https://github.com/scylladb/scylla/issues/3292#issuecomment-374221036","issue_url":"https://api.github.com/repos/scylladb/scylla/issues/3292","id":374221036,"user":{"login":"glommer","id":331197,"avatar_url":"https://avatars3.githubusercontent.com/u/331197?v=4","gravatar_id":"","url":"https://api.github.com/users/glommer","html_url":"https://github.com/glommer","followers_url":"https://api.github.com/users/glommer/followers","following_url":"https://api.github.com/users/glommer/following{/other_user}","gists_url":"https://api.github.com/users/glommer/gists{/gist_id}","starred_url":"https://api.github.com/users/glommer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/glommer/subscriptions","organizations_url":"https://api.github.com/users/glommer/orgs","repos_url":"https://api.github.com/users/glommer/repos","events_url":"https://api.github.com/users/glommer/events{/privacy}","received_events_url":"https://api.github.com/users/glommer/received_events","type":"User","site_admin":false},"created_at":"2018-03-19T13:59:26Z","updated_at":"2018-03-19T13:59:26Z","author_association":"CONTRIBUTOR","body":"good point - it was already there. My bad"}}
{"actor":{"display_login":"glommer"}}