{"action":"created","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/1101","repository_url":"https://api.github.com/repos/scylladb/scylla","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/1101/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/1101/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/1101/events","html_url":"https://github.com/scylladb/scylla/issues/1101","id":143747540,"number":1101,"title":"Investigate little throughput drops caused by compaction/memtable flush","user":{"login":"dorlaor","id":1735237,"avatar_url":"https://avatars.githubusercontent.com/u/1735237?v=3","gravatar_id":"","url":"https://api.github.com/users/dorlaor","html_url":"https://github.com/dorlaor","followers_url":"https://api.github.com/users/dorlaor/followers","following_url":"https://api.github.com/users/dorlaor/following{/other_user}","gists_url":"https://api.github.com/users/dorlaor/gists{/gist_id}","starred_url":"https://api.github.com/users/dorlaor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dorlaor/subscriptions","organizations_url":"https://api.github.com/users/dorlaor/orgs","repos_url":"https://api.github.com/users/dorlaor/repos","events_url":"https://api.github.com/users/dorlaor/events{/privacy}","received_events_url":"https://api.github.com/users/dorlaor/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/scylladb/scylla/labels/enhancement","name":"enhancement","color":"84b6eb"},{"url":"https://api.github.com/repos/scylladb/scylla/labels/performance","name":"performance","color":"bfe5bf"}],"state":"open","locked":false,"assignee":{"login":"glommer","id":331197,"avatar_url":"https://avatars.githubusercontent.com/u/331197?v=3","gravatar_id":"","url":"https://api.github.com/users/glommer","html_url":"https://github.com/glommer","followers_url":"https://api.github.com/users/glommer/followers","following_url":"https://api.github.com/users/glommer/following{/other_user}","gists_url":"https://api.github.com/users/glommer/gists{/gist_id}","starred_url":"https://api.github.com/users/glommer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/glommer/subscriptions","organizations_url":"https://api.github.com/users/glommer/orgs","repos_url":"https://api.github.com/users/glommer/repos","events_url":"https://api.github.com/users/glommer/events{/privacy}","received_events_url":"https://api.github.com/users/glommer/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/scylladb/scylla/milestones/12","html_url":"https://github.com/scylladb/scylla/milestones/1.1","labels_url":"https://api.github.com/repos/scylladb/scylla/milestones/12/labels","id":1661891,"number":12,"title":"1.1","description":"","creator":{"login":"tzach","id":170200,"avatar_url":"https://avatars.githubusercontent.com/u/170200?v=3","gravatar_id":"","url":"https://api.github.com/users/tzach","html_url":"https://github.com/tzach","followers_url":"https://api.github.com/users/tzach/followers","following_url":"https://api.github.com/users/tzach/following{/other_user}","gists_url":"https://api.github.com/users/tzach/gists{/gist_id}","starred_url":"https://api.github.com/users/tzach/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tzach/subscriptions","organizations_url":"https://api.github.com/users/tzach/orgs","repos_url":"https://api.github.com/users/tzach/repos","events_url":"https://api.github.com/users/tzach/events{/privacy}","received_events_url":"https://api.github.com/users/tzach/received_events","type":"User","site_admin":false},"open_issues":70,"closed_issues":45,"state":"open","created_at":"2016-03-23T13:18:57Z","updated_at":"2016-04-20T18:27:05Z","due_on":null,"closed_at":null},"comments":4,"created_at":"2016-03-26T20:37:19Z","updated_at":"2016-04-21T04:24:22Z","closed_at":null,"body":"Installation details\r\nScylla version (or git commit hash): RC1\r\nCluster size: 3 nodes, RF=3, CL=Q\r\nOS (RHEL/CentOS/Ubuntu/AWS AMI): AWS, i8.2xl\r\n\r\nGlauber provided these graphs. One can easily identify a pattern of a big drop of ~100k ops\r\nevery couple of seconds with two intermediate\r\n![disk-usage](https://cloud.githubusercontent.com/assets/1735237/14062314/8c37340e-f3ab-11e5-8181-7d4d60eec9fe.png)\r\n![results](https://cloud.githubusercontent.com/assets/1735237/14062313/8c36642a-f3ab-11e5-89a0-c10b10b038e1.png)\r\n\r\n small drops.\r\n\r\nWe need to smooth them up."},"comment":{"url":"https://api.github.com/repos/scylladb/scylla/issues/comments/212734155","html_url":"https://github.com/scylladb/scylla/issues/1101#issuecomment-212734155","issue_url":"https://api.github.com/repos/scylladb/scylla/issues/1101","id":212734155,"user":{"login":"nyh","id":584227,"avatar_url":"https://avatars.githubusercontent.com/u/584227?v=3","gravatar_id":"","url":"https://api.github.com/users/nyh","html_url":"https://github.com/nyh","followers_url":"https://api.github.com/users/nyh/followers","following_url":"https://api.github.com/users/nyh/following{/other_user}","gists_url":"https://api.github.com/users/nyh/gists{/gist_id}","starred_url":"https://api.github.com/users/nyh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nyh/subscriptions","organizations_url":"https://api.github.com/users/nyh/orgs","repos_url":"https://api.github.com/users/nyh/repos","events_url":"https://api.github.com/users/nyh/events{/privacy}","received_events_url":"https://api.github.com/users/nyh/received_events","type":"User","site_admin":false},"created_at":"2016-04-21T04:24:22Z","updated_at":"2016-04-21T04:24:22Z","body":"On Thu, Apr 21, 2016 at 1:49 AM, Glauber Costa <notifications@github.com>\nwrote:\n\n> Just posted a patch that smoothes this out. It has a bit of seastar\n> infrastructure too.\n>\n> Effects on the load:\n>\n> [image: compactions-v2]\n> <https://cloud.githubusercontent.com/assets/331197/14692885/6e825688-0728-11e6-9f98-98a36f962fa1.png>\n>\n> Effects on the disk:\n>\n> [image: memtables-v2]\n> <https://cloud.githubusercontent.com/assets/331197/14692886/6e827672-0728-11e6-9a8c-f99a58a29368.png>\n>\n> We can see that compactions are not made slower, but are smoothed out over\n> time.\n>\n\nHow do we see in this graph that \"compactions are not made slower\"?\n\nWhen we really need a compaction to finish quickly (e.g., we have a lot of\nsstables and reads\nare becoming slower, or alternatively, the machine is almost idle),  can\nit?\n\nNadav.\n"}}