{"action":"created","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/3628","repository_url":"https://api.github.com/repos/scylladb/scylla","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/3628/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/3628/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/3628/events","html_url":"https://github.com/scylladb/scylla/issues/3628","id":344227306,"node_id":"MDU6SXNzdWUzNDQyMjczMDY=","number":3628,"title":"Performance regression of 780% in p99th latency compared to 2.2.0 for 100% read test","user":{"login":"roydahan","id":20959584,"node_id":"MDQ6VXNlcjIwOTU5NTg0","avatar_url":"https://avatars3.githubusercontent.com/u/20959584?v=4","gravatar_id":"","url":"https://api.github.com/users/roydahan","html_url":"https://github.com/roydahan","followers_url":"https://api.github.com/users/roydahan/followers","following_url":"https://api.github.com/users/roydahan/following{/other_user}","gists_url":"https://api.github.com/users/roydahan/gists{/gist_id}","starred_url":"https://api.github.com/users/roydahan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/roydahan/subscriptions","organizations_url":"https://api.github.com/users/roydahan/orgs","repos_url":"https://api.github.com/users/roydahan/repos","events_url":"https://api.github.com/users/roydahan/events{/privacy}","received_events_url":"https://api.github.com/users/roydahan/received_events","type":"User","site_admin":false},"labels":[{"id":470674858,"node_id":"MDU6TGFiZWw0NzA2NzQ4NTg=","url":"https://api.github.com/repos/scylladb/scylla/labels/Regression","name":"Regression","color":"e99695","default":false},{"id":160570956,"node_id":"MDU6TGFiZWwxNjA1NzA5NTY=","url":"https://api.github.com/repos/scylladb/scylla/labels/bug","name":"bug","color":"eb6420","default":true},{"id":302689228,"node_id":"MDU6TGFiZWwzMDI2ODkyMjg=","url":"https://api.github.com/repos/scylladb/scylla/labels/high","name":"high","color":"d4c5f9","default":false},{"id":246224279,"node_id":"MDU6TGFiZWwyNDYyMjQyNzk=","url":"https://api.github.com/repos/scylladb/scylla/labels/performance","name":"performance","color":"bfe5bf","default":false}],"state":"open","locked":false,"assignee":{"login":"tgrabiec","id":283695,"node_id":"MDQ6VXNlcjI4MzY5NQ==","avatar_url":"https://avatars3.githubusercontent.com/u/283695?v=4","gravatar_id":"","url":"https://api.github.com/users/tgrabiec","html_url":"https://github.com/tgrabiec","followers_url":"https://api.github.com/users/tgrabiec/followers","following_url":"https://api.github.com/users/tgrabiec/following{/other_user}","gists_url":"https://api.github.com/users/tgrabiec/gists{/gist_id}","starred_url":"https://api.github.com/users/tgrabiec/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tgrabiec/subscriptions","organizations_url":"https://api.github.com/users/tgrabiec/orgs","repos_url":"https://api.github.com/users/tgrabiec/repos","events_url":"https://api.github.com/users/tgrabiec/events{/privacy}","received_events_url":"https://api.github.com/users/tgrabiec/received_events","type":"User","site_admin":false},"assignees":[{"login":"tgrabiec","id":283695,"node_id":"MDQ6VXNlcjI4MzY5NQ==","avatar_url":"https://avatars3.githubusercontent.com/u/283695?v=4","gravatar_id":"","url":"https://api.github.com/users/tgrabiec","html_url":"https://github.com/tgrabiec","followers_url":"https://api.github.com/users/tgrabiec/followers","following_url":"https://api.github.com/users/tgrabiec/following{/other_user}","gists_url":"https://api.github.com/users/tgrabiec/gists{/gist_id}","starred_url":"https://api.github.com/users/tgrabiec/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tgrabiec/subscriptions","organizations_url":"https://api.github.com/users/tgrabiec/orgs","repos_url":"https://api.github.com/users/tgrabiec/repos","events_url":"https://api.github.com/users/tgrabiec/events{/privacy}","received_events_url":"https://api.github.com/users/tgrabiec/received_events","type":"User","site_admin":false},{"login":"glommer","id":331197,"node_id":"MDQ6VXNlcjMzMTE5Nw==","avatar_url":"https://avatars3.githubusercontent.com/u/331197?v=4","gravatar_id":"","url":"https://api.github.com/users/glommer","html_url":"https://github.com/glommer","followers_url":"https://api.github.com/users/glommer/followers","following_url":"https://api.github.com/users/glommer/following{/other_user}","gists_url":"https://api.github.com/users/glommer/gists{/gist_id}","starred_url":"https://api.github.com/users/glommer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/glommer/subscriptions","organizations_url":"https://api.github.com/users/glommer/orgs","repos_url":"https://api.github.com/users/glommer/repos","events_url":"https://api.github.com/users/glommer/events{/privacy}","received_events_url":"https://api.github.com/users/glommer/received_events","type":"User","site_admin":false},{"login":"avikivity","id":1017210,"node_id":"MDQ6VXNlcjEwMTcyMTA=","avatar_url":"https://avatars1.githubusercontent.com/u/1017210?v=4","gravatar_id":"","url":"https://api.github.com/users/avikivity","html_url":"https://github.com/avikivity","followers_url":"https://api.github.com/users/avikivity/followers","following_url":"https://api.github.com/users/avikivity/following{/other_user}","gists_url":"https://api.github.com/users/avikivity/gists{/gist_id}","starred_url":"https://api.github.com/users/avikivity/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/avikivity/subscriptions","organizations_url":"https://api.github.com/users/avikivity/orgs","repos_url":"https://api.github.com/users/avikivity/repos","events_url":"https://api.github.com/users/avikivity/events{/privacy}","received_events_url":"https://api.github.com/users/avikivity/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/scylladb/scylla/milestones/26","html_url":"https://github.com/scylladb/scylla/milestone/26","labels_url":"https://api.github.com/repos/scylladb/scylla/milestones/26/labels","id":2916755,"node_id":"MDk6TWlsZXN0b25lMjkxNjc1NQ==","number":26,"title":"2.3","description":null,"creator":{"login":"slivne","id":3465480,"node_id":"MDQ6VXNlcjM0NjU0ODA=","avatar_url":"https://avatars0.githubusercontent.com/u/3465480?v=4","gravatar_id":"","url":"https://api.github.com/users/slivne","html_url":"https://github.com/slivne","followers_url":"https://api.github.com/users/slivne/followers","following_url":"https://api.github.com/users/slivne/following{/other_user}","gists_url":"https://api.github.com/users/slivne/gists{/gist_id}","starred_url":"https://api.github.com/users/slivne/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slivne/subscriptions","organizations_url":"https://api.github.com/users/slivne/orgs","repos_url":"https://api.github.com/users/slivne/repos","events_url":"https://api.github.com/users/slivne/events{/privacy}","received_events_url":"https://api.github.com/users/slivne/received_events","type":"User","site_admin":false},"open_issues":23,"closed_issues":80,"state":"open","created_at":"2017-11-16T10:37:14Z","updated_at":"2018-08-15T11:18:52Z","due_on":null,"closed_at":null},"comments":68,"created_at":"2018-07-24T22:32:01Z","updated_at":"2018-08-17T19:28:14Z","closed_at":null,"author_association":"NONE","body":"*Installation details*\r\nScylla version (or git commit hash): 2.3.rc0-0.20180722.a77bb1fe3\r\nCluster size: 3\r\nOS (RHEL/CentOS/Ubuntu/AWS AMI): AWS AMI (ami-905252ef)\r\n\r\ntest_latency_read results showing 780% regression in p99th latency compared to 2.2.0:\r\n\r\nVersion  | Op rate total | Latency mean | Latency 99th percentile\r\n-- | -- | -- | --\r\n2.2.0 |  39997.0 [2018-07-19 10:26:37] | 1.4 [2018-07-19 10:26:37] |  3.1 [2018-07-19 10:26:37]\r\n2.3.0 | 37200.0 (6% Regression) | 8.2 (485% Regression) | 27.3 (780% Regression)\r\n\r\n2.3.0 p99th latency looks abnormal and reaches peaks of ~400ms:\r\n![screen shot 2018-07-25 at 1 26 42](https://user-images.githubusercontent.com/20959584/43169475-e0d115ce-8fa9-11e8-8254-4fc6e9a5a862.png)\r\n\r\nTest is populating 1TB of data and then start a c-s read command:\r\n`cassandra-stress read no-warmup  cl=QUORUM duration=50m -schema 'replication(factor=3)' -port jmx=6868 -mode cql3 native -rate 'threads=100 limit=10000/s' -errors ignore -col 'size=FIXED(1024) n=FIXED(1)' -pop 'dist=gauss(1..1000000000,500000000,50000000)'\r\n`\r\n(During the first part of the test we can still see compactions that are leftovers of the write population)\r\n\r\nFull screenshot:\r\n![screencapture-34-230-6-17-3000-dashboard-db-scylla-per-server-metrics-nemesis-master-test-latency-2-3-2018-07-25-01_31_03](https://user-images.githubusercontent.com/20959584/43169633-7f214686-8faa-11e8-8f0a-bbf02a692918.png)\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/scylladb/scylla/issues/comments/413966287","html_url":"https://github.com/scylladb/scylla/issues/3628#issuecomment-413966287","issue_url":"https://api.github.com/repos/scylladb/scylla/issues/3628","id":413966287,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMzk2NjI4Nw==","user":{"login":"glommer","id":331197,"node_id":"MDQ6VXNlcjMzMTE5Nw==","avatar_url":"https://avatars3.githubusercontent.com/u/331197?v=4","gravatar_id":"","url":"https://api.github.com/users/glommer","html_url":"https://github.com/glommer","followers_url":"https://api.github.com/users/glommer/followers","following_url":"https://api.github.com/users/glommer/following{/other_user}","gists_url":"https://api.github.com/users/glommer/gists{/gist_id}","starred_url":"https://api.github.com/users/glommer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/glommer/subscriptions","organizations_url":"https://api.github.com/users/glommer/orgs","repos_url":"https://api.github.com/users/glommer/repos","events_url":"https://api.github.com/users/glommer/events{/privacy}","received_events_url":"https://api.github.com/users/glommer/received_events","type":"User","site_admin":false},"created_at":"2018-08-17T19:28:14Z","updated_at":"2018-08-17T19:28:14Z","author_association":"CONTRIBUTOR","body":"We are doing around exactly 1000 reads/sec total.\r\n\r\nThe graph above is for a single shard."}}
{"actor":{"display_login":"glommer"}}