{"action":"opened","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/944","repository_url":"https://api.github.com/repos/scylladb/scylla","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/944/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/944/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/944/events","html_url":"https://github.com/scylladb/scylla/issues/944","id":136115216,"number":944,"title":"Repair is chocking the disk","user":{"login":"noamha","id":2642373,"avatar_url":"https://avatars.githubusercontent.com/u/2642373?v=3","gravatar_id":"","url":"https://api.github.com/users/noamha","html_url":"https://github.com/noamha","followers_url":"https://api.github.com/users/noamha/followers","following_url":"https://api.github.com/users/noamha/following{/other_user}","gists_url":"https://api.github.com/users/noamha/gists{/gist_id}","starred_url":"https://api.github.com/users/noamha/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/noamha/subscriptions","organizations_url":"https://api.github.com/users/noamha/orgs","repos_url":"https://api.github.com/users/noamha/repos","events_url":"https://api.github.com/users/noamha/events{/privacy}","received_events_url":"https://api.github.com/users/noamha/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-02-24T16:38:16Z","updated_at":"2016-02-24T16:38:16Z","closed_at":null,"body":"Configuration:\r\n- 4 nodes x Rackspace OnMetal IO v1 CentOS 7.0\r\n- Scylla 0.18 Head\r\n\r\nSteps:\r\n- Started a 3 nodes cluster\r\n- Init Keyspace on all 3 servers\r\n```\r\ncassandra-stress write n=1 cl=ALL -schema \"replication(factor=3)\" -mode native cql3 -node $SERVERS\r\n```\r\n- Populate with 5M keys\r\ntime cassandra-stress write cl=TWO n=5000000 -mode native cql3 -rate threads=700 -node $SERVERS -pop seq=1..5000000 -col 'size=FIXED(2048) n=FIXED(5)'\r\n\r\n- Write a constant very low stress\r\ntime cassandra-stress write cl=TWO n=50000000 -mode native cql3 -rate threads=10 -node $SERVERS -pop seq=5000000..1000000000 -col 'size=FIXED(2048) n=FIXED(5)'\r\n\r\n- Stop node3\r\n- Delete sstables from node3\r\n- Start node3\r\n- Run nodetool repair\r\n\r\nWhen load running without streaming:\r\n```\r\nDevice:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util\r\nsda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00\r\nsdb               0.00     0.00  151.50  794.00 18958.00 78210.00   205.54     0.24    0.26    0.47    0.21   0.21  20.20\r\nsdc               0.00     0.00  148.00  845.50 18820.00 78600.00   196.11     0.25    0.26    0.50    0.21   0.20  19.95\r\nmd0               0.00     0.00  299.50 1640.00 37778.00 156810.00   200.66     0.00    0.00    0.00    0.00   0.00   0.00\r\n\r\navg-cpu:  %user   %nice %system %iowait  %steal   %idle\r\n          17.86    0.00    5.73    0.00    0.00   76.42\r\n\r\nDevice:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util\r\nsda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00\r\nsdb               0.00     0.00  344.50 1125.50 43302.00 120498.00   222.86     0.39    0.26    0.43    0.22   0.21  31.35\r\nsdc               0.00     0.00  345.00 1178.50 43272.00 119812.00   214.09     0.42    0.28    0.40    0.24   0.22  33.45\r\nmd0               0.00     0.00  689.50 2304.50 86574.00 240310.00   218.36     0.00    0.00    0.00    0.00   0.00   0.00\r\n\r\navg-cpu:  %user   %nice %system %iowait  %steal   %idle\r\n          15.83    0.00    5.34    0.00    0.00   78.83\r\n\r\nDevice:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util\r\nsda               0.00     0.50    0.00    1.00     0.00     6.00    12.00     0.00    0.50    0.00    0.50   0.50   0.05\r\nsdb               0.00    52.00    0.00  573.00     0.00 42946.00   149.90     0.12    0.22    0.00    0.22   0.17   9.75\r\nsdc               0.00    45.00    0.00  601.50     0.00 43373.75   144.22     0.14    0.24    0.00    0.24   0.17  10.10\r\nmd0               0.00     0.00    0.00 1271.50     0.00 86319.75   135.78     0.00    0.00    0.00    0.00   0.00   0.00\r\n\r\navg-cpu:  %user   %nice %system %iowait  %steal   %idle\r\n          17.53    0.00    5.81    0.00    0.00   76.66\r\n\r\nDevice:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util\r\nsda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00\r\nsdb               0.00     0.00  426.00 1321.00 53722.00 145974.00   228.62     0.52    0.30    0.50    0.23   0.22  38.85\r\nsdc               0.00     0.00  426.50 1383.00 53848.00 145490.00   220.32     0.55    0.30    0.54    0.23   0.22  40.00\r\nmd0               0.00     0.00  853.00 2706.00 107634.00 291530.00   224.31     0.00    0.00    0.00    0.00   0.00   0.00\r\n\r\navg-cpu:  %user   %nice %system %iowait  %steal   %idle\r\n          16.96    0.00    5.45    0.00    0.00   77.59\r\n\r\nDevice:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util\r\nsda               0.00     0.00    0.00    1.00     0.00     8.00    16.00     0.00    0.00    0.00    0.00   0.00   0.00\r\nsdb               0.00     0.00  108.00  978.50 12834.00 101240.00   209.98     0.27    0.25    0.51    0.22   0.20  21.50\r\nsdc               0.00     0.00  106.50 1026.00 12762.00 101062.00   201.01     0.27    0.24    0.53    0.21   0.19  21.15\r\nmd0               0.00     0.00  214.00 2003.00 25532.00 202236.00   205.47     0.00    0.00    0.00    0.00   0.00   0.00\r\n\r\navg-cpu:  %user   %nice %system %iowait  %steal   %idle\r\n          17.00    0.00    5.58    0.01    0.00   77.41\r\n\r\nDevice:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util\r\nsda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00\r\nsdb               0.00     0.00    6.50  818.00    26.00 80112.00   194.39     0.18    0.22    0.00    0.22   0.20  16.45\r\nsdc               0.00     0.00    5.00  862.00    20.00 80382.00   185.47     0.20    0.23    0.00    0.23   0.19  16.90\r\nmd0               0.00     0.00   11.50 1680.00    46.00 160494.00   189.82     0.00    0.00    0.00    0.00   0.00   0.00\r\n```\r\n\r\nAnd then with streaming:\r\n```\r\navg-cpu:  %user   %nice %system %iowait  %steal   %idle\r\n          91.08    0.00    8.01    0.92    0.00    0.00\r\n\r\nDevice:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util\r\nsda               0.00     2.40    7.40    7.80   947.20  2763.20   488.21     0.56   36.95    2.08   70.03   4.12   6.26\r\nsdb            6828.40     4.40 16285.00  541.40 1391047.10 64697.20   173.03    83.40    4.95    5.03    2.77   0.06  99.74\r\nsdc            6835.80     5.80 16239.00  543.00 1389381.80 64641.00   173.28    89.07    5.30    5.38    2.91   0.06 100.00\r\nmd0               0.00     0.00 39107.40 1095.60 2780217.20 129389.40   144.75     0.00    0.00    0.00    0.00   0.00   0.00\r\n\r\navg-cpu:  %user   %nice %system %iowait  %steal   %idle\r\n          90.33    0.00    8.25    1.42    0.00    0.00\r\n\r\nDevice:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util\r\nsda               0.00     3.60    7.20    1.20   921.60    20.80   224.38     0.01    0.62    0.58    0.83   0.62   0.52\r\nsdb            6721.80     0.40 16320.20  566.20 1367960.50 68045.60   170.08    76.34    4.52    4.59    2.47   0.06  99.92\r\nsdc            6739.80     0.20 16242.60  561.80 1366457.60 67780.00   170.70    91.82    5.46    5.55    2.73   0.06 100.00\r\nmd0               0.00     0.00 39057.80 1128.80 2734830.30 135800.00   142.87     0.00    0.00    0.00    0.00   0.00   0.00\r\n\r\navg-cpu:  %user   %nice %system %iowait  %steal   %idle\r\n          90.66    0.00    8.04    1.31    0.00    0.00\r\n\r\nDevice:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util\r\nsda               0.00     1.20    7.00    0.60   896.00     8.00   237.89     0.00    0.50    0.49    0.67   0.50   0.38\r\nsdb            6954.40     0.00 16688.60  474.40 1407675.60 55935.20   170.55    74.86    4.36    4.41    2.51   0.06  99.92\r\nsdc            6976.80     0.20 16660.20  470.40 1409578.40 55828.80   171.09    95.25    5.56    5.63    2.95   0.06  99.90\r\nmd0               0.00     0.00 40068.00  945.80 2817365.30 111840.00   142.84     0.00    0.00    0.00    0.00   0.00   0.00\r\n\r\navg-cpu:  %user   %nice %system %iowait  %steal   %idle\r\n          90.32    0.00    8.36    1.33    0.00    0.00\r\n\r\nDevice:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util\r\nsda               0.00     1.20    7.40    7.80   947.20  2734.40   484.42     0.80   52.84    4.76   98.46   4.17   6.34\r\nsdb            6794.60     0.00 16445.20  540.20 1379517.40 63631.20   169.93    67.19    3.96    4.01    2.34   0.06  99.92\r\nsdc            6816.80     0.80 16379.00  532.60 1377293.80 63286.40   170.37   101.63    6.00    6.10    2.96   0.06  99.82\r\nmd0               0.00     0.00 39390.40 1074.00 2756860.60 126918.40   142.53     0.00    0.00    0.00    0.00   0.00   0.00\r\n\r\navg-cpu:  %user   %nice %system %iowait  %steal   %idle\r\n          88.75    0.00    8.65    2.60    0.00    0.00\r\n\r\nDevice:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util\r\nsda               0.00     3.80    7.80    1.20   998.40    21.60   226.67     0.00    0.51    0.44    1.00   0.51   0.46\r\nsdb            6128.40     0.40 15120.80  762.60 1263690.00 92049.60   170.71    65.10    4.09    4.19    2.28   0.06  99.64\r\nsdc            6140.60     0.80 15070.40  759.00 1263606.10 92077.60   171.29   102.57    6.47    6.65    3.04   0.06  99.74\r\nmd0               0.00     0.00 36108.00 1523.20 2527288.30 184072.00   144.10     0.00    0.00    0.00    0.00   0.00   0.00\r\n\r\navg-cpu:  %user   %nice %system %iowait  %steal   %idle\r\n          89.70    0.00    8.38    1.92    0.00    0.01\r\n\r\nDevice:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util\r\nsda               0.00     1.20    8.00    0.60  1024.00     8.00   240.00     0.01    0.63    0.60    1.00   0.60   0.52\r\nsdb            6631.60     0.20 16053.00  560.20 1361785.10 67208.00   172.03    71.24    4.28    4.35    2.39   0.06  99.82\r\nsdc            6648.80     0.00 16021.00  561.80 1364759.00 67152.00   172.70    96.96    5.85    5.95    2.89   0.06  99.70\r\nmd0               0.00     0.00 38477.80 1122.00 2726136.10 134338.40   144.47     0.00    0.00    0.00    0.00   0.00   0.00\r\n```\r\n- As you can see read is at ~1300MB / 16000 which is around 80kb\r\n- Read request is at 16,000 while write request is at 500\r\n- Bandwidth for read is 1.3GB while write is 60MB\r\n- Load stress was operating at 10,000 op/s and with repair is around 2,000 op/s\r\n\r\n\r\n\r\n![with-streaming](https://cloud.githubusercontent.com/assets/2642373/13292652/440697cc-db25-11e5-871c-2825b6486afa.png)\r\n\r\n\r\n![without-streaming](https://cloud.githubusercontent.com/assets/2642373/13292511/b570a5e8-db24-11e5-82cd-3e590343db95.png)\r\n\r\n"}}