{"action":"created","issue":{"url":"https://api.github.com/repos/snabbco/snabb/issues/976","repository_url":"https://api.github.com/repos/snabbco/snabb","labels_url":"https://api.github.com/repos/snabbco/snabb/issues/976/labels{/name}","comments_url":"https://api.github.com/repos/snabbco/snabb/issues/976/comments","events_url":"https://api.github.com/repos/snabbco/snabb/issues/976/events","html_url":"https://github.com/snabbco/snabb/issues/976","id":168316777,"number":976,"title":"NFV test matrix preview","user":{"login":"lukego","id":13791,"avatar_url":"https://avatars.githubusercontent.com/u/13791?v=3","gravatar_id":"","url":"https://api.github.com/users/lukego","html_url":"https://github.com/lukego","followers_url":"https://api.github.com/users/lukego/followers","following_url":"https://api.github.com/users/lukego/following{/other_user}","gists_url":"https://api.github.com/users/lukego/gists{/gist_id}","starred_url":"https://api.github.com/users/lukego/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lukego/subscriptions","organizations_url":"https://api.github.com/users/lukego/orgs","repos_url":"https://api.github.com/users/lukego/repos","events_url":"https://api.github.com/users/lukego/events{/privacy}","received_events_url":"https://api.github.com/users/lukego/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/snabbco/snabb/labels/idea","name":"idea","color":"009800"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2016-07-29T12:44:43Z","updated_at":"2016-08-03T12:40:06Z","closed_at":null,"body":"Check out this new [NFV test matrix report](https://hydra.snabb.co/build/176830/download/2/report.html)!\r\n\r\nThis is a fully automated workflow where Hydra detects changes on branches under test, executes many end-to-end benchmarks (>10,000) with VMs in different setups, and produces a report that shows benchmark results both in broad overview and slice-and-diced by different factors. The tests run on a 10-machine cluster and take around one day to complete. *Awesome work, @domenkozar!!!*\r\n\r\nWe should be able to hook this up to upstream branches including `master` and `next` once we land #969 that allows Snabb NFV to run without a physical hardware NIC. Just for now it is connected to a couple of branches that do include this PR.\r\n\r\n### Observations\r\n\r\nThis is a preview in the sense that we have not been able to thoroughly sanity-check the results yet and some of them are different than we have seen in the past. So take all of this with a grain of salt.\r\n\r\nOn the **Overall** graph we can see that the two Snabb branches being tested, `matrix` and `matrix-next`, look practically identical.\r\n\r\nOn the **iperf** graph we can see results clustered around three places: 0 Gbps (failure), 6 Gbps, and 10-15 Gbps. On the **iperf configuration** graph we can see the explanation: the *filter* tests are failing (likely our packet filter is blocking iperf); the *ipsec* tests are delivering ~6 Gbps; and the baseline and L2TPv3 tests are delivering ~10-15 Gbps.\r\n\r\nOn the **l2fwd** graph we can see that most results are failing. Question is, why? I am not sure. The **l2fwd success and failure** stats may point to QEMU version and packet size being important factors? (These tests are much more reliable with SnabbBot so what is the difference here? Can be related to the guest kernel.)\r\n\r\n### Onward\r\n\r\nSo! Now we can automatically see whether changes are good for Snabb NFV. We want to merge changes that make the humps higher (more consistency) and that move the area under the curves to the right (faster average). Then we also want harsher test cases that reveal problems by spreading the curves out and moving them to the left :).\r\n\r\nIf this works well in practice then we could gradually add coverage for all the other Snabb applications too.\r\n\r\n### Feedback?\r\n\r\nQuestions, suggestions, comments?"},"comment":{"url":"https://api.github.com/repos/snabbco/snabb/issues/comments/237225794","html_url":"https://github.com/snabbco/snabb/issues/976#issuecomment-237225794","issue_url":"https://api.github.com/repos/snabbco/snabb/issues/976","id":237225794,"user":{"login":"lukego","id":13791,"avatar_url":"https://avatars.githubusercontent.com/u/13791?v=3","gravatar_id":"","url":"https://api.github.com/users/lukego","html_url":"https://github.com/lukego","followers_url":"https://api.github.com/users/lukego/followers","following_url":"https://api.github.com/users/lukego/following{/other_user}","gists_url":"https://api.github.com/users/lukego/gists{/gist_id}","starred_url":"https://api.github.com/users/lukego/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lukego/subscriptions","organizations_url":"https://api.github.com/users/lukego/orgs","repos_url":"https://api.github.com/users/lukego/repos","events_url":"https://api.github.com/users/lukego/events{/privacy}","received_events_url":"https://api.github.com/users/lukego/received_events","type":"User","site_admin":false},"created_at":"2016-08-03T12:40:06Z","updated_at":"2016-08-03T12:40:06Z","body":"@kbara I pushed a new version of the report that includes overlay histograms now. The Y-axis also now gives the exact number of results per bin. Better? https://hydra.snabb.co/build/203152/download/2/report.html\r\n\r\n(I also added the red-and-white \"success rate\" tables. I have a fix in the pipeline that will make the l2fwd picture look much better soon.)"}}
{"actor":{"display_login":"lukego"}}