{"action":"created","issue":{"url":"https://api.github.com/repos/dotnet/performance/issues/165","repository_url":"https://api.github.com/repos/dotnet/performance","labels_url":"https://api.github.com/repos/dotnet/performance/issues/165/labels{/name}","comments_url":"https://api.github.com/repos/dotnet/performance/issues/165/comments","events_url":"https://api.github.com/repos/dotnet/performance/issues/165/events","html_url":"https://github.com/dotnet/performance/pull/165","id":387467291,"node_id":"MDExOlB1bGxSZXF1ZXN0MjM1OTQ5NDU3","number":165,"title":"Results Comparer","user":{"login":"adamsitnik","id":6011991,"node_id":"MDQ6VXNlcjYwMTE5OTE=","avatar_url":"https://avatars2.githubusercontent.com/u/6011991?v=4","gravatar_id":"","url":"https://api.github.com/users/adamsitnik","html_url":"https://github.com/adamsitnik","followers_url":"https://api.github.com/users/adamsitnik/followers","following_url":"https://api.github.com/users/adamsitnik/following{/other_user}","gists_url":"https://api.github.com/users/adamsitnik/gists{/gist_id}","starred_url":"https://api.github.com/users/adamsitnik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adamsitnik/subscriptions","organizations_url":"https://api.github.com/users/adamsitnik/orgs","repos_url":"https://api.github.com/users/adamsitnik/repos","events_url":"https://api.github.com/users/adamsitnik/events{/privacy}","received_events_url":"https://api.github.com/users/adamsitnik/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":11,"created_at":"2018-12-04T20:31:22Z","updated_at":"2018-12-05T19:44:06Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/dotnet/performance/pulls/165","html_url":"https://github.com/dotnet/performance/pull/165","diff_url":"https://github.com/dotnet/performance/pull/165.diff","patch_url":"https://github.com/dotnet/performance/pull/165.patch"},"body":"# Results Comparer\r\n\r\nThis simple tool allows for easy comparison of provided benchmark results.\r\n\r\nIt can be used to compare:\r\n* historical results (eg. before and after my changes)\r\n* results for different OSes (eg. Windows vs Ubuntu)\r\n* results for different CPU architectures (eg. x64 vs ARM64)\r\n* results for different target frameworks (eg. .NET Core 2.1 vs 2.2)\r\n\r\nAll you need to provide is:\r\n* `--base` - path to folder/file with baseline results\r\n* `--diff` - path to folder/file with diff results\r\n* `--treshold`  - threshold for Statistical Test. Examples: 5%, 10ms, 100ns, 1s\r\n\r\nOptional arguments:\r\n* `--top` - filter the diff to top/bottom `N` results\r\n\r\nSample: compare the results stored in `C:\\results\\windows` vs `C:\\results\\ubuntu` using `1%` threshold and print only TOP 10.\r\n\r\n```cmd\r\ndotnet run --base \"C:\\results\\windows\" --diff \"C:\\results\\ubuntu\" --treshold 1% --top 10\r\n```\r\n\r\n**Note**: the tool supports only `*full.json` results exported by BenchmarkDotNet. This exporter is enabled by default in this repository.\r\n\r\n**Note**: if you have run your benchmarks for multiple jobs (eg. `-r netcoreapp2.1 netcoreapp2.2`) and you want to compare these historical results you can use `--merge` to point to the folder/file with such results.\r\n\r\n## Reading the results\r\n\r\nSample results:\r\n\r\n```log\r\nSLOWER:\r\n-30,35% BenchmarksGame.SpectralNorm_3.RunBench [can have several modes]\r\n-5,45% BenchmarksGame.KNucleotide_9.RunBench [bimodal]\r\n-4,27% BenchmarksGame.BinaryTrees_2.RunBench\r\n-2,42% BenchmarksGame.MandelBrot_7.Bench(size: 4000, lineLength: 500, checksum: \"C7-E6-66-43-66-73-F8-A8-D3-B4-D7-97-2F-FC-A1-D3\")\r\n\r\nFASTER:\r\n3,59% BenchmarksGame.FannkuchRedux_5.RunBench(n: 10, expectedSum: 38)\r\n```\r\n\r\nExplanation:\r\n* if there is no difference, the results are omitted\r\n* if there is no match (we use full benchmark names to match the benchmarks), the results are omitted\r\n* every line contains:\r\n  * ratio (`1.0 - diffMedian / baseMedian`)\r\n  * id (full benchmark name, the same we use in BenchView)\r\n  * optional information about the modality of the benchmark"},"comment":{"url":"https://api.github.com/repos/dotnet/performance/issues/comments/444618856","html_url":"https://github.com/dotnet/performance/pull/165#issuecomment-444618856","issue_url":"https://api.github.com/repos/dotnet/performance/issues/165","id":444618856,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NDYxODg1Ng==","user":{"login":"billwert","id":4523120,"node_id":"MDQ6VXNlcjQ1MjMxMjA=","avatar_url":"https://avatars3.githubusercontent.com/u/4523120?v=4","gravatar_id":"","url":"https://api.github.com/users/billwert","html_url":"https://github.com/billwert","followers_url":"https://api.github.com/users/billwert/followers","following_url":"https://api.github.com/users/billwert/following{/other_user}","gists_url":"https://api.github.com/users/billwert/gists{/gist_id}","starred_url":"https://api.github.com/users/billwert/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/billwert/subscriptions","organizations_url":"https://api.github.com/users/billwert/orgs","repos_url":"https://api.github.com/users/billwert/repos","events_url":"https://api.github.com/users/billwert/events{/privacy}","received_events_url":"https://api.github.com/users/billwert/received_events","type":"User","site_admin":false},"created_at":"2018-12-05T19:44:06Z","updated_at":"2018-12-05T19:44:06Z","author_association":"NONE","body":"> one thing is Windows vs Ubuntu or x64 vs ARM64\r\n> the other is: I run the benchmarks before applying any changes and save the results somewhere, then I apply the changes and save them to other location and compare the perf.\r\n\r\n@adamsitnik sure, makes sense. I'm more curious about whether or not this is something we'd want to incorporate into our automation strategy in general (think reporting), because modularizing it would be nice in that case. We can cross this bridge later however.\r\n\r\nOne other question: Why is this stand alone and not a mode in Benchmark.NET?"}}
{"actor":{"display_login":"billwert"}}