{"action":"created","issue":{"url":"https://api.github.com/repos/bos/criterion/issues/122","repository_url":"https://api.github.com/repos/bos/criterion","labels_url":"https://api.github.com/repos/bos/criterion/issues/122/labels{/name}","comments_url":"https://api.github.com/repos/bos/criterion/issues/122/comments","events_url":"https://api.github.com/repos/bos/criterion/issues/122/events","html_url":"https://github.com/bos/criterion/issues/122","id":172730772,"number":122,"title":"Feedback on `criterion` in literature","user":{"login":"Icelandjack","id":501899,"avatar_url":"https://avatars.githubusercontent.com/u/501899?v=3","gravatar_id":"","url":"https://api.github.com/users/Icelandjack","html_url":"https://github.com/Icelandjack","followers_url":"https://api.github.com/users/Icelandjack/followers","following_url":"https://api.github.com/users/Icelandjack/following{/other_user}","gists_url":"https://api.github.com/users/Icelandjack/gists{/gist_id}","starred_url":"https://api.github.com/users/Icelandjack/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Icelandjack/subscriptions","organizations_url":"https://api.github.com/users/Icelandjack/orgs","repos_url":"https://api.github.com/users/Icelandjack/repos","events_url":"https://api.github.com/users/Icelandjack/events{/privacy}","received_events_url":"https://api.github.com/users/Icelandjack/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-08-23T15:28:38Z","updated_at":"2016-08-23T16:09:59Z","closed_at":"2016-08-23T16:09:59Z","body":"A recent paper [Robust benchmarking in noisy environments](http://arxiv.org/pdf/1608.04295.pdf) has feedback on `criterion`\r\n\r\n> While  it  is  impossible  to  eliminate  performance  variation entirely, benchmarking methodologies that attempt to account for both measurement error and external sources of variation do exist. For example, the Haskell microbenchmarking  package `criterion` attempts  to  thwart  error  due  to timer inaccuracy by growing the number of benchmark executions  per  timing  measurement  as  more  timing  measurements are  obtained.  After  all  measurements  are  taken,  a  summary estimate of the benchmark run time is obtained by examining the  derivative  of  the  ordinary  least  squares  regression  line at  the  point  of  a  single  evaluation  point.  There  are  three disadvantages  to  this  approach.  First,  the  least  squares  fit  is\r\nsensitive to outliers (though `criterion` does warn the user if outliers are detected). Second, measurements made earlier in this experiment are highly vulnerable to timer error, since few benchmark repetitions are used. These early measurements can skew  the  regression,  and  hence  also  skew  the  final  run  time estimate.  Third,  measurements  made  later  in  the  experiment can  repeat  the  benchmark  more  times  than  are  necessary to  overcome  timer  error,  constituting  an  inefficient  use  of experiment time."},"comment":{"url":"https://api.github.com/repos/bos/criterion/issues/comments/241786417","html_url":"https://github.com/bos/criterion/issues/122#issuecomment-241786417","issue_url":"https://api.github.com/repos/bos/criterion/issues/122","id":241786417,"user":{"login":"bos","id":23374,"avatar_url":"https://avatars.githubusercontent.com/u/23374?v=3","gravatar_id":"","url":"https://api.github.com/users/bos","html_url":"https://github.com/bos","followers_url":"https://api.github.com/users/bos/followers","following_url":"https://api.github.com/users/bos/following{/other_user}","gists_url":"https://api.github.com/users/bos/gists{/gist_id}","starred_url":"https://api.github.com/users/bos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bos/subscriptions","organizations_url":"https://api.github.com/users/bos/orgs","repos_url":"https://api.github.com/users/bos/repos","events_url":"https://api.github.com/users/bos/events{/privacy}","received_events_url":"https://api.github.com/users/bos/received_events","type":"User","site_admin":false},"created_at":"2016-08-23T16:09:59Z","updated_at":"2016-08-23T16:09:59Z","body":"That's a cool paper. Thanks for the link!"}}
{"actor":{"display_login":"bos"}}