{"action":"created","issue":{"url":"https://api.github.com/repos/evanphx/benchmark-ips/issues/68","repository_url":"https://api.github.com/repos/evanphx/benchmark-ips","labels_url":"https://api.github.com/repos/evanphx/benchmark-ips/issues/68/labels{/name}","comments_url":"https://api.github.com/repos/evanphx/benchmark-ips/issues/68/comments","events_url":"https://api.github.com/repos/evanphx/benchmark-ips/issues/68/events","html_url":"https://github.com/evanphx/benchmark-ips/pull/68","id":153147176,"number":68,"title":"Configurable sampling frequency/duration","user":{"login":"thedarkone","id":15688,"avatar_url":"https://avatars.githubusercontent.com/u/15688?v=3","gravatar_id":"","url":"https://api.github.com/users/thedarkone","html_url":"https://github.com/thedarkone","followers_url":"https://api.github.com/users/thedarkone/followers","following_url":"https://api.github.com/users/thedarkone/following{/other_user}","gists_url":"https://api.github.com/users/thedarkone/gists{/gist_id}","starred_url":"https://api.github.com/users/thedarkone/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thedarkone/subscriptions","organizations_url":"https://api.github.com/users/thedarkone/orgs","repos_url":"https://api.github.com/users/thedarkone/repos","events_url":"https://api.github.com/users/thedarkone/events{/privacy}","received_events_url":"https://api.github.com/users/thedarkone/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-05-05T00:45:33Z","updated_at":"2016-05-09T21:37:51Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/evanphx/benchmark-ips/pulls/68","html_url":"https://github.com/evanphx/benchmark-ips/pull/68","diff_url":"https://github.com/evanphx/benchmark-ips/pull/68.diff","patch_url":"https://github.com/evanphx/benchmark-ips/pull/68.patch"},"body":"Grrrr, alright, I've had enough :cry: :rage: :neckbeard:.\r\n\r\nDon't know if I'm going to be able to get this merged, but lets at least try (I'm open to all kinds of suggestions and will amend the PR as necessary).\r\n\r\nHere's the deal: `benchmark-ips` has been immensely beneficial to the state of accurate Ruby benchmarking, however I feel that in some cases it does lead its users astray. Namely because it measures and reports SD and newly (#60) rightfully refuses to decide benchmarks on flimsy statistical evidence, this leads some developers to inaccurately conclude that 2 versions of the code being benchmarked are equal in performance, when in fact this is demonstrably false. It is not that the benchmark is undecidable, it is just their configuration of `benchmark-ips` that prevents them from arriving to the correct result.\r\n\r\nIn my experience (subjective anecdotal evidence for the win :innocent:), in the absence of hard time limits on benchmark duration, it is absolutely the case that noisy benchmarks **can be** tamed and decided. Ignoring variance while benchmarking is of course a fools errand, but sometimes some noise is acceptable and we want to know which version of the code is faster (it is also usually the case that both versions have comparable SD, so it is not about plunging for the average while ignoring the latency/SD).\r\n\r\nIn this PR I propose to make sample duration/interval (or in other words sampling frequency) configurable, additionally in case of statistically ambiguous results, `benchmark-ips` would now propose to the user to tweak the configuration and attempt to re-run the experiment with a longer duration and decreased sampling frequency.\r\n\r\nThe PR is a spiritual successor to @chrisseaton's PR #60, let me know what you guys think.\r\n\r\n------\r\n\r\nTested with MRI:\r\n\r\n```ruby\r\n# bench.rb\r\nrequire 'benchmark/ips'\r\n\r\n# I like to make my VMs work instead of sleeping ;)\r\ndef work(num)\r\n  nnum = num * 300 # magic const to make the example work\r\n  i = 0\r\n  while i < nnum\r\n    i+=1\r\n  end\r\n  i\r\nend\r\n\r\nBenchmark.ips do |x|\r\n  x.report(\"work(1000 + r(500))\") do\r\n    work(1000 + rand(500))\r\n  end\r\n\r\n  x.report(\"work(1000 + r(400))\") do\r\n    work(1000 + rand(400))\r\n  end\r\n\r\n  x.compare!\r\nend\r\n```\r\n\r\n```\r\n> ruby -Ilib bench.rb ⏎\r\nWarming up --------------------------------------\r\n work(1000 + r(500))    15.000  i/100ms\r\n work(1000 + r(400))    15.000  i/100ms\r\nCalculating -------------------------------------\r\n work(1000 + r(500))    155.226  (± 2.6%) i/s -    780.000  in   5.029015s\r\n work(1000 + r(400))    162.102  (± 3.1%) i/s -    810.000  in   5.000809s\r\n\r\nComparison:\r\n work(1000 + r(400)):      162.1 i/s\r\n work(1000 + r(500)):      155.2 i/s - same-ish: difference falls within error\r\n\r\n-------------------------------------------------------------------------\r\n\r\nSome reports were within standard deviation of each other. Because of that\r\nbenchmark-ips was unable to decide which of them is faster. It is quite\r\npossible that with a slightly tweaked configuration benchmark-ips will\r\nbe able to provide more accurate results.\r\n\r\nPlease try re-running your benchmark with the following additional\r\nconfiguration:\r\n\r\n  Benchmark.ips do |x|\r\n    x.sample_duration = 0.4 # <--- new config\r\n    x.time = 10 # <--- new config\r\n    # ...\r\n\r\nPlease note that running the benchmark with the new configuration\r\nwill take longer and might not result in a more accurate measurement.\r\nIn that case benchmark-ips will re-print this warning with an even\r\nmore aggressive configuration option suggestions. It is a good idea\r\nto try to follow benchmark-ips's advice a couple of times (each time\r\nre-running the benchmark with new options), letting it escalate\r\nits configuration repeatedly (a good rule is to give up after 3-4\r\niterations).\r\n```\r\n\r\nImplement suggested changes:\r\n\r\n```ruby\r\n# bench.rb\r\nrequire 'benchmark/ips'\r\n\r\n# I like to make my VMs work instead of sleeping ;)\r\ndef work(num)\r\n  nnum = num * 300 # magic const to make the example work\r\n  i = 0\r\n  while i < nnum\r\n    i+=1\r\n  end\r\n  i\r\nend\r\n\r\nBenchmark.ips do |x|\r\n  x.sample_duration = 0.4\r\n  x.time = 10\r\n  \r\n  x.report(\"work(1000 + r(500))\") do\r\n    work(1000 + rand(500))\r\n  end\r\n\r\n  x.report(\"work(1000 + r(400))\") do\r\n    work(1000 + rand(400))\r\n  end\r\n\r\n  x.compare!\r\nend\r\n```\r\n\r\n```\r\n> ruby -Ilib bench.rb ⏎\r\nWarming up --------------------------------------\r\n work(1000 + r(500))    61.000  i/100ms\r\n work(1000 + r(400))    64.000  i/100ms\r\nCalculating -------------------------------------\r\n work(1000 + r(500))    154.372  (± 1.3%) i/s -      1.586k in  10.276211s\r\n work(1000 + r(400))    161.165  (± 1.2%) i/s -      1.664k in  10.326133s\r\n\r\nComparison:\r\n work(1000 + r(400)):      161.2 i/s\r\n work(1000 + r(500)):      154.4 i/s - 1.04x slower\r\n```\r\n✨ 🎉 "},"comment":{"url":"https://api.github.com/repos/evanphx/benchmark-ips/issues/comments/217997815","html_url":"https://github.com/evanphx/benchmark-ips/pull/68#issuecomment-217997815","issue_url":"https://api.github.com/repos/evanphx/benchmark-ips/issues/68","id":217997815,"user":{"login":"evanphx","id":7,"avatar_url":"https://avatars.githubusercontent.com/u/7?v=3","gravatar_id":"","url":"https://api.github.com/users/evanphx","html_url":"https://github.com/evanphx","followers_url":"https://api.github.com/users/evanphx/followers","following_url":"https://api.github.com/users/evanphx/following{/other_user}","gists_url":"https://api.github.com/users/evanphx/gists{/gist_id}","starred_url":"https://api.github.com/users/evanphx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/evanphx/subscriptions","organizations_url":"https://api.github.com/users/evanphx/orgs","repos_url":"https://api.github.com/users/evanphx/repos","events_url":"https://api.github.com/users/evanphx/events{/privacy}","received_events_url":"https://api.github.com/users/evanphx/received_events","type":"User","site_admin":false},"created_at":"2016-05-09T21:37:51Z","updated_at":"2016-05-09T21:37:51Z","body":"I don't mind exposing how this works, but:\r\n\r\n* What is the unit of  `sample_duration`?\r\n* What is the unit of `time`?\r\n* This breaks the public API of `Benchmark.compare`, which we probably should not do."}}