{"action":"created","comment":{"url":"https://api.github.com/repos/nodejs/node/pulls/comments/71975773","id":71975773,"diff_hunk":"@@ -1,181 +1,73 @@\n 'use strict';\n-var usage = 'node benchmark/compare.js ' +\n-            '<node-binary1> <node-binary2> ' +\n-            '[--html] [--red|-r] [--green|-g] ' +\n-            '[-- <type> [testFilter]]';\n \n-var show = 'both';\n-var nodes = [];\n-var html = false;\n-var benchmarks;\n-\n-for (var i = 2; i < process.argv.length; i++) {\n-  var arg = process.argv[i];\n-  switch (arg) {\n-    case '--red': case '-r':\n-      show = show === 'green' ? 'both' : 'red';\n-      break;\n-    case '--green': case '-g':\n-      show = show === 'red' ? 'both' : 'green';\n-      break;\n-    case '--html':\n-      html = true;\n-      break;\n-    case '-h': case '-?': case '--help':\n-      console.log(usage);\n-      process.exit(0);\n-      break;\n-    case '--':\n-      benchmarks = [];\n-      break;\n-    default:\n-      if (Array.isArray(benchmarks))\n-        benchmarks.push(arg);\n-      else\n-        nodes.push(arg);\n-      break;\n-  }\n+const fork = require('child_process').fork;\n+const path = require('path');\n+const CLI = require('./_cli.js');\n+\n+//\n+// Parse arguments\n+//\n+const cli = CLI(`usage: ./node benchmark/compare.js <type> ...","path":"benchmark/compare.js","position":null,"original_position":45,"commit_id":"9abe9105dbce4eecac42b7760127a9b422be1179","original_commit_id":"c471fdd78d69c6e402e893010f7cc2d92985d444","user":{"login":"addaleax","id":899444,"avatar_url":"https://avatars.githubusercontent.com/u/899444?v=3","gravatar_id":"","url":"https://api.github.com/users/addaleax","html_url":"https://github.com/addaleax","followers_url":"https://api.github.com/users/addaleax/followers","following_url":"https://api.github.com/users/addaleax/following{/other_user}","gists_url":"https://api.github.com/users/addaleax/gists{/gist_id}","starred_url":"https://api.github.com/users/addaleax/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/addaleax/subscriptions","organizations_url":"https://api.github.com/users/addaleax/orgs","repos_url":"https://api.github.com/users/addaleax/repos","events_url":"https://api.github.com/users/addaleax/events{/privacy}","received_events_url":"https://api.github.com/users/addaleax/received_events","type":"User","site_admin":false},"body":"> I will try and change the argument order, this appears to cause a lot of confusion for many people, but I would love to understand why.\r\n\r\nIf I had to guess, I’d say it’s because that’s the order usually suggested in man pages and `--help` texts, and maybe because the positional arguments are the ones one is most likely to spend more time editing before hitting enter… idk, maybe there’s more to it.","created_at":"2016-07-23T17:53:21Z","updated_at":"2016-07-23T17:53:21Z","html_url":"https://github.com/nodejs/node/pull/7094#discussion_r71975773","pull_request_url":"https://api.github.com/repos/nodejs/node/pulls/7094","_links":{"self":{"href":"https://api.github.com/repos/nodejs/node/pulls/comments/71975773"},"html":{"href":"https://github.com/nodejs/node/pull/7094#discussion_r71975773"},"pull_request":{"href":"https://api.github.com/repos/nodejs/node/pulls/7094"}}},"pull_request":{"url":"https://api.github.com/repos/nodejs/node/pulls/7094","id":72199973,"html_url":"https://github.com/nodejs/node/pull/7094","diff_url":"https://github.com/nodejs/node/pull/7094.diff","patch_url":"https://github.com/nodejs/node/pull/7094.patch","issue_url":"https://api.github.com/repos/nodejs/node/issues/7094","number":7094,"state":"open","locked":false,"title":"Refactor benchmark tools for statistical significance","user":{"login":"AndreasMadsen","id":505333,"avatar_url":"https://avatars.githubusercontent.com/u/505333?v=3","gravatar_id":"","url":"https://api.github.com/users/AndreasMadsen","html_url":"https://github.com/AndreasMadsen","followers_url":"https://api.github.com/users/AndreasMadsen/followers","following_url":"https://api.github.com/users/AndreasMadsen/following{/other_user}","gists_url":"https://api.github.com/users/AndreasMadsen/gists{/gist_id}","starred_url":"https://api.github.com/users/AndreasMadsen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AndreasMadsen/subscriptions","organizations_url":"https://api.github.com/users/AndreasMadsen/orgs","repos_url":"https://api.github.com/users/AndreasMadsen/repos","events_url":"https://api.github.com/users/AndreasMadsen/events{/privacy}","received_events_url":"https://api.github.com/users/AndreasMadsen/received_events","type":"User","site_admin":false},"body":"##### Checklist\r\n\r\n- [x] tests and code linting passes\r\n- [ ] a test and/or benchmark is included\r\n- [x] documentation is changed or added\r\n- [x] the commit message follows commit guidelines\r\n\r\n##### Affected core subsystem(s)\r\nbenchmark\r\n\r\n##### Description of change\r\n\r\nI have been rather confused about the benchmark suite and I don't think it is as user friendly as the rest of nodecore. This PR attempt to remove most of the confusion I was facing when I started using it. Primarily it:\r\n\r\n* removes unused/undocumented files\r\n* allows partially setting the benchmarks variables using process arguments.\r\n* refactor `compare.js` such comparing node versions and getting statistical significance is easy.\r\n* refactor the `plot.R` tool (now called `scatter`) to show a scatter plot with confidence bars.\r\n* refactor cli tools such the cli API is more homogeneous.\r\n* documents all the tools.\r\n* removes the implicit `process.exit(0)` after `bench.end()`.\r\n* uses `process.send` to avoid most parsing (the benchmark cli arguments haven't changed).\r\n\r\nThe specifics are documented in the commit messages. Please also see the the [new README](https://github.com/AndreasMadsen/node/blob/benchmark-refactor/benchmark/README.md) as quite a lot have changed (be sure the to check my spelling!).\r\n\r\n_Note that some benchmark takes a very long time to complete, e.g. `timers/timers.js\ttype=depth thousands=500` takes 11.25 min. Thus running it 30 times for statistical significance is unreasonable. I suspect the only reason why it is set to so many iterations is to get a small variance, but with the the new `compare` tool the variance can be estimated instead of being reduced. Thus we can reduce the number of iterations and still get the information we need. But I suggest we do that in another pull request, as is very different discussion._\r\n\r\n**Motivation (long story)**: I wanted to benchmark the effect of some `async_wrap` changes. I went to the `benchmark/` directory and read the [README](https://github.com/nodejs/node/blob/master/benchmark/README.md). However I quickly discovered that it was primarily about running benchmarks a single time and how to write benchmarks. And most importantly it didn't explain how to compare two node versions. _This is now documented in the [new README](https://github.com/AndreasMadsen/node/blob/benchmark-refactor/benchmark/README.md)._\r\n\r\nI then had to search for the tools myself and discovered the large amount of benchmarks files which where not put into categorized directories. I assumed they where somehow extra significant, but in reality they just appear to be unused. _These files are now removed._\r\n\r\nAfter discovering the compare tool, which has the cli API\r\n```\r\nnode benchmark/compare.js\r\n            <node-binary1> <node-binary2> +\r\n            [--html] [--red|-r] [--green|-g] +\r\n            [-- <type> [testFilter]]\r\n```\r\nI was confused about what the `--red`, `--green` was and how the `node-binary1` and `node-binary2` compared, should I write `./node-old ./node-new` or `./node-new ./node-old` if I wanted a positive improvement factor to signify an improvement? _The new compare API is_:\r\n```\r\nusage: ./node benchmark/compare.js <type> ...\r\n  --new    ./new-node-binary  new node binary (required)\r\n  --old    ./old-node-binary  old node binary (required)\r\n  --runs   30                 number of samples\r\n  --filter pattern            string to filter benchmark scripts\r\n  --var    variable=value     set benchmark variable (can be repeated)\r\n```\r\n\r\nAfter understanding `common.js` this it was still unclear if the performance was statistically significant different. I tried running the benchmark 5 times and got that 4/5 was an improvement, I was expecting it to have the same performance or be slower. (spoiler: it wasn't significant). _The compare.js script now runs the benchmarks many times (30 by default) and there is an R script to analyse the csv results._\r\n\r\nAt this point I wanted to do a rewrite of the benchmark tools (not the benchmarks themself) and changed a few other things in the process as well. - I'm a mathematician so I care a lot about statistical significance :)\r\n","created_at":"2016-06-01T15:18:38Z","updated_at":"2016-07-23T17:53:21Z","closed_at":null,"merged_at":null,"merge_commit_sha":"dea2c77f9631a484ac77056b931b98d46c32206b","assignee":{"login":"Fishrock123","id":1093990,"avatar_url":"https://avatars.githubusercontent.com/u/1093990?v=3","gravatar_id":"","url":"https://api.github.com/users/Fishrock123","html_url":"https://github.com/Fishrock123","followers_url":"https://api.github.com/users/Fishrock123/followers","following_url":"https://api.github.com/users/Fishrock123/following{/other_user}","gists_url":"https://api.github.com/users/Fishrock123/gists{/gist_id}","starred_url":"https://api.github.com/users/Fishrock123/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Fishrock123/subscriptions","organizations_url":"https://api.github.com/users/Fishrock123/orgs","repos_url":"https://api.github.com/users/Fishrock123/repos","events_url":"https://api.github.com/users/Fishrock123/events{/privacy}","received_events_url":"https://api.github.com/users/Fishrock123/received_events","type":"User","site_admin":false},"assignees":[{"login":"Fishrock123","id":1093990,"avatar_url":"https://avatars.githubusercontent.com/u/1093990?v=3","gravatar_id":"","url":"https://api.github.com/users/Fishrock123","html_url":"https://github.com/Fishrock123","followers_url":"https://api.github.com/users/Fishrock123/followers","following_url":"https://api.github.com/users/Fishrock123/following{/other_user}","gists_url":"https://api.github.com/users/Fishrock123/gists{/gist_id}","starred_url":"https://api.github.com/users/Fishrock123/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Fishrock123/subscriptions","organizations_url":"https://api.github.com/users/Fishrock123/orgs","repos_url":"https://api.github.com/users/Fishrock123/repos","events_url":"https://api.github.com/users/Fishrock123/events{/privacy}","received_events_url":"https://api.github.com/users/Fishrock123/received_events","type":"User","site_admin":false}],"milestone":null,"commits_url":"https://api.github.com/repos/nodejs/node/pulls/7094/commits","review_comments_url":"https://api.github.com/repos/nodejs/node/pulls/7094/comments","review_comment_url":"https://api.github.com/repos/nodejs/node/pulls/comments{/number}","comments_url":"https://api.github.com/repos/nodejs/node/issues/7094/comments","statuses_url":"https://api.github.com/repos/nodejs/node/statuses/9abe9105dbce4eecac42b7760127a9b422be1179","head":{"label":"AndreasMadsen:benchmark-refactor","ref":"benchmark-refactor","sha":"9abe9105dbce4eecac42b7760127a9b422be1179","user":{"login":"AndreasMadsen","id":505333,"avatar_url":"https://avatars.githubusercontent.com/u/505333?v=3","gravatar_id":"","url":"https://api.github.com/users/AndreasMadsen","html_url":"https://github.com/AndreasMadsen","followers_url":"https://api.github.com/users/AndreasMadsen/followers","following_url":"https://api.github.com/users/AndreasMadsen/following{/other_user}","gists_url":"https://api.github.com/users/AndreasMadsen/gists{/gist_id}","starred_url":"https://api.github.com/users/AndreasMadsen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AndreasMadsen/subscriptions","organizations_url":"https://api.github.com/users/AndreasMadsen/orgs","repos_url":"https://api.github.com/users/AndreasMadsen/repos","events_url":"https://api.github.com/users/AndreasMadsen/events{/privacy}","received_events_url":"https://api.github.com/users/AndreasMadsen/received_events","type":"User","site_admin":false},"repo":{"id":42876352,"name":"node","full_name":"AndreasMadsen/node","owner":{"login":"AndreasMadsen","id":505333,"avatar_url":"https://avatars.githubusercontent.com/u/505333?v=3","gravatar_id":"","url":"https://api.github.com/users/AndreasMadsen","html_url":"https://github.com/AndreasMadsen","followers_url":"https://api.github.com/users/AndreasMadsen/followers","following_url":"https://api.github.com/users/AndreasMadsen/following{/other_user}","gists_url":"https://api.github.com/users/AndreasMadsen/gists{/gist_id}","starred_url":"https://api.github.com/users/AndreasMadsen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AndreasMadsen/subscriptions","organizations_url":"https://api.github.com/users/AndreasMadsen/orgs","repos_url":"https://api.github.com/users/AndreasMadsen/repos","events_url":"https://api.github.com/users/AndreasMadsen/events{/privacy}","received_events_url":"https://api.github.com/users/AndreasMadsen/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/AndreasMadsen/node","description":"Node.js JavaScript runtime :sparkles::turtle::rocket::sparkles:","fork":true,"url":"https://api.github.com/repos/AndreasMadsen/node","forks_url":"https://api.github.com/repos/AndreasMadsen/node/forks","keys_url":"https://api.github.com/repos/AndreasMadsen/node/keys{/key_id}","collaborators_url":"https://api.github.com/repos/AndreasMadsen/node/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/AndreasMadsen/node/teams","hooks_url":"https://api.github.com/repos/AndreasMadsen/node/hooks","issue_events_url":"https://api.github.com/repos/AndreasMadsen/node/issues/events{/number}","events_url":"https://api.github.com/repos/AndreasMadsen/node/events","assignees_url":"https://api.github.com/repos/AndreasMadsen/node/assignees{/user}","branches_url":"https://api.github.com/repos/AndreasMadsen/node/branches{/branch}","tags_url":"https://api.github.com/repos/AndreasMadsen/node/tags","blobs_url":"https://api.github.com/repos/AndreasMadsen/node/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/AndreasMadsen/node/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/AndreasMadsen/node/git/refs{/sha}","trees_url":"https://api.github.com/repos/AndreasMadsen/node/git/trees{/sha}","statuses_url":"https://api.github.com/repos/AndreasMadsen/node/statuses/{sha}","languages_url":"https://api.github.com/repos/AndreasMadsen/node/languages","stargazers_url":"https://api.github.com/repos/AndreasMadsen/node/stargazers","contributors_url":"https://api.github.com/repos/AndreasMadsen/node/contributors","subscribers_url":"https://api.github.com/repos/AndreasMadsen/node/subscribers","subscription_url":"https://api.github.com/repos/AndreasMadsen/node/subscription","commits_url":"https://api.github.com/repos/AndreasMadsen/node/commits{/sha}","git_commits_url":"https://api.github.com/repos/AndreasMadsen/node/git/commits{/sha}","comments_url":"https://api.github.com/repos/AndreasMadsen/node/comments{/number}","issue_comment_url":"https://api.github.com/repos/AndreasMadsen/node/issues/comments{/number}","contents_url":"https://api.github.com/repos/AndreasMadsen/node/contents/{+path}","compare_url":"https://api.github.com/repos/AndreasMadsen/node/compare/{base}...{head}","merges_url":"https://api.github.com/repos/AndreasMadsen/node/merges","archive_url":"https://api.github.com/repos/AndreasMadsen/node/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/AndreasMadsen/node/downloads","issues_url":"https://api.github.com/repos/AndreasMadsen/node/issues{/number}","pulls_url":"https://api.github.com/repos/AndreasMadsen/node/pulls{/number}","milestones_url":"https://api.github.com/repos/AndreasMadsen/node/milestones{/number}","notifications_url":"https://api.github.com/repos/AndreasMadsen/node/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/AndreasMadsen/node/labels{/name}","releases_url":"https://api.github.com/repos/AndreasMadsen/node/releases{/id}","deployments_url":"https://api.github.com/repos/AndreasMadsen/node/deployments","created_at":"2015-09-21T15:48:29Z","updated_at":"2016-03-18T20:12:28Z","pushed_at":"2016-07-23T13:09:57Z","git_url":"git://github.com/AndreasMadsen/node.git","ssh_url":"git@github.com:AndreasMadsen/node.git","clone_url":"https://github.com/AndreasMadsen/node.git","svn_url":"https://github.com/AndreasMadsen/node","homepage":"https://nodejs.org","size":221257,"stargazers_count":0,"watchers_count":0,"language":"JavaScript","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"nodejs:master","ref":"master","sha":"b3127df59ab23baa68e915d62ea1997adb5669e0","user":{"login":"nodejs","id":9950313,"avatar_url":"https://avatars.githubusercontent.com/u/9950313?v=3","gravatar_id":"","url":"https://api.github.com/users/nodejs","html_url":"https://github.com/nodejs","followers_url":"https://api.github.com/users/nodejs/followers","following_url":"https://api.github.com/users/nodejs/following{/other_user}","gists_url":"https://api.github.com/users/nodejs/gists{/gist_id}","starred_url":"https://api.github.com/users/nodejs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nodejs/subscriptions","organizations_url":"https://api.github.com/users/nodejs/orgs","repos_url":"https://api.github.com/users/nodejs/repos","events_url":"https://api.github.com/users/nodejs/events{/privacy}","received_events_url":"https://api.github.com/users/nodejs/received_events","type":"Organization","site_admin":false},"repo":{"id":27193779,"name":"node","full_name":"nodejs/node","owner":{"login":"nodejs","id":9950313,"avatar_url":"https://avatars.githubusercontent.com/u/9950313?v=3","gravatar_id":"","url":"https://api.github.com/users/nodejs","html_url":"https://github.com/nodejs","followers_url":"https://api.github.com/users/nodejs/followers","following_url":"https://api.github.com/users/nodejs/following{/other_user}","gists_url":"https://api.github.com/users/nodejs/gists{/gist_id}","starred_url":"https://api.github.com/users/nodejs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nodejs/subscriptions","organizations_url":"https://api.github.com/users/nodejs/orgs","repos_url":"https://api.github.com/users/nodejs/repos","events_url":"https://api.github.com/users/nodejs/events{/privacy}","received_events_url":"https://api.github.com/users/nodejs/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/nodejs/node","description":"Node.js JavaScript runtime :sparkles::turtle::rocket::sparkles:","fork":false,"url":"https://api.github.com/repos/nodejs/node","forks_url":"https://api.github.com/repos/nodejs/node/forks","keys_url":"https://api.github.com/repos/nodejs/node/keys{/key_id}","collaborators_url":"https://api.github.com/repos/nodejs/node/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/nodejs/node/teams","hooks_url":"https://api.github.com/repos/nodejs/node/hooks","issue_events_url":"https://api.github.com/repos/nodejs/node/issues/events{/number}","events_url":"https://api.github.com/repos/nodejs/node/events","assignees_url":"https://api.github.com/repos/nodejs/node/assignees{/user}","branches_url":"https://api.github.com/repos/nodejs/node/branches{/branch}","tags_url":"https://api.github.com/repos/nodejs/node/tags","blobs_url":"https://api.github.com/repos/nodejs/node/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/nodejs/node/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/nodejs/node/git/refs{/sha}","trees_url":"https://api.github.com/repos/nodejs/node/git/trees{/sha}","statuses_url":"https://api.github.com/repos/nodejs/node/statuses/{sha}","languages_url":"https://api.github.com/repos/nodejs/node/languages","stargazers_url":"https://api.github.com/repos/nodejs/node/stargazers","contributors_url":"https://api.github.com/repos/nodejs/node/contributors","subscribers_url":"https://api.github.com/repos/nodejs/node/subscribers","subscription_url":"https://api.github.com/repos/nodejs/node/subscription","commits_url":"https://api.github.com/repos/nodejs/node/commits{/sha}","git_commits_url":"https://api.github.com/repos/nodejs/node/git/commits{/sha}","comments_url":"https://api.github.com/repos/nodejs/node/comments{/number}","issue_comment_url":"https://api.github.com/repos/nodejs/node/issues/comments{/number}","contents_url":"https://api.github.com/repos/nodejs/node/contents/{+path}","compare_url":"https://api.github.com/repos/nodejs/node/compare/{base}...{head}","merges_url":"https://api.github.com/repos/nodejs/node/merges","archive_url":"https://api.github.com/repos/nodejs/node/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/nodejs/node/downloads","issues_url":"https://api.github.com/repos/nodejs/node/issues{/number}","pulls_url":"https://api.github.com/repos/nodejs/node/pulls{/number}","milestones_url":"https://api.github.com/repos/nodejs/node/milestones{/number}","notifications_url":"https://api.github.com/repos/nodejs/node/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/nodejs/node/labels{/name}","releases_url":"https://api.github.com/repos/nodejs/node/releases{/id}","deployments_url":"https://api.github.com/repos/nodejs/node/deployments","created_at":"2014-11-26T19:57:11Z","updated_at":"2016-07-23T17:52:51Z","pushed_at":"2016-07-23T15:27:52Z","git_url":"git://github.com/nodejs/node.git","ssh_url":"git@github.com:nodejs/node.git","clone_url":"https://github.com/nodejs/node.git","svn_url":"https://github.com/nodejs/node","homepage":"https://nodejs.org","size":302042,"stargazers_count":25068,"watchers_count":25068,"language":"JavaScript","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":3790,"mirror_url":null,"open_issues_count":746,"forks":3790,"open_issues":746,"watchers":25068,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/nodejs/node/pulls/7094"},"html":{"href":"https://github.com/nodejs/node/pull/7094"},"issue":{"href":"https://api.github.com/repos/nodejs/node/issues/7094"},"comments":{"href":"https://api.github.com/repos/nodejs/node/issues/7094/comments"},"review_comments":{"href":"https://api.github.com/repos/nodejs/node/pulls/7094/comments"},"review_comment":{"href":"https://api.github.com/repos/nodejs/node/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/nodejs/node/pulls/7094/commits"},"statuses":{"href":"https://api.github.com/repos/nodejs/node/statuses/9abe9105dbce4eecac42b7760127a9b422be1179"}}}}
{"actor":{"display_login":"addaleax"}}