{"action":"reopened","number":1765,"pull_request":{"url":"https://api.github.com/repos/ianstormtaylor/slate/pulls/1765","id":180205664,"node_id":"MDExOlB1bGxSZXF1ZXN0MTgwMjA1NjY0","html_url":"https://github.com/ianstormtaylor/slate/pull/1765","diff_url":"https://github.com/ianstormtaylor/slate/pull/1765.diff","patch_url":"https://github.com/ianstormtaylor/slate/pull/1765.patch","issue_url":"https://api.github.com/repos/ianstormtaylor/slate/issues/1765","number":1765,"state":"open","locked":false,"title":"Benchmark consistence && Allow users to select benches to run","user":{"login":"zhujinxuan","id":746159,"node_id":"MDQ6VXNlcjc0NjE1OQ==","avatar_url":"https://avatars3.githubusercontent.com/u/746159?v=4","gravatar_id":"","url":"https://api.github.com/users/zhujinxuan","html_url":"https://github.com/zhujinxuan","followers_url":"https://api.github.com/users/zhujinxuan/followers","following_url":"https://api.github.com/users/zhujinxuan/following{/other_user}","gists_url":"https://api.github.com/users/zhujinxuan/gists{/gist_id}","starred_url":"https://api.github.com/users/zhujinxuan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zhujinxuan/subscriptions","organizations_url":"https://api.github.com/users/zhujinxuan/orgs","repos_url":"https://api.github.com/users/zhujinxuan/repos","events_url":"https://api.github.com/users/zhujinxuan/events{/privacy}","received_events_url":"https://api.github.com/users/zhujinxuan/received_events","type":"User","site_admin":false},"body":"## How to use this PR before merge\r\n0. fecth this branch to your local repo\r\n1. git checkout benchmark-consistence && yarn build && yarn benchmark:save\r\n2. git checout your-repo && yarn build\r\n3. git checkout benchmark-consistence && yarn benchmark\r\n\r\n\r\n\r\n--------\r\n## Problem:\r\nWe always have an inconsistent benchmark result.  The problems are caused by\r\nMost important problem:\r\n1. before and after in the Matcha is called by each suite, not by each bench.  \r\nOther problems:\r\n1. We need to call `global.gc()` before each bench.  Avoid calculating gc times in bench\r\n2. We shall not use babel-node in benchmark.  Because code parsing time and other time related in babel-code will be introduced in the benchmark.  \r\n3. We shall have different input in each bench run, to avoid some potential optimization.  (Generate new blocks/value with same data)\r\n4. a user-side problem:  I do not want to wait with no output when running the benchmark.\r\n5. Perhaps we shall use process.cpuUsage, to eliminate the effects of node optimization and influence of other processes.\r\n6. [Not in this PR] Perhaps we shall run node --expose-gc benchmark multi times for better robustness. \r\n\r\n## Proposal: Why we need a benchmark framework instead of using the existing one:\r\n1. : Other benchmark framework uses `process.hrtime()` to evaluate the time consuming.  It is good for I/O heavy tasks, but our focus is CPU heavy.  If we want to eliminate the problem by other processes, we would prefer process.cpuUsage. (For benchmarkJS, it uses lodash.now, which is a slow Date.now system call)\r\n\r\n2.  Most benchmark frameworks requires a fixed run number, but our tasks varies from 4 ops/sec (from-json) to 10,000 ops/sec (get-blocks).  It is very unhappy to special numbers for each tasks.\r\n\r\n3. There is an adaptive mode in matcha, without specializing the run number, but\r\n   3.1 It is out of maintenance\r\n   3.2 It calls process.hrtime() in each loop for adaptive mode.  I am not sure whether the time waiting for `process.hrtime` is comparable to 10,000 ops/sec.  \r\n   3.3 It does not solve other problems.\r\n\r\n4.  We want to exclude the time of gc in our processes, then we want to `node --expose-gc` and run `global.gc` each time before we run each bench.  \r\n\r\n5. Allow users to config about which benchmarks to run, we do not want to re-run all benchmarks every time.","created_at":"2018-04-09T03:58:24Z","updated_at":"2018-06-11T01:37:12Z","closed_at":null,"merged_at":null,"merge_commit_sha":"2c0c277fd6c18520cec8f9b119a9a268ce14bf0c","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[{"id":404055048,"node_id":"MDU6TGFiZWw0MDQwNTUwNDg=","url":"https://api.github.com/repos/ianstormtaylor/slate/labels/%E2%9A%91%20performance","name":"⚑ performance","color":"eeeeee","default":false}],"milestone":null,"commits_url":"https://api.github.com/repos/ianstormtaylor/slate/pulls/1765/commits","review_comments_url":"https://api.github.com/repos/ianstormtaylor/slate/pulls/1765/comments","review_comment_url":"https://api.github.com/repos/ianstormtaylor/slate/pulls/comments{/number}","comments_url":"https://api.github.com/repos/ianstormtaylor/slate/issues/1765/comments","statuses_url":"https://api.github.com/repos/ianstormtaylor/slate/statuses/19db5e80808e032cb5352449eaa1274689bada39","head":{"label":"zhujinxuan:benchmark-consistence","ref":"benchmark-consistence","sha":"19db5e80808e032cb5352449eaa1274689bada39","user":{"login":"zhujinxuan","id":746159,"node_id":"MDQ6VXNlcjc0NjE1OQ==","avatar_url":"https://avatars3.githubusercontent.com/u/746159?v=4","gravatar_id":"","url":"https://api.github.com/users/zhujinxuan","html_url":"https://github.com/zhujinxuan","followers_url":"https://api.github.com/users/zhujinxuan/followers","following_url":"https://api.github.com/users/zhujinxuan/following{/other_user}","gists_url":"https://api.github.com/users/zhujinxuan/gists{/gist_id}","starred_url":"https://api.github.com/users/zhujinxuan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zhujinxuan/subscriptions","organizations_url":"https://api.github.com/users/zhujinxuan/orgs","repos_url":"https://api.github.com/users/zhujinxuan/repos","events_url":"https://api.github.com/users/zhujinxuan/events{/privacy}","received_events_url":"https://api.github.com/users/zhujinxuan/received_events","type":"User","site_admin":false},"repo":{"id":117600430,"node_id":"MDEwOlJlcG9zaXRvcnkxMTc2MDA0MzA=","name":"slate","full_name":"zhujinxuan/slate","owner":{"login":"zhujinxuan","id":746159,"node_id":"MDQ6VXNlcjc0NjE1OQ==","avatar_url":"https://avatars3.githubusercontent.com/u/746159?v=4","gravatar_id":"","url":"https://api.github.com/users/zhujinxuan","html_url":"https://github.com/zhujinxuan","followers_url":"https://api.github.com/users/zhujinxuan/followers","following_url":"https://api.github.com/users/zhujinxuan/following{/other_user}","gists_url":"https://api.github.com/users/zhujinxuan/gists{/gist_id}","starred_url":"https://api.github.com/users/zhujinxuan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zhujinxuan/subscriptions","organizations_url":"https://api.github.com/users/zhujinxuan/orgs","repos_url":"https://api.github.com/users/zhujinxuan/repos","events_url":"https://api.github.com/users/zhujinxuan/events{/privacy}","received_events_url":"https://api.github.com/users/zhujinxuan/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/zhujinxuan/slate","description":"A completely customizable framework for building rich text editors.","fork":true,"url":"https://api.github.com/repos/zhujinxuan/slate","forks_url":"https://api.github.com/repos/zhujinxuan/slate/forks","keys_url":"https://api.github.com/repos/zhujinxuan/slate/keys{/key_id}","collaborators_url":"https://api.github.com/repos/zhujinxuan/slate/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/zhujinxuan/slate/teams","hooks_url":"https://api.github.com/repos/zhujinxuan/slate/hooks","issue_events_url":"https://api.github.com/repos/zhujinxuan/slate/issues/events{/number}","events_url":"https://api.github.com/repos/zhujinxuan/slate/events","assignees_url":"https://api.github.com/repos/zhujinxuan/slate/assignees{/user}","branches_url":"https://api.github.com/repos/zhujinxuan/slate/branches{/branch}","tags_url":"https://api.github.com/repos/zhujinxuan/slate/tags","blobs_url":"https://api.github.com/repos/zhujinxuan/slate/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/zhujinxuan/slate/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/zhujinxuan/slate/git/refs{/sha}","trees_url":"https://api.github.com/repos/zhujinxuan/slate/git/trees{/sha}","statuses_url":"https://api.github.com/repos/zhujinxuan/slate/statuses/{sha}","languages_url":"https://api.github.com/repos/zhujinxuan/slate/languages","stargazers_url":"https://api.github.com/repos/zhujinxuan/slate/stargazers","contributors_url":"https://api.github.com/repos/zhujinxuan/slate/contributors","subscribers_url":"https://api.github.com/repos/zhujinxuan/slate/subscribers","subscription_url":"https://api.github.com/repos/zhujinxuan/slate/subscription","commits_url":"https://api.github.com/repos/zhujinxuan/slate/commits{/sha}","git_commits_url":"https://api.github.com/repos/zhujinxuan/slate/git/commits{/sha}","comments_url":"https://api.github.com/repos/zhujinxuan/slate/comments{/number}","issue_comment_url":"https://api.github.com/repos/zhujinxuan/slate/issues/comments{/number}","contents_url":"https://api.github.com/repos/zhujinxuan/slate/contents/{+path}","compare_url":"https://api.github.com/repos/zhujinxuan/slate/compare/{base}...{head}","merges_url":"https://api.github.com/repos/zhujinxuan/slate/merges","archive_url":"https://api.github.com/repos/zhujinxuan/slate/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/zhujinxuan/slate/downloads","issues_url":"https://api.github.com/repos/zhujinxuan/slate/issues{/number}","pulls_url":"https://api.github.com/repos/zhujinxuan/slate/pulls{/number}","milestones_url":"https://api.github.com/repos/zhujinxuan/slate/milestones{/number}","notifications_url":"https://api.github.com/repos/zhujinxuan/slate/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/zhujinxuan/slate/labels{/name}","releases_url":"https://api.github.com/repos/zhujinxuan/slate/releases{/id}","deployments_url":"https://api.github.com/repos/zhujinxuan/slate/deployments","created_at":"2018-01-15T22:06:20Z","updated_at":"2018-05-30T23:25:10Z","pushed_at":"2018-06-11T01:28:57Z","git_url":"git://github.com/zhujinxuan/slate.git","ssh_url":"git@github.com:zhujinxuan/slate.git","clone_url":"https://github.com/zhujinxuan/slate.git","svn_url":"https://github.com/zhujinxuan/slate","homepage":"http://slatejs.org","size":182336,"stargazers_count":0,"watchers_count":0,"language":"JavaScript","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"mit","name":"MIT License","spdx_id":"MIT","url":"https://api.github.com/licenses/mit","node_id":"MDc6TGljZW5zZTEz"},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"ianstormtaylor:master","ref":"master","sha":"b26535fd947cb54153800799a1dfc5d4b7729218","user":{"login":"ianstormtaylor","id":311752,"node_id":"MDQ6VXNlcjMxMTc1Mg==","avatar_url":"https://avatars0.githubusercontent.com/u/311752?v=4","gravatar_id":"","url":"https://api.github.com/users/ianstormtaylor","html_url":"https://github.com/ianstormtaylor","followers_url":"https://api.github.com/users/ianstormtaylor/followers","following_url":"https://api.github.com/users/ianstormtaylor/following{/other_user}","gists_url":"https://api.github.com/users/ianstormtaylor/gists{/gist_id}","starred_url":"https://api.github.com/users/ianstormtaylor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ianstormtaylor/subscriptions","organizations_url":"https://api.github.com/users/ianstormtaylor/orgs","repos_url":"https://api.github.com/users/ianstormtaylor/repos","events_url":"https://api.github.com/users/ianstormtaylor/events{/privacy}","received_events_url":"https://api.github.com/users/ianstormtaylor/received_events","type":"User","site_admin":false},"repo":{"id":61412022,"node_id":"MDEwOlJlcG9zaXRvcnk2MTQxMjAyMg==","name":"slate","full_name":"ianstormtaylor/slate","owner":{"login":"ianstormtaylor","id":311752,"node_id":"MDQ6VXNlcjMxMTc1Mg==","avatar_url":"https://avatars0.githubusercontent.com/u/311752?v=4","gravatar_id":"","url":"https://api.github.com/users/ianstormtaylor","html_url":"https://github.com/ianstormtaylor","followers_url":"https://api.github.com/users/ianstormtaylor/followers","following_url":"https://api.github.com/users/ianstormtaylor/following{/other_user}","gists_url":"https://api.github.com/users/ianstormtaylor/gists{/gist_id}","starred_url":"https://api.github.com/users/ianstormtaylor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ianstormtaylor/subscriptions","organizations_url":"https://api.github.com/users/ianstormtaylor/orgs","repos_url":"https://api.github.com/users/ianstormtaylor/repos","events_url":"https://api.github.com/users/ianstormtaylor/events{/privacy}","received_events_url":"https://api.github.com/users/ianstormtaylor/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/ianstormtaylor/slate","description":"A completely customizable framework for building rich text editors.","fork":false,"url":"https://api.github.com/repos/ianstormtaylor/slate","forks_url":"https://api.github.com/repos/ianstormtaylor/slate/forks","keys_url":"https://api.github.com/repos/ianstormtaylor/slate/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ianstormtaylor/slate/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ianstormtaylor/slate/teams","hooks_url":"https://api.github.com/repos/ianstormtaylor/slate/hooks","issue_events_url":"https://api.github.com/repos/ianstormtaylor/slate/issues/events{/number}","events_url":"https://api.github.com/repos/ianstormtaylor/slate/events","assignees_url":"https://api.github.com/repos/ianstormtaylor/slate/assignees{/user}","branches_url":"https://api.github.com/repos/ianstormtaylor/slate/branches{/branch}","tags_url":"https://api.github.com/repos/ianstormtaylor/slate/tags","blobs_url":"https://api.github.com/repos/ianstormtaylor/slate/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ianstormtaylor/slate/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ianstormtaylor/slate/git/refs{/sha}","trees_url":"https://api.github.com/repos/ianstormtaylor/slate/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ianstormtaylor/slate/statuses/{sha}","languages_url":"https://api.github.com/repos/ianstormtaylor/slate/languages","stargazers_url":"https://api.github.com/repos/ianstormtaylor/slate/stargazers","contributors_url":"https://api.github.com/repos/ianstormtaylor/slate/contributors","subscribers_url":"https://api.github.com/repos/ianstormtaylor/slate/subscribers","subscription_url":"https://api.github.com/repos/ianstormtaylor/slate/subscription","commits_url":"https://api.github.com/repos/ianstormtaylor/slate/commits{/sha}","git_commits_url":"https://api.github.com/repos/ianstormtaylor/slate/git/commits{/sha}","comments_url":"https://api.github.com/repos/ianstormtaylor/slate/comments{/number}","issue_comment_url":"https://api.github.com/repos/ianstormtaylor/slate/issues/comments{/number}","contents_url":"https://api.github.com/repos/ianstormtaylor/slate/contents/{+path}","compare_url":"https://api.github.com/repos/ianstormtaylor/slate/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ianstormtaylor/slate/merges","archive_url":"https://api.github.com/repos/ianstormtaylor/slate/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ianstormtaylor/slate/downloads","issues_url":"https://api.github.com/repos/ianstormtaylor/slate/issues{/number}","pulls_url":"https://api.github.com/repos/ianstormtaylor/slate/pulls{/number}","milestones_url":"https://api.github.com/repos/ianstormtaylor/slate/milestones{/number}","notifications_url":"https://api.github.com/repos/ianstormtaylor/slate/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ianstormtaylor/slate/labels{/name}","releases_url":"https://api.github.com/repos/ianstormtaylor/slate/releases{/id}","deployments_url":"https://api.github.com/repos/ianstormtaylor/slate/deployments","created_at":"2016-06-18T01:52:42Z","updated_at":"2018-06-11T01:05:27Z","pushed_at":"2018-06-11T01:28:59Z","git_url":"git://github.com/ianstormtaylor/slate.git","ssh_url":"git@github.com:ianstormtaylor/slate.git","clone_url":"https://github.com/ianstormtaylor/slate.git","svn_url":"https://github.com/ianstormtaylor/slate","homepage":"http://slatejs.org","size":193302,"stargazers_count":7601,"watchers_count":7601,"language":"JavaScript","has_issues":true,"has_projects":false,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":601,"mirror_url":null,"archived":false,"open_issues_count":244,"license":{"key":"mit","name":"MIT License","spdx_id":"MIT","url":"https://api.github.com/licenses/mit","node_id":"MDc6TGljZW5zZTEz"},"forks":601,"open_issues":244,"watchers":7601,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/ianstormtaylor/slate/pulls/1765"},"html":{"href":"https://github.com/ianstormtaylor/slate/pull/1765"},"issue":{"href":"https://api.github.com/repos/ianstormtaylor/slate/issues/1765"},"comments":{"href":"https://api.github.com/repos/ianstormtaylor/slate/issues/1765/comments"},"review_comments":{"href":"https://api.github.com/repos/ianstormtaylor/slate/pulls/1765/comments"},"review_comment":{"href":"https://api.github.com/repos/ianstormtaylor/slate/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/ianstormtaylor/slate/pulls/1765/commits"},"statuses":{"href":"https://api.github.com/repos/ianstormtaylor/slate/statuses/19db5e80808e032cb5352449eaa1274689bada39"}},"author_association":"CONTRIBUTOR","merged":false,"mergeable":null,"rebaseable":null,"mergeable_state":"unknown","merged_by":null,"comments":46,"review_comments":44,"maintainer_can_modify":true,"commits":88,"additions":1691,"deletions":489,"changed_files":78}}
{"actor":{"display_login":"zhujinxuan"}}