{"action":"created","comment":{"url":"https://api.github.com/repos/reactioncommerce/reaction/pulls/comments/204886216","pull_request_review_id":140048628,"id":204886216,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIwNDg4NjIxNg==","diff_hunk":"@@ -187,7 +187,8 @@ export const Accounts = new SimpleSchema({\n   \"groups\": {\n     type: Array, // groupIds that user belongs to\n     optional: true,\n-    defaultValue: []\n+    defaultValue: [],\n+    index: 1","path":"imports/collections/schemas/accounts.js","position":6,"original_position":6,"commit_id":"d3798a0041f7b74dc02bc61b8fac83909f42c3da","original_commit_id":"a3476cdcf5d160191ab84ef032d403a6abb2681c","user":{"login":"dancastellon","id":849570,"node_id":"MDQ6VXNlcjg0OTU3MA==","avatar_url":"https://avatars3.githubusercontent.com/u/849570?v=4","gravatar_id":"","url":"https://api.github.com/users/dancastellon","html_url":"https://github.com/dancastellon","followers_url":"https://api.github.com/users/dancastellon/followers","following_url":"https://api.github.com/users/dancastellon/following{/other_user}","gists_url":"https://api.github.com/users/dancastellon/gists{/gist_id}","starred_url":"https://api.github.com/users/dancastellon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dancastellon/subscriptions","organizations_url":"https://api.github.com/users/dancastellon/orgs","repos_url":"https://api.github.com/users/dancastellon/repos","events_url":"https://api.github.com/users/dancastellon/events{/privacy}","received_events_url":"https://api.github.com/users/dancastellon/received_events","type":"User","site_admin":false},"body":"@Akarshit While there aren't a lot of groups, there are a lot of accounts, which is I added this index to Accounts.groups. If I don't have this index, wouldn't Mongo have to scan all 2.1 million accounts? I'm running the retailer dataset","created_at":"2018-07-24T19:41:46Z","updated_at":"2018-07-24T19:41:46Z","html_url":"https://github.com/reactioncommerce/reaction/pull/4449#discussion_r204886216","pull_request_url":"https://api.github.com/repos/reactioncommerce/reaction/pulls/4449","author_association":"NONE","_links":{"self":{"href":"https://api.github.com/repos/reactioncommerce/reaction/pulls/comments/204886216"},"html":{"href":"https://github.com/reactioncommerce/reaction/pull/4449#discussion_r204886216"},"pull_request":{"href":"https://api.github.com/repos/reactioncommerce/reaction/pulls/4449"}},"in_reply_to_id":203272712},"pull_request":{"url":"https://api.github.com/repos/reactioncommerce/reaction/pulls/4449","id":202029232,"node_id":"MDExOlB1bGxSZXF1ZXN0MjAyMDI5MjMy","html_url":"https://github.com/reactioncommerce/reaction/pull/4449","diff_url":"https://github.com/reactioncommerce/reaction/pull/4449.diff","patch_url":"https://github.com/reactioncommerce/reaction/pull/4449.patch","issue_url":"https://api.github.com/repos/reactioncommerce/reaction/issues/4449","number":4449,"state":"open","locked":false,"title":"WIP #4384 Improve app startup time when large number of Accounts/Users exists","user":{"login":"dancastellon","id":849570,"node_id":"MDQ6VXNlcjg0OTU3MA==","avatar_url":"https://avatars3.githubusercontent.com/u/849570?v=4","gravatar_id":"","url":"https://api.github.com/users/dancastellon","html_url":"https://github.com/dancastellon","followers_url":"https://api.github.com/users/dancastellon/followers","following_url":"https://api.github.com/users/dancastellon/following{/other_user}","gists_url":"https://api.github.com/users/dancastellon/gists{/gist_id}","starred_url":"https://api.github.com/users/dancastellon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dancastellon/subscriptions","organizations_url":"https://api.github.com/users/dancastellon/orgs","repos_url":"https://api.github.com/users/dancastellon/repos","events_url":"https://api.github.com/users/dancastellon/events{/privacy}","received_events_url":"https://api.github.com/users/dancastellon/received_events","type":"User","site_admin":false},"body":"Resolves #4384  \r\nImpact: **major**  \r\nType: **performance**\r\n\r\n## Issue\r\nWhen an RC install with a large number of Accounts/Users starts up, currently every user is loaded in a single query and updated in another query, to add roles to their accounts. The core function in question - `addRolesToGroups` - is called 3 times throughout the app and is intended to add new roles to groups and their associated accounts/users. Even if the roles passed to the function call are already in the groups passed, all Accounts are loaded and updated.\r\n\r\n## Solution\r\nThe solution: 1) make it so Groups and Accounts/Users are only updated if the given roles aren't already present in the group, 2) batch the Accounts find query and Users update query to update a maximum of 100k Accounts/Users per call (to avoid a RangeError), and 3) have the heaviest queries run asynchronously in the background so that the app can finish starting up and start serving user requests.\r\n\r\nThere is one catch: With larger datasets (2,100,000 Accounts/Users tested), although the heaviest queries are run in the background, and the app is able to load, it is noticeably slower than usual while clicking through pages, due to the Mongo load. On the other hand, this large processing is now only done on the first startup. Subsequent startups no longer add roles to 2.1m accounts, because they already exist.\r\n\r\n## Breaking changes\r\nNone\r\n\r\n\r\n## Testing\r\n1. Use reaction-data-loader to load the `ret` (retailer) dataset\r\n2. Start the app with `export METEOR_DISABLE_OPTIMISTIC_CACHING=1 && REACTION_LOG_LEVEL=\"DEBUG\" reaction`\r\n3. Confirm in logs that \"Reaction initialization finished\" takes less less than 15s\r\n4. Confirm you can visit app in browser and click around\r\n5. Confirm you see debug messages about the group batch processing. Ex: `Processing user role update #2 of 21 for customer group`, but can still use the app\r\n6. When finished processing, restart the app\r\n7. Confirm you see messages such as `Skipping roles already assigned to...`, and that once the app starts up, there are no `Processing user role updates...` messages\r\n8. Confirm roles are still added to correct groups and users","created_at":"2018-07-17T18:26:45Z","updated_at":"2018-07-24T19:41:46Z","closed_at":null,"merged_at":null,"merge_commit_sha":"fb4b486bb8d186d6509a85e5f60b01f0b06b3c16","assignee":null,"assignees":[],"requested_reviewers":[{"login":"zenweasel","id":72819,"node_id":"MDQ6VXNlcjcyODE5","avatar_url":"https://avatars1.githubusercontent.com/u/72819?v=4","gravatar_id":"","url":"https://api.github.com/users/zenweasel","html_url":"https://github.com/zenweasel","followers_url":"https://api.github.com/users/zenweasel/followers","following_url":"https://api.github.com/users/zenweasel/following{/other_user}","gists_url":"https://api.github.com/users/zenweasel/gists{/gist_id}","starred_url":"https://api.github.com/users/zenweasel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zenweasel/subscriptions","organizations_url":"https://api.github.com/users/zenweasel/orgs","repos_url":"https://api.github.com/users/zenweasel/repos","events_url":"https://api.github.com/users/zenweasel/events{/privacy}","received_events_url":"https://api.github.com/users/zenweasel/received_events","type":"User","site_admin":false}],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/reactioncommerce/reaction/pulls/4449/commits","review_comments_url":"https://api.github.com/repos/reactioncommerce/reaction/pulls/4449/comments","review_comment_url":"https://api.github.com/repos/reactioncommerce/reaction/pulls/comments{/number}","comments_url":"https://api.github.com/repos/reactioncommerce/reaction/issues/4449/comments","statuses_url":"https://api.github.com/repos/reactioncommerce/reaction/statuses/d3798a0041f7b74dc02bc61b8fac83909f42c3da","head":{"label":"reactioncommerce:perf-4384-dancastellon-num-accounts-affecting-startup","ref":"perf-4384-dancastellon-num-accounts-affecting-startup","sha":"d3798a0041f7b74dc02bc61b8fac83909f42c3da","user":{"login":"reactioncommerce","id":5605462,"node_id":"MDEyOk9yZ2FuaXphdGlvbjU2MDU0NjI=","avatar_url":"https://avatars0.githubusercontent.com/u/5605462?v=4","gravatar_id":"","url":"https://api.github.com/users/reactioncommerce","html_url":"https://github.com/reactioncommerce","followers_url":"https://api.github.com/users/reactioncommerce/followers","following_url":"https://api.github.com/users/reactioncommerce/following{/other_user}","gists_url":"https://api.github.com/users/reactioncommerce/gists{/gist_id}","starred_url":"https://api.github.com/users/reactioncommerce/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/reactioncommerce/subscriptions","organizations_url":"https://api.github.com/users/reactioncommerce/orgs","repos_url":"https://api.github.com/users/reactioncommerce/repos","events_url":"https://api.github.com/users/reactioncommerce/events{/privacy}","received_events_url":"https://api.github.com/users/reactioncommerce/received_events","type":"Organization","site_admin":false},"repo":{"id":12926963,"node_id":"MDEwOlJlcG9zaXRvcnkxMjkyNjk2Mw==","name":"reaction","full_name":"reactioncommerce/reaction","owner":{"login":"reactioncommerce","id":5605462,"node_id":"MDEyOk9yZ2FuaXphdGlvbjU2MDU0NjI=","avatar_url":"https://avatars0.githubusercontent.com/u/5605462?v=4","gravatar_id":"","url":"https://api.github.com/users/reactioncommerce","html_url":"https://github.com/reactioncommerce","followers_url":"https://api.github.com/users/reactioncommerce/followers","following_url":"https://api.github.com/users/reactioncommerce/following{/other_user}","gists_url":"https://api.github.com/users/reactioncommerce/gists{/gist_id}","starred_url":"https://api.github.com/users/reactioncommerce/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/reactioncommerce/subscriptions","organizations_url":"https://api.github.com/users/reactioncommerce/orgs","repos_url":"https://api.github.com/users/reactioncommerce/repos","events_url":"https://api.github.com/users/reactioncommerce/events{/privacy}","received_events_url":"https://api.github.com/users/reactioncommerce/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/reactioncommerce/reaction","description":"Reaction is a customizable, real-time reactive, JavaScript commerce platform.","fork":false,"url":"https://api.github.com/repos/reactioncommerce/reaction","forks_url":"https://api.github.com/repos/reactioncommerce/reaction/forks","keys_url":"https://api.github.com/repos/reactioncommerce/reaction/keys{/key_id}","collaborators_url":"https://api.github.com/repos/reactioncommerce/reaction/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/reactioncommerce/reaction/teams","hooks_url":"https://api.github.com/repos/reactioncommerce/reaction/hooks","issue_events_url":"https://api.github.com/repos/reactioncommerce/reaction/issues/events{/number}","events_url":"https://api.github.com/repos/reactioncommerce/reaction/events","assignees_url":"https://api.github.com/repos/reactioncommerce/reaction/assignees{/user}","branches_url":"https://api.github.com/repos/reactioncommerce/reaction/branches{/branch}","tags_url":"https://api.github.com/repos/reactioncommerce/reaction/tags","blobs_url":"https://api.github.com/repos/reactioncommerce/reaction/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/reactioncommerce/reaction/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/reactioncommerce/reaction/git/refs{/sha}","trees_url":"https://api.github.com/repos/reactioncommerce/reaction/git/trees{/sha}","statuses_url":"https://api.github.com/repos/reactioncommerce/reaction/statuses/{sha}","languages_url":"https://api.github.com/repos/reactioncommerce/reaction/languages","stargazers_url":"https://api.github.com/repos/reactioncommerce/reaction/stargazers","contributors_url":"https://api.github.com/repos/reactioncommerce/reaction/contributors","subscribers_url":"https://api.github.com/repos/reactioncommerce/reaction/subscribers","subscription_url":"https://api.github.com/repos/reactioncommerce/reaction/subscription","commits_url":"https://api.github.com/repos/reactioncommerce/reaction/commits{/sha}","git_commits_url":"https://api.github.com/repos/reactioncommerce/reaction/git/commits{/sha}","comments_url":"https://api.github.com/repos/reactioncommerce/reaction/comments{/number}","issue_comment_url":"https://api.github.com/repos/reactioncommerce/reaction/issues/comments{/number}","contents_url":"https://api.github.com/repos/reactioncommerce/reaction/contents/{+path}","compare_url":"https://api.github.com/repos/reactioncommerce/reaction/compare/{base}...{head}","merges_url":"https://api.github.com/repos/reactioncommerce/reaction/merges","archive_url":"https://api.github.com/repos/reactioncommerce/reaction/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/reactioncommerce/reaction/downloads","issues_url":"https://api.github.com/repos/reactioncommerce/reaction/issues{/number}","pulls_url":"https://api.github.com/repos/reactioncommerce/reaction/pulls{/number}","milestones_url":"https://api.github.com/repos/reactioncommerce/reaction/milestones{/number}","notifications_url":"https://api.github.com/repos/reactioncommerce/reaction/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/reactioncommerce/reaction/labels{/name}","releases_url":"https://api.github.com/repos/reactioncommerce/reaction/releases{/id}","deployments_url":"https://api.github.com/repos/reactioncommerce/reaction/deployments","created_at":"2013-09-18T15:30:36Z","updated_at":"2018-07-24T06:57:34Z","pushed_at":"2018-07-24T19:39:42Z","git_url":"git://github.com/reactioncommerce/reaction.git","ssh_url":"git@github.com:reactioncommerce/reaction.git","clone_url":"https://github.com/reactioncommerce/reaction.git","svn_url":"https://github.com/reactioncommerce/reaction","homepage":"https://reactioncommerce.com/","size":49097,"stargazers_count":7432,"watchers_count":7432,"language":"JavaScript","has_issues":true,"has_projects":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":1402,"mirror_url":null,"archived":false,"open_issues_count":395,"license":{"key":"gpl-3.0","name":"GNU General Public License v3.0","spdx_id":"GPL-3.0","url":"https://api.github.com/licenses/gpl-3.0","node_id":"MDc6TGljZW5zZTk="},"forks":1402,"open_issues":395,"watchers":7432,"default_branch":"master"}},"base":{"label":"reactioncommerce:release-1.14.0","ref":"release-1.14.0","sha":"0c77fa4f5496008c0e83da9388c12891d6ee1670","user":{"login":"reactioncommerce","id":5605462,"node_id":"MDEyOk9yZ2FuaXphdGlvbjU2MDU0NjI=","avatar_url":"https://avatars0.githubusercontent.com/u/5605462?v=4","gravatar_id":"","url":"https://api.github.com/users/reactioncommerce","html_url":"https://github.com/reactioncommerce","followers_url":"https://api.github.com/users/reactioncommerce/followers","following_url":"https://api.github.com/users/reactioncommerce/following{/other_user}","gists_url":"https://api.github.com/users/reactioncommerce/gists{/gist_id}","starred_url":"https://api.github.com/users/reactioncommerce/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/reactioncommerce/subscriptions","organizations_url":"https://api.github.com/users/reactioncommerce/orgs","repos_url":"https://api.github.com/users/reactioncommerce/repos","events_url":"https://api.github.com/users/reactioncommerce/events{/privacy}","received_events_url":"https://api.github.com/users/reactioncommerce/received_events","type":"Organization","site_admin":false},"repo":{"id":12926963,"node_id":"MDEwOlJlcG9zaXRvcnkxMjkyNjk2Mw==","name":"reaction","full_name":"reactioncommerce/reaction","owner":{"login":"reactioncommerce","id":5605462,"node_id":"MDEyOk9yZ2FuaXphdGlvbjU2MDU0NjI=","avatar_url":"https://avatars0.githubusercontent.com/u/5605462?v=4","gravatar_id":"","url":"https://api.github.com/users/reactioncommerce","html_url":"https://github.com/reactioncommerce","followers_url":"https://api.github.com/users/reactioncommerce/followers","following_url":"https://api.github.com/users/reactioncommerce/following{/other_user}","gists_url":"https://api.github.com/users/reactioncommerce/gists{/gist_id}","starred_url":"https://api.github.com/users/reactioncommerce/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/reactioncommerce/subscriptions","organizations_url":"https://api.github.com/users/reactioncommerce/orgs","repos_url":"https://api.github.com/users/reactioncommerce/repos","events_url":"https://api.github.com/users/reactioncommerce/events{/privacy}","received_events_url":"https://api.github.com/users/reactioncommerce/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/reactioncommerce/reaction","description":"Reaction is a customizable, real-time reactive, JavaScript commerce platform.","fork":false,"url":"https://api.github.com/repos/reactioncommerce/reaction","forks_url":"https://api.github.com/repos/reactioncommerce/reaction/forks","keys_url":"https://api.github.com/repos/reactioncommerce/reaction/keys{/key_id}","collaborators_url":"https://api.github.com/repos/reactioncommerce/reaction/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/reactioncommerce/reaction/teams","hooks_url":"https://api.github.com/repos/reactioncommerce/reaction/hooks","issue_events_url":"https://api.github.com/repos/reactioncommerce/reaction/issues/events{/number}","events_url":"https://api.github.com/repos/reactioncommerce/reaction/events","assignees_url":"https://api.github.com/repos/reactioncommerce/reaction/assignees{/user}","branches_url":"https://api.github.com/repos/reactioncommerce/reaction/branches{/branch}","tags_url":"https://api.github.com/repos/reactioncommerce/reaction/tags","blobs_url":"https://api.github.com/repos/reactioncommerce/reaction/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/reactioncommerce/reaction/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/reactioncommerce/reaction/git/refs{/sha}","trees_url":"https://api.github.com/repos/reactioncommerce/reaction/git/trees{/sha}","statuses_url":"https://api.github.com/repos/reactioncommerce/reaction/statuses/{sha}","languages_url":"https://api.github.com/repos/reactioncommerce/reaction/languages","stargazers_url":"https://api.github.com/repos/reactioncommerce/reaction/stargazers","contributors_url":"https://api.github.com/repos/reactioncommerce/reaction/contributors","subscribers_url":"https://api.github.com/repos/reactioncommerce/reaction/subscribers","subscription_url":"https://api.github.com/repos/reactioncommerce/reaction/subscription","commits_url":"https://api.github.com/repos/reactioncommerce/reaction/commits{/sha}","git_commits_url":"https://api.github.com/repos/reactioncommerce/reaction/git/commits{/sha}","comments_url":"https://api.github.com/repos/reactioncommerce/reaction/comments{/number}","issue_comment_url":"https://api.github.com/repos/reactioncommerce/reaction/issues/comments{/number}","contents_url":"https://api.github.com/repos/reactioncommerce/reaction/contents/{+path}","compare_url":"https://api.github.com/repos/reactioncommerce/reaction/compare/{base}...{head}","merges_url":"https://api.github.com/repos/reactioncommerce/reaction/merges","archive_url":"https://api.github.com/repos/reactioncommerce/reaction/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/reactioncommerce/reaction/downloads","issues_url":"https://api.github.com/repos/reactioncommerce/reaction/issues{/number}","pulls_url":"https://api.github.com/repos/reactioncommerce/reaction/pulls{/number}","milestones_url":"https://api.github.com/repos/reactioncommerce/reaction/milestones{/number}","notifications_url":"https://api.github.com/repos/reactioncommerce/reaction/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/reactioncommerce/reaction/labels{/name}","releases_url":"https://api.github.com/repos/reactioncommerce/reaction/releases{/id}","deployments_url":"https://api.github.com/repos/reactioncommerce/reaction/deployments","created_at":"2013-09-18T15:30:36Z","updated_at":"2018-07-24T06:57:34Z","pushed_at":"2018-07-24T19:39:42Z","git_url":"git://github.com/reactioncommerce/reaction.git","ssh_url":"git@github.com:reactioncommerce/reaction.git","clone_url":"https://github.com/reactioncommerce/reaction.git","svn_url":"https://github.com/reactioncommerce/reaction","homepage":"https://reactioncommerce.com/","size":49097,"stargazers_count":7432,"watchers_count":7432,"language":"JavaScript","has_issues":true,"has_projects":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":1402,"mirror_url":null,"archived":false,"open_issues_count":395,"license":{"key":"gpl-3.0","name":"GNU General Public License v3.0","spdx_id":"GPL-3.0","url":"https://api.github.com/licenses/gpl-3.0","node_id":"MDc6TGljZW5zZTk="},"forks":1402,"open_issues":395,"watchers":7432,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/reactioncommerce/reaction/pulls/4449"},"html":{"href":"https://github.com/reactioncommerce/reaction/pull/4449"},"issue":{"href":"https://api.github.com/repos/reactioncommerce/reaction/issues/4449"},"comments":{"href":"https://api.github.com/repos/reactioncommerce/reaction/issues/4449/comments"},"review_comments":{"href":"https://api.github.com/repos/reactioncommerce/reaction/pulls/4449/comments"},"review_comment":{"href":"https://api.github.com/repos/reactioncommerce/reaction/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/reactioncommerce/reaction/pulls/4449/commits"},"statuses":{"href":"https://api.github.com/repos/reactioncommerce/reaction/statuses/d3798a0041f7b74dc02bc61b8fac83909f42c3da"}},"author_association":"NONE"}}
{"actor":{"display_login":"dancastellon"}}