{"action":"created","comment":{"url":"https://api.github.com/repos/bufferapp/core-authentication-service/pulls/comments/245264108","pull_request_review_id":189323748,"id":245264108,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDI0NTI2NDEwOA==","diff_hunk":"@@ -48,6 +50,10 @@ module.exports = async function doSanityCheck(replyUserCollection) {\n     }\n   }\n \n+  const missingAuthServiceIds = await getAccountsWithReplyLink({","path":"migration/stage1/doSanityCheck.js","position":53,"original_position":20,"commit_id":"b482707174845ae5e22e0a434f8b68e78bb7fc5b","original_commit_id":"b482707174845ae5e22e0a434f8b68e78bb7fc5b","user":{"login":"esclapes","id":829971,"node_id":"MDQ6VXNlcjgyOTk3MQ==","avatar_url":"https://avatars1.githubusercontent.com/u/829971?v=4","gravatar_id":"","url":"https://api.github.com/users/esclapes","html_url":"https://github.com/esclapes","followers_url":"https://api.github.com/users/esclapes/followers","following_url":"https://api.github.com/users/esclapes/following{/other_user}","gists_url":"https://api.github.com/users/esclapes/gists{/gist_id}","starred_url":"https://api.github.com/users/esclapes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/esclapes/subscriptions","organizations_url":"https://api.github.com/users/esclapes/orgs","repos_url":"https://api.github.com/users/esclapes/repos","events_url":"https://api.github.com/users/esclapes/events{/privacy}","received_events_url":"https://api.github.com/users/esclapes/received_events","type":"User","site_admin":false},"body":"Great question and thoughts here @djfarrelly, sharing a bit more of context here:\r\n\r\n> find any documents in the Account model's collection that say they're linked to a Reply user, but there is not Reply user found.\r\n\r\nNot exactly. The Reply users do exist, but do not have the `services.authServiceId` property set. This is an edge case, but we found 28 cases, and the real issue is that **these users would throw a duplicated key error when we try to migrate them**, because they already exists in the Auth DB.\r\n\r\n> this sanity check is really a validation script that all documents were migrated correctly\r\n\r\nCurrently, the sanity check runs _before_ the migration and (periodically thereafter), to double check that data in both databases is in sync and the migration assumptions are valid. The initial idea was that any errors found by the sanity check would be  investigated and solved by hand, but we ended up adding this automatic fix.\r\n\r\nI do see how this scripted fixing of Reply records does not really fit well within the overall flow of the migration. However, I still find it useful to reduce manual work, **how about we remove it from the general flow and keep it as a separate utility?**\r\n\r\ncc @CoxyBoris ","created_at":"2019-01-04T10:40:13Z","updated_at":"2019-01-04T10:40:13Z","html_url":"https://github.com/bufferapp/core-authentication-service/pull/66#discussion_r245264108","pull_request_url":"https://api.github.com/repos/bufferapp/core-authentication-service/pulls/66","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/bufferapp/core-authentication-service/pulls/comments/245264108"},"html":{"href":"https://github.com/bufferapp/core-authentication-service/pull/66#discussion_r245264108"},"pull_request":{"href":"https://api.github.com/repos/bufferapp/core-authentication-service/pulls/66"}},"in_reply_to_id":245194210},"pull_request":{"url":"https://api.github.com/repos/bufferapp/core-authentication-service/pulls/66","id":239084381,"node_id":"MDExOlB1bGxSZXF1ZXN0MjM5MDg0Mzgx","html_url":"https://github.com/bufferapp/core-authentication-service/pull/66","diff_url":"https://github.com/bufferapp/core-authentication-service/pull/66.diff","patch_url":"https://github.com/bufferapp/core-authentication-service/pull/66.patch","issue_url":"https://api.github.com/repos/bufferapp/core-authentication-service/issues/66","number":66,"state":"open","locked":false,"title":"CORE-327 migration script stage 1","user":{"login":"esclapes","id":829971,"node_id":"MDQ6VXNlcjgyOTk3MQ==","avatar_url":"https://avatars1.githubusercontent.com/u/829971?v=4","gravatar_id":"","url":"https://api.github.com/users/esclapes","html_url":"https://github.com/esclapes","followers_url":"https://api.github.com/users/esclapes/followers","following_url":"https://api.github.com/users/esclapes/following{/other_user}","gists_url":"https://api.github.com/users/esclapes/gists{/gist_id}","starred_url":"https://api.github.com/users/esclapes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/esclapes/subscriptions","organizations_url":"https://api.github.com/users/esclapes/orgs","repos_url":"https://api.github.com/users/esclapes/repos","events_url":"https://api.github.com/users/esclapes/events{/privacy}","received_events_url":"https://api.github.com/users/esclapes/received_events","type":"User","site_admin":false},"body":"This includes the migration logic for _Stage 1_: aka importing reply users into the authentication service\r\n\r\n**You need to rebuild the container after checking out this branch for the `-slim` image to be used. Just run the `buffer-dev` command `./dev rebuild core-authentication-service`**\r\n\r\nThe script will prompt you to provice the Mongo URIs for source and destination databases. There is a provided default URI (`mongodb://localhost:27019/stage1Test`) for easy local testing.\r\n\r\nThe script has three modes:\r\n\r\n* **Sanity check**: will read the source DB for users that have been already pushed (aka have the `services.authServiceId` property set) and will check that there is a existing matching account and the email and product link are in sync. It will report how many records have been checked and the found errors.\r\n* **Dry run**: will do the sanity check plus it will iterated through all non-migrated reply users to simulate the migration without changing anny of the records. This means that the new account will be generated and validated in memory but not saved.\r\n* **The real thing**: will do the migration of non-migrated reply users, saving the newly generated accounts and updating the reply users collection to set the `services.authServiceId` value.\r\n\r\n## Test env setup\r\n\r\nThere are a few unit tests for the migration, but it is far from being 100% tested. To test you may want to set up a local env for running the migration. These steps worked for me:\r\n\r\n1. Create a mongo database called `stage1Test1` in the `core-authentication-service-mongo` container, which is exposed in `localhost:27017`. (I used compass for this step).\r\n2. Get an export of the source and destination collections (you will need a mongo URI with access to each database):\r\n```\r\nmkdir private\r\nmongoexport --uri $REPLY_MONGO_URI --collection users -o ./private/replyUserExport.nljson\r\nmongoexport --uri $AUTH_MONGO_URI --collection accounts -o ./private/authUserExport.nljson\r\n```\r\n3. Import the databases to your local database:\r\n```\r\nmongoimport --uri mongodb://localhost:27019/stage1Test --file ./private/replyUserExport.nljson --collection users\r\nmongoimport --uri mongodb://localhost:27019/stage1Test --file ./private/authDevUsers.nljson --collection accounts\r\n```\r\n4. Run the migration `npm run migrate:stage1` it can be run outside of the container as the script is independent of the local link.\r\n\r\n\r\n","created_at":"2018-12-17T09:57:03Z","updated_at":"2019-01-04T10:40:13Z","closed_at":null,"merged_at":null,"merge_commit_sha":"b3f401eb3dffc139ccfa56e8c3428c6261e7bb65","assignee":null,"assignees":[],"requested_reviewers":[{"login":"philippemiguet","id":727592,"node_id":"MDQ6VXNlcjcyNzU5Mg==","avatar_url":"https://avatars2.githubusercontent.com/u/727592?v=4","gravatar_id":"","url":"https://api.github.com/users/philippemiguet","html_url":"https://github.com/philippemiguet","followers_url":"https://api.github.com/users/philippemiguet/followers","following_url":"https://api.github.com/users/philippemiguet/following{/other_user}","gists_url":"https://api.github.com/users/philippemiguet/gists{/gist_id}","starred_url":"https://api.github.com/users/philippemiguet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/philippemiguet/subscriptions","organizations_url":"https://api.github.com/users/philippemiguet/orgs","repos_url":"https://api.github.com/users/philippemiguet/repos","events_url":"https://api.github.com/users/philippemiguet/events{/privacy}","received_events_url":"https://api.github.com/users/philippemiguet/received_events","type":"User","site_admin":false},{"login":"CoxyBoris","id":4578065,"node_id":"MDQ6VXNlcjQ1NzgwNjU=","avatar_url":"https://avatars0.githubusercontent.com/u/4578065?v=4","gravatar_id":"","url":"https://api.github.com/users/CoxyBoris","html_url":"https://github.com/CoxyBoris","followers_url":"https://api.github.com/users/CoxyBoris/followers","following_url":"https://api.github.com/users/CoxyBoris/following{/other_user}","gists_url":"https://api.github.com/users/CoxyBoris/gists{/gist_id}","starred_url":"https://api.github.com/users/CoxyBoris/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/CoxyBoris/subscriptions","organizations_url":"https://api.github.com/users/CoxyBoris/orgs","repos_url":"https://api.github.com/users/CoxyBoris/repos","events_url":"https://api.github.com/users/CoxyBoris/events{/privacy}","received_events_url":"https://api.github.com/users/CoxyBoris/received_events","type":"User","site_admin":false}],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/bufferapp/core-authentication-service/pulls/66/commits","review_comments_url":"https://api.github.com/repos/bufferapp/core-authentication-service/pulls/66/comments","review_comment_url":"https://api.github.com/repos/bufferapp/core-authentication-service/pulls/comments{/number}","comments_url":"https://api.github.com/repos/bufferapp/core-authentication-service/issues/66/comments","statuses_url":"https://api.github.com/repos/bufferapp/core-authentication-service/statuses/b482707174845ae5e22e0a434f8b68e78bb7fc5b","head":{"label":"bufferapp:task/CORE-327-migration-script-stage-1","ref":"task/CORE-327-migration-script-stage-1","sha":"b482707174845ae5e22e0a434f8b68e78bb7fc5b","user":{"login":"bufferapp","id":798826,"node_id":"MDEyOk9yZ2FuaXphdGlvbjc5ODgyNg==","avatar_url":"https://avatars0.githubusercontent.com/u/798826?v=4","gravatar_id":"","url":"https://api.github.com/users/bufferapp","html_url":"https://github.com/bufferapp","followers_url":"https://api.github.com/users/bufferapp/followers","following_url":"https://api.github.com/users/bufferapp/following{/other_user}","gists_url":"https://api.github.com/users/bufferapp/gists{/gist_id}","starred_url":"https://api.github.com/users/bufferapp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bufferapp/subscriptions","organizations_url":"https://api.github.com/users/bufferapp/orgs","repos_url":"https://api.github.com/users/bufferapp/repos","events_url":"https://api.github.com/users/bufferapp/events{/privacy}","received_events_url":"https://api.github.com/users/bufferapp/received_events","type":"Organization","site_admin":false},"repo":{"id":127939778,"node_id":"MDEwOlJlcG9zaXRvcnkxMjc5Mzk3Nzg=","name":"core-authentication-service","full_name":"bufferapp/core-authentication-service","private":false,"owner":{"login":"bufferapp","id":798826,"node_id":"MDEyOk9yZ2FuaXphdGlvbjc5ODgyNg==","avatar_url":"https://avatars0.githubusercontent.com/u/798826?v=4","gravatar_id":"","url":"https://api.github.com/users/bufferapp","html_url":"https://github.com/bufferapp","followers_url":"https://api.github.com/users/bufferapp/followers","following_url":"https://api.github.com/users/bufferapp/following{/other_user}","gists_url":"https://api.github.com/users/bufferapp/gists{/gist_id}","starred_url":"https://api.github.com/users/bufferapp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bufferapp/subscriptions","organizations_url":"https://api.github.com/users/bufferapp/orgs","repos_url":"https://api.github.com/users/bufferapp/repos","events_url":"https://api.github.com/users/bufferapp/events{/privacy}","received_events_url":"https://api.github.com/users/bufferapp/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/bufferapp/core-authentication-service","description":"Buffer Authentication Service","fork":false,"url":"https://api.github.com/repos/bufferapp/core-authentication-service","forks_url":"https://api.github.com/repos/bufferapp/core-authentication-service/forks","keys_url":"https://api.github.com/repos/bufferapp/core-authentication-service/keys{/key_id}","collaborators_url":"https://api.github.com/repos/bufferapp/core-authentication-service/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/bufferapp/core-authentication-service/teams","hooks_url":"https://api.github.com/repos/bufferapp/core-authentication-service/hooks","issue_events_url":"https://api.github.com/repos/bufferapp/core-authentication-service/issues/events{/number}","events_url":"https://api.github.com/repos/bufferapp/core-authentication-service/events","assignees_url":"https://api.github.com/repos/bufferapp/core-authentication-service/assignees{/user}","branches_url":"https://api.github.com/repos/bufferapp/core-authentication-service/branches{/branch}","tags_url":"https://api.github.com/repos/bufferapp/core-authentication-service/tags","blobs_url":"https://api.github.com/repos/bufferapp/core-authentication-service/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/bufferapp/core-authentication-service/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/bufferapp/core-authentication-service/git/refs{/sha}","trees_url":"https://api.github.com/repos/bufferapp/core-authentication-service/git/trees{/sha}","statuses_url":"https://api.github.com/repos/bufferapp/core-authentication-service/statuses/{sha}","languages_url":"https://api.github.com/repos/bufferapp/core-authentication-service/languages","stargazers_url":"https://api.github.com/repos/bufferapp/core-authentication-service/stargazers","contributors_url":"https://api.github.com/repos/bufferapp/core-authentication-service/contributors","subscribers_url":"https://api.github.com/repos/bufferapp/core-authentication-service/subscribers","subscription_url":"https://api.github.com/repos/bufferapp/core-authentication-service/subscription","commits_url":"https://api.github.com/repos/bufferapp/core-authentication-service/commits{/sha}","git_commits_url":"https://api.github.com/repos/bufferapp/core-authentication-service/git/commits{/sha}","comments_url":"https://api.github.com/repos/bufferapp/core-authentication-service/comments{/number}","issue_comment_url":"https://api.github.com/repos/bufferapp/core-authentication-service/issues/comments{/number}","contents_url":"https://api.github.com/repos/bufferapp/core-authentication-service/contents/{+path}","compare_url":"https://api.github.com/repos/bufferapp/core-authentication-service/compare/{base}...{head}","merges_url":"https://api.github.com/repos/bufferapp/core-authentication-service/merges","archive_url":"https://api.github.com/repos/bufferapp/core-authentication-service/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/bufferapp/core-authentication-service/downloads","issues_url":"https://api.github.com/repos/bufferapp/core-authentication-service/issues{/number}","pulls_url":"https://api.github.com/repos/bufferapp/core-authentication-service/pulls{/number}","milestones_url":"https://api.github.com/repos/bufferapp/core-authentication-service/milestones{/number}","notifications_url":"https://api.github.com/repos/bufferapp/core-authentication-service/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/bufferapp/core-authentication-service/labels{/name}","releases_url":"https://api.github.com/repos/bufferapp/core-authentication-service/releases{/id}","deployments_url":"https://api.github.com/repos/bufferapp/core-authentication-service/deployments","created_at":"2018-04-03T16:53:07Z","updated_at":"2018-12-10T12:53:51Z","pushed_at":"2019-01-03T17:50:40Z","git_url":"git://github.com/bufferapp/core-authentication-service.git","ssh_url":"git@github.com:bufferapp/core-authentication-service.git","clone_url":"https://github.com/bufferapp/core-authentication-service.git","svn_url":"https://github.com/bufferapp/core-authentication-service","homepage":null,"size":840,"stargazers_count":2,"watchers_count":2,"language":"JavaScript","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"archived":false,"open_issues_count":2,"license":{"key":"mit","name":"MIT License","spdx_id":"MIT","url":"https://api.github.com/licenses/mit","node_id":"MDc6TGljZW5zZTEz"},"forks":1,"open_issues":2,"watchers":2,"default_branch":"master"}},"base":{"label":"bufferapp:master","ref":"master","sha":"5640cfb40afa2fbab2d5cbcbbe772729e6ed7379","user":{"login":"bufferapp","id":798826,"node_id":"MDEyOk9yZ2FuaXphdGlvbjc5ODgyNg==","avatar_url":"https://avatars0.githubusercontent.com/u/798826?v=4","gravatar_id":"","url":"https://api.github.com/users/bufferapp","html_url":"https://github.com/bufferapp","followers_url":"https://api.github.com/users/bufferapp/followers","following_url":"https://api.github.com/users/bufferapp/following{/other_user}","gists_url":"https://api.github.com/users/bufferapp/gists{/gist_id}","starred_url":"https://api.github.com/users/bufferapp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bufferapp/subscriptions","organizations_url":"https://api.github.com/users/bufferapp/orgs","repos_url":"https://api.github.com/users/bufferapp/repos","events_url":"https://api.github.com/users/bufferapp/events{/privacy}","received_events_url":"https://api.github.com/users/bufferapp/received_events","type":"Organization","site_admin":false},"repo":{"id":127939778,"node_id":"MDEwOlJlcG9zaXRvcnkxMjc5Mzk3Nzg=","name":"core-authentication-service","full_name":"bufferapp/core-authentication-service","private":false,"owner":{"login":"bufferapp","id":798826,"node_id":"MDEyOk9yZ2FuaXphdGlvbjc5ODgyNg==","avatar_url":"https://avatars0.githubusercontent.com/u/798826?v=4","gravatar_id":"","url":"https://api.github.com/users/bufferapp","html_url":"https://github.com/bufferapp","followers_url":"https://api.github.com/users/bufferapp/followers","following_url":"https://api.github.com/users/bufferapp/following{/other_user}","gists_url":"https://api.github.com/users/bufferapp/gists{/gist_id}","starred_url":"https://api.github.com/users/bufferapp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bufferapp/subscriptions","organizations_url":"https://api.github.com/users/bufferapp/orgs","repos_url":"https://api.github.com/users/bufferapp/repos","events_url":"https://api.github.com/users/bufferapp/events{/privacy}","received_events_url":"https://api.github.com/users/bufferapp/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/bufferapp/core-authentication-service","description":"Buffer Authentication Service","fork":false,"url":"https://api.github.com/repos/bufferapp/core-authentication-service","forks_url":"https://api.github.com/repos/bufferapp/core-authentication-service/forks","keys_url":"https://api.github.com/repos/bufferapp/core-authentication-service/keys{/key_id}","collaborators_url":"https://api.github.com/repos/bufferapp/core-authentication-service/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/bufferapp/core-authentication-service/teams","hooks_url":"https://api.github.com/repos/bufferapp/core-authentication-service/hooks","issue_events_url":"https://api.github.com/repos/bufferapp/core-authentication-service/issues/events{/number}","events_url":"https://api.github.com/repos/bufferapp/core-authentication-service/events","assignees_url":"https://api.github.com/repos/bufferapp/core-authentication-service/assignees{/user}","branches_url":"https://api.github.com/repos/bufferapp/core-authentication-service/branches{/branch}","tags_url":"https://api.github.com/repos/bufferapp/core-authentication-service/tags","blobs_url":"https://api.github.com/repos/bufferapp/core-authentication-service/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/bufferapp/core-authentication-service/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/bufferapp/core-authentication-service/git/refs{/sha}","trees_url":"https://api.github.com/repos/bufferapp/core-authentication-service/git/trees{/sha}","statuses_url":"https://api.github.com/repos/bufferapp/core-authentication-service/statuses/{sha}","languages_url":"https://api.github.com/repos/bufferapp/core-authentication-service/languages","stargazers_url":"https://api.github.com/repos/bufferapp/core-authentication-service/stargazers","contributors_url":"https://api.github.com/repos/bufferapp/core-authentication-service/contributors","subscribers_url":"https://api.github.com/repos/bufferapp/core-authentication-service/subscribers","subscription_url":"https://api.github.com/repos/bufferapp/core-authentication-service/subscription","commits_url":"https://api.github.com/repos/bufferapp/core-authentication-service/commits{/sha}","git_commits_url":"https://api.github.com/repos/bufferapp/core-authentication-service/git/commits{/sha}","comments_url":"https://api.github.com/repos/bufferapp/core-authentication-service/comments{/number}","issue_comment_url":"https://api.github.com/repos/bufferapp/core-authentication-service/issues/comments{/number}","contents_url":"https://api.github.com/repos/bufferapp/core-authentication-service/contents/{+path}","compare_url":"https://api.github.com/repos/bufferapp/core-authentication-service/compare/{base}...{head}","merges_url":"https://api.github.com/repos/bufferapp/core-authentication-service/merges","archive_url":"https://api.github.com/repos/bufferapp/core-authentication-service/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/bufferapp/core-authentication-service/downloads","issues_url":"https://api.github.com/repos/bufferapp/core-authentication-service/issues{/number}","pulls_url":"https://api.github.com/repos/bufferapp/core-authentication-service/pulls{/number}","milestones_url":"https://api.github.com/repos/bufferapp/core-authentication-service/milestones{/number}","notifications_url":"https://api.github.com/repos/bufferapp/core-authentication-service/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/bufferapp/core-authentication-service/labels{/name}","releases_url":"https://api.github.com/repos/bufferapp/core-authentication-service/releases{/id}","deployments_url":"https://api.github.com/repos/bufferapp/core-authentication-service/deployments","created_at":"2018-04-03T16:53:07Z","updated_at":"2018-12-10T12:53:51Z","pushed_at":"2019-01-03T17:50:40Z","git_url":"git://github.com/bufferapp/core-authentication-service.git","ssh_url":"git@github.com:bufferapp/core-authentication-service.git","clone_url":"https://github.com/bufferapp/core-authentication-service.git","svn_url":"https://github.com/bufferapp/core-authentication-service","homepage":null,"size":840,"stargazers_count":2,"watchers_count":2,"language":"JavaScript","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"archived":false,"open_issues_count":2,"license":{"key":"mit","name":"MIT License","spdx_id":"MIT","url":"https://api.github.com/licenses/mit","node_id":"MDc6TGljZW5zZTEz"},"forks":1,"open_issues":2,"watchers":2,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/bufferapp/core-authentication-service/pulls/66"},"html":{"href":"https://github.com/bufferapp/core-authentication-service/pull/66"},"issue":{"href":"https://api.github.com/repos/bufferapp/core-authentication-service/issues/66"},"comments":{"href":"https://api.github.com/repos/bufferapp/core-authentication-service/issues/66/comments"},"review_comments":{"href":"https://api.github.com/repos/bufferapp/core-authentication-service/pulls/66/comments"},"review_comment":{"href":"https://api.github.com/repos/bufferapp/core-authentication-service/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/bufferapp/core-authentication-service/pulls/66/commits"},"statuses":{"href":"https://api.github.com/repos/bufferapp/core-authentication-service/statuses/b482707174845ae5e22e0a434f8b68e78bb7fc5b"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"esclapes"}}