{"action":"created","comment":{"url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/pulls/comments/48724520","id":48724520,"diff_hunk":"@@ -0,0 +1,206 @@\n+import time\n+\n+import boto.rds\n+import psycopg2\n+import bcrypt\n+\n+from . import utils\n+\n+\n+class RDS(object):\n+    def __init__(self, region, logger=None, profile_name=None):\n+        self.conn = boto.rds.connect_to_region(\n+            region,\n+            profile_name=profile_name)\n+        self.log = logger or (lambda *args, **kwargs: None)\n+\n+    def get_instance(self, url=None, id=None):\n+        for instance in self.conn.get_all_dbinstances():\n+            if url and instance.endpoint[0] == url.split(':')[0]:\n+                return instance\n+            if id and instance.id == id:\n+                return instance\n+\n+    def create_new_snapshot(self, snapshot_id, instance_id):\n+        \"\"\"Create a new RDS snapshot deleting the existing one if there\n+        \"\"\"\n+        try:\n+            self.conn.get_all_dbsnapshots(snapshot_id)\n+            self.conn.delete_dbsnapshot(snapshot_id)","path":"dmaws/rds.py","position":29,"original_position":29,"commit_id":"1b0208e99e762502712632540ee985b43ad0e2c2","original_commit_id":"1b0208e99e762502712632540ee985b43ad0e2c2","user":{"login":"robyoung","id":14287,"avatar_url":"https://avatars.githubusercontent.com/u/14287?v=3","gravatar_id":"","url":"https://api.github.com/users/robyoung","html_url":"https://github.com/robyoung","followers_url":"https://api.github.com/users/robyoung/followers","following_url":"https://api.github.com/users/robyoung/following{/other_user}","gists_url":"https://api.github.com/users/robyoung/gists{/gist_id}","starred_url":"https://api.github.com/users/robyoung/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/robyoung/subscriptions","organizations_url":"https://api.github.com/users/robyoung/orgs","repos_url":"https://api.github.com/users/robyoung/repos","events_url":"https://api.github.com/users/robyoung/events{/privacy}","received_events_url":"https://api.github.com/users/robyoung/received_events","type":"User","site_admin":false},"body":"Yes, good call.","created_at":"2016-01-04T11:56:49Z","updated_at":"2016-01-04T11:56:49Z","html_url":"https://github.com/alphagov/digitalmarketplace-aws/pull/170#discussion_r48724520","pull_request_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/pulls/170","_links":{"self":{"href":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/pulls/comments/48724520"},"html":{"href":"https://github.com/alphagov/digitalmarketplace-aws/pull/170#discussion_r48724520"},"pull_request":{"href":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/pulls/170"}}},"pull_request":{"url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/pulls/170","id":54860900,"html_url":"https://github.com/alphagov/digitalmarketplace-aws/pull/170","diff_url":"https://github.com/alphagov/digitalmarketplace-aws/pull/170.diff","patch_url":"https://github.com/alphagov/digitalmarketplace-aws/pull/170.patch","issue_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/issues/170","number":170,"state":"open","locked":false,"title":"Add command to export database data","user":{"login":"robyoung","id":14287,"avatar_url":"https://avatars.githubusercontent.com/u/14287?v=3","gravatar_id":"","url":"https://api.github.com/users/robyoung","html_url":"https://github.com/robyoung","followers_url":"https://api.github.com/users/robyoung/followers","following_url":"https://api.github.com/users/robyoung/following{/other_user}","gists_url":"https://api.github.com/users/robyoung/gists{/gist_id}","starred_url":"https://api.github.com/users/robyoung/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/robyoung/subscriptions","organizations_url":"https://api.github.com/users/robyoung/orgs","repos_url":"https://api.github.com/users/robyoung/repos","events_url":"https://api.github.com/users/robyoung/events{/privacy}","received_events_url":"https://api.github.com/users/robyoung/received_events","type":"User","site_admin":false},"body":"This adds a command to export data from the database of an environment cleaned up for a specific environment. The result of running this command is two `pg_dump` files (one for staging and one for preview) that can be loaded into another database.\r\n\r\n## The basic process\r\n- Create a new snapshot of database\r\n- Restore a new instance from this snapshot\r\n- Run SQL commands to scrub data for staging\r\n- `pg_dump` to a file for staging\r\n- Run SQL commands to further scrub data for preview\r\n- `pg_dump` to a file for preview\r\n\r\n## The data scrubbing\r\nMost of the scrubbing is related to the requirement that no data related to an open framework should live anywhere other than production and we should avoid the risk of sending emails to users from anywhere other than production. I think that this demands more discussion as staging is considered equal to production and would benefit from having the same data in it.\r\n\r\n### Staging\r\n- Update all users to remove their name and email address\r\n- Remove any draft services and supplier framework records for frameworks that are still open\r\n- Remove any suppliers that are not associated with any framework. This leads on from the previous point, they are likely to be a supplier on an 'open' framework.\r\n- Remove all audit events. These contain personal information and changes related so applications to 'open' frameworks peppered throughout. Unpicking it would be hard work and error prone.\r\n- Blank out declarations leaving just the status as these contain sensitive information.\r\n\r\n### Preview\r\n- Update all passwords to 'Password1234' so that we can easily assume the role of accounts.\r\n\r\n## Overwriting data\r\nImporting this into another environment will overwrite the entire database resulting in anything manually entered or modified on that environment being lost. This is probably ok for staging but may be difficult on preview. I suggest that we don't automatically import on preview but rather do it on demand.","created_at":"2015-12-31T11:50:07Z","updated_at":"2016-01-04T11:56:49Z","closed_at":null,"merged_at":null,"merge_commit_sha":"d95ffd564a0d2b0432c0d32a2ea3572683aa22eb","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/pulls/170/commits","review_comments_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/pulls/170/comments","review_comment_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/pulls/comments{/number}","comments_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/issues/170/comments","statuses_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/statuses/1b0208e99e762502712632540ee985b43ad0e2c2","head":{"label":"alphagov:syncdata","ref":"syncdata","sha":"1b0208e99e762502712632540ee985b43ad0e2c2","user":{"login":"alphagov","id":596977,"avatar_url":"https://avatars.githubusercontent.com/u/596977?v=3","gravatar_id":"","url":"https://api.github.com/users/alphagov","html_url":"https://github.com/alphagov","followers_url":"https://api.github.com/users/alphagov/followers","following_url":"https://api.github.com/users/alphagov/following{/other_user}","gists_url":"https://api.github.com/users/alphagov/gists{/gist_id}","starred_url":"https://api.github.com/users/alphagov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alphagov/subscriptions","organizations_url":"https://api.github.com/users/alphagov/orgs","repos_url":"https://api.github.com/users/alphagov/repos","events_url":"https://api.github.com/users/alphagov/events{/privacy}","received_events_url":"https://api.github.com/users/alphagov/received_events","type":"Organization","site_admin":false},"repo":{"id":31316356,"name":"digitalmarketplace-aws","full_name":"alphagov/digitalmarketplace-aws","owner":{"login":"alphagov","id":596977,"avatar_url":"https://avatars.githubusercontent.com/u/596977?v=3","gravatar_id":"","url":"https://api.github.com/users/alphagov","html_url":"https://github.com/alphagov","followers_url":"https://api.github.com/users/alphagov/followers","following_url":"https://api.github.com/users/alphagov/following{/other_user}","gists_url":"https://api.github.com/users/alphagov/gists{/gist_id}","starred_url":"https://api.github.com/users/alphagov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alphagov/subscriptions","organizations_url":"https://api.github.com/users/alphagov/orgs","repos_url":"https://api.github.com/users/alphagov/repos","events_url":"https://api.github.com/users/alphagov/events{/privacy}","received_events_url":"https://api.github.com/users/alphagov/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/alphagov/digitalmarketplace-aws","description":"Ansible playbooks to set up AWS infrastructure for Digital Marketplace apps","fork":false,"url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws","forks_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/forks","keys_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/keys{/key_id}","collaborators_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/teams","hooks_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/hooks","issue_events_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/issues/events{/number}","events_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/events","assignees_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/assignees{/user}","branches_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/branches{/branch}","tags_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/tags","blobs_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/git/refs{/sha}","trees_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/git/trees{/sha}","statuses_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/statuses/{sha}","languages_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/languages","stargazers_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/stargazers","contributors_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/contributors","subscribers_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/subscribers","subscription_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/subscription","commits_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/commits{/sha}","git_commits_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/git/commits{/sha}","comments_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/comments{/number}","issue_comment_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/issues/comments{/number}","contents_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/contents/{+path}","compare_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/compare/{base}...{head}","merges_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/merges","archive_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/downloads","issues_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/issues{/number}","pulls_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/pulls{/number}","milestones_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/milestones{/number}","notifications_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/labels{/name}","releases_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/releases{/id}","created_at":"2015-02-25T14:07:07Z","updated_at":"2015-07-24T21:23:45Z","pushed_at":"2016-01-04T11:46:06Z","git_url":"git://github.com/alphagov/digitalmarketplace-aws.git","ssh_url":"git@github.com:alphagov/digitalmarketplace-aws.git","clone_url":"https://github.com/alphagov/digitalmarketplace-aws.git","svn_url":"https://github.com/alphagov/digitalmarketplace-aws","homepage":null,"size":474,"stargazers_count":1,"watchers_count":1,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":1,"forks":1,"open_issues":1,"watchers":1,"default_branch":"master"}},"base":{"label":"alphagov:master","ref":"master","sha":"ef13f8ab73bdb5e298c4539cc403dcd85c128207","user":{"login":"alphagov","id":596977,"avatar_url":"https://avatars.githubusercontent.com/u/596977?v=3","gravatar_id":"","url":"https://api.github.com/users/alphagov","html_url":"https://github.com/alphagov","followers_url":"https://api.github.com/users/alphagov/followers","following_url":"https://api.github.com/users/alphagov/following{/other_user}","gists_url":"https://api.github.com/users/alphagov/gists{/gist_id}","starred_url":"https://api.github.com/users/alphagov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alphagov/subscriptions","organizations_url":"https://api.github.com/users/alphagov/orgs","repos_url":"https://api.github.com/users/alphagov/repos","events_url":"https://api.github.com/users/alphagov/events{/privacy}","received_events_url":"https://api.github.com/users/alphagov/received_events","type":"Organization","site_admin":false},"repo":{"id":31316356,"name":"digitalmarketplace-aws","full_name":"alphagov/digitalmarketplace-aws","owner":{"login":"alphagov","id":596977,"avatar_url":"https://avatars.githubusercontent.com/u/596977?v=3","gravatar_id":"","url":"https://api.github.com/users/alphagov","html_url":"https://github.com/alphagov","followers_url":"https://api.github.com/users/alphagov/followers","following_url":"https://api.github.com/users/alphagov/following{/other_user}","gists_url":"https://api.github.com/users/alphagov/gists{/gist_id}","starred_url":"https://api.github.com/users/alphagov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alphagov/subscriptions","organizations_url":"https://api.github.com/users/alphagov/orgs","repos_url":"https://api.github.com/users/alphagov/repos","events_url":"https://api.github.com/users/alphagov/events{/privacy}","received_events_url":"https://api.github.com/users/alphagov/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/alphagov/digitalmarketplace-aws","description":"Ansible playbooks to set up AWS infrastructure for Digital Marketplace apps","fork":false,"url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws","forks_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/forks","keys_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/keys{/key_id}","collaborators_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/teams","hooks_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/hooks","issue_events_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/issues/events{/number}","events_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/events","assignees_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/assignees{/user}","branches_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/branches{/branch}","tags_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/tags","blobs_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/git/refs{/sha}","trees_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/git/trees{/sha}","statuses_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/statuses/{sha}","languages_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/languages","stargazers_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/stargazers","contributors_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/contributors","subscribers_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/subscribers","subscription_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/subscription","commits_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/commits{/sha}","git_commits_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/git/commits{/sha}","comments_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/comments{/number}","issue_comment_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/issues/comments{/number}","contents_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/contents/{+path}","compare_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/compare/{base}...{head}","merges_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/merges","archive_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/downloads","issues_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/issues{/number}","pulls_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/pulls{/number}","milestones_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/milestones{/number}","notifications_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/labels{/name}","releases_url":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/releases{/id}","created_at":"2015-02-25T14:07:07Z","updated_at":"2015-07-24T21:23:45Z","pushed_at":"2016-01-04T11:46:06Z","git_url":"git://github.com/alphagov/digitalmarketplace-aws.git","ssh_url":"git@github.com:alphagov/digitalmarketplace-aws.git","clone_url":"https://github.com/alphagov/digitalmarketplace-aws.git","svn_url":"https://github.com/alphagov/digitalmarketplace-aws","homepage":null,"size":474,"stargazers_count":1,"watchers_count":1,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":1,"forks":1,"open_issues":1,"watchers":1,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/pulls/170"},"html":{"href":"https://github.com/alphagov/digitalmarketplace-aws/pull/170"},"issue":{"href":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/issues/170"},"comments":{"href":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/issues/170/comments"},"review_comments":{"href":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/pulls/170/comments"},"review_comment":{"href":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/pulls/170/commits"},"statuses":{"href":"https://api.github.com/repos/alphagov/digitalmarketplace-aws/statuses/1b0208e99e762502712632540ee985b43ad0e2c2"}}}}