{"action":"created","comment":{"url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/pulls/comments/182555274","pull_request_review_id":113374835,"id":182555274,"diff_hunk":"@@ -0,0 +1,79 @@\n+\"\"\"\n+This module contains integration tests which verify the behavior of the storage\n+module against supported cloud storage providers. The tests for the local\n+(directory-based) storage backend are re-used and run for each supported cloud\n+storage provider. To add tests for a new provider, create a new subclass of\n+CloudTestFile.\n+\n+To run tests for Google Cloud Platform, set the following environment variables:\n+- GCP_STORAGE_KEY\n+- GCP_STORAGE_SECRET\n+\n+\"\"\"\n+import os\n+import random\n+import unittest\n+\n+import requests\n+\n+from server.extensions import storage\n+from tests.test_files import TestFile\n+\n+\n+class CloudTestFile(TestFile):\n+    storage_provider = \"\"\n+    key_env_name = \"\"\n+    secret_env_name = \"\"\n+\n+    @classmethod\n+    def setUpClass(cls):\n+        super().setUpClass()\n+\n+        cls.storage_key = os.environ.get(cls.key_env_name)\n+        cls.storage_secret = os.environ.get(cls.secret_env_name)\n+        if not cls.storage_key or not cls.storage_secret:\n+            raise unittest.SkipTest(\"Cloud storage credentials for \"\n+                                    \"%s not configured\" % cls.storage_provider)\n+\n+    def create_app(self):\n+        os.environ[\"STORAGE_PROVIDER\"] = self.storage_provider\n+        os.environ[\"STORAGE_KEY\"] = self.storage_key\n+        os.environ[\"STORAGE_SECRET\"] = self.storage_secret\n+        os.environ.setdefault(\"STORAGE_CONTAINER\", \"okpycloudfilestest%d\" % random.randint(0, 100000))","path":"tests/test_files_cloud.py","position":null,"original_position":42,"commit_id":"66dfffbf6014553ba07123ed40af7473bac8306d","original_commit_id":"aff57b68d44265e3c4a07a9381ac5a13f08f75f2","user":{"login":"c-w","id":1086421,"avatar_url":"https://avatars0.githubusercontent.com/u/1086421?v=4","gravatar_id":"","url":"https://api.github.com/users/c-w","html_url":"https://github.com/c-w","followers_url":"https://api.github.com/users/c-w/followers","following_url":"https://api.github.com/users/c-w/following{/other_user}","gists_url":"https://api.github.com/users/c-w/gists{/gist_id}","starred_url":"https://api.github.com/users/c-w/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/c-w/subscriptions","organizations_url":"https://api.github.com/users/c-w/orgs","repos_url":"https://api.github.com/users/c-w/repos","events_url":"https://api.github.com/users/c-w/events{/privacy}","received_events_url":"https://api.github.com/users/c-w/received_events","type":"User","site_admin":false},"body":"Done in 7b2040c","created_at":"2018-04-18T20:14:59Z","updated_at":"2018-04-18T20:14:59Z","html_url":"https://github.com/Cal-CS-61A-Staff/ok/pull/1246#discussion_r182555274","pull_request_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/pulls/1246","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/pulls/comments/182555274"},"html":{"href":"https://github.com/Cal-CS-61A-Staff/ok/pull/1246#discussion_r182555274"},"pull_request":{"href":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/pulls/1246"}},"in_reply_to_id":182161478},"pull_request":{"url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/pulls/1246","id":182233743,"html_url":"https://github.com/Cal-CS-61A-Staff/ok/pull/1246","diff_url":"https://github.com/Cal-CS-61A-Staff/ok/pull/1246.diff","patch_url":"https://github.com/Cal-CS-61A-Staff/ok/pull/1246.patch","issue_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/issues/1246","number":1246,"state":"open","locked":false,"title":"Add integration tests for cloud storage","user":{"login":"c-w","id":1086421,"avatar_url":"https://avatars0.githubusercontent.com/u/1086421?v=4","gravatar_id":"","url":"https://api.github.com/users/c-w","html_url":"https://github.com/c-w","followers_url":"https://api.github.com/users/c-w/followers","following_url":"https://api.github.com/users/c-w/following{/other_user}","gists_url":"https://api.github.com/users/c-w/gists{/gist_id}","starred_url":"https://api.github.com/users/c-w/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/c-w/subscriptions","organizations_url":"https://api.github.com/users/c-w/orgs","repos_url":"https://api.github.com/users/c-w/repos","events_url":"https://api.github.com/users/c-w/events{/privacy}","received_events_url":"https://api.github.com/users/c-w/received_events","type":"User","site_admin":false},"body":"In order to facilitate onboarding of new storage backends in the future (e.g. Azure Blob Storage), this pull request refactors the existing tests for the local storage backend to also run for cloud storage backends.\r\n\r\nThe integration tests are disabled by default. For Google Cloud Storage, the integration tests can be enabled by setting the environment variables `GCP_STORAGE_KEY` and `GCP_STORAGE_SECRET`.\r\n\r\nThis pull request also fixes two edge-cases in the storage implementation that were uncovered by the tests:\r\n\r\n1) The use of `datetime.strftime('%s')` relies on platform-specific undocumented behavior and may not work across Python versions or deployment environments. This was switched to a cross-platform way of converting a datetime to epoch via `datetime.timestamp()`.\r\n\r\n2) Submitting malicious files to Google Cloud Storage returned a 400 error rejecting the file. To mirror the behavior of the local storage provider, \"special\" parts of files uploaded to non-local storage such as `.` and `..` now get sanitized to underscores.\r\n\r\nFurthermore, the pull request also:\r\n\r\n3) Updates to the latest version of apache-libcloud which improves handling of missing files. Without this update, fetching non-existing blobs produces a 302 result and `test_malicious_local_get_blob` fails.\r\n\r\n4) Adds new tests to verify the results of binary downloads (previously only text downloads were verified).\r\n\r\n5) Disables parallel test execution on CircleCI. The cloud storage tests rely on mutating environment variables before creation of the Flask app which means that the tests are no longer guaranteed to be threadsafe.\r\n\r\nAll the changes described above can be found in d16b5ac. The commit depends on 42e40d6 which introduces more parameterization to the test settings such that the storage backend used during the tests can be configured via environment variables.\r\n\r\nThe pull request also contains a number of additional minor fixes:\r\n- 3b273be and 0ff08d0: Remove creation of directories in settings module and instead ensure that the storage module always creates a container if it doesn't yet exist.\r\n- 0559126: Add SQLite journal file to git-ignore to avoid committing the binary file (e.g during long tests runs).\r\n- aff57b6: Remove direct usage of pytest for exception checking and instead use standard unittest like the rest of the tests in the project.\r\n","created_at":"2018-04-17T17:03:30Z","updated_at":"2018-04-18T20:14:59Z","closed_at":null,"merged_at":null,"merge_commit_sha":"5ddd8e1a16a28c7473c2f81bb21282ec39a74bf8","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/pulls/1246/commits","review_comments_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/pulls/1246/comments","review_comment_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/pulls/comments{/number}","comments_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/issues/1246/comments","statuses_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/statuses/66dfffbf6014553ba07123ed40af7473bac8306d","head":{"label":"c-w:enhancement/c-w/cloud-storage-backend-tests","ref":"enhancement/c-w/cloud-storage-backend-tests","sha":"66dfffbf6014553ba07123ed40af7473bac8306d","user":{"login":"c-w","id":1086421,"avatar_url":"https://avatars0.githubusercontent.com/u/1086421?v=4","gravatar_id":"","url":"https://api.github.com/users/c-w","html_url":"https://github.com/c-w","followers_url":"https://api.github.com/users/c-w/followers","following_url":"https://api.github.com/users/c-w/following{/other_user}","gists_url":"https://api.github.com/users/c-w/gists{/gist_id}","starred_url":"https://api.github.com/users/c-w/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/c-w/subscriptions","organizations_url":"https://api.github.com/users/c-w/orgs","repos_url":"https://api.github.com/users/c-w/repos","events_url":"https://api.github.com/users/c-w/events{/privacy}","received_events_url":"https://api.github.com/users/c-w/received_events","type":"User","site_admin":false},"repo":{"id":129758517,"name":"ok","full_name":"c-w/ok","owner":{"login":"c-w","id":1086421,"avatar_url":"https://avatars0.githubusercontent.com/u/1086421?v=4","gravatar_id":"","url":"https://api.github.com/users/c-w","html_url":"https://github.com/c-w","followers_url":"https://api.github.com/users/c-w/followers","following_url":"https://api.github.com/users/c-w/following{/other_user}","gists_url":"https://api.github.com/users/c-w/gists{/gist_id}","starred_url":"https://api.github.com/users/c-w/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/c-w/subscriptions","organizations_url":"https://api.github.com/users/c-w/orgs","repos_url":"https://api.github.com/users/c-w/repos","events_url":"https://api.github.com/users/c-w/events{/privacy}","received_events_url":"https://api.github.com/users/c-w/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/c-w/ok","description":"ok.py supports programming projects by running tests, tracking progress, and assisting in debugging.","fork":true,"url":"https://api.github.com/repos/c-w/ok","forks_url":"https://api.github.com/repos/c-w/ok/forks","keys_url":"https://api.github.com/repos/c-w/ok/keys{/key_id}","collaborators_url":"https://api.github.com/repos/c-w/ok/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/c-w/ok/teams","hooks_url":"https://api.github.com/repos/c-w/ok/hooks","issue_events_url":"https://api.github.com/repos/c-w/ok/issues/events{/number}","events_url":"https://api.github.com/repos/c-w/ok/events","assignees_url":"https://api.github.com/repos/c-w/ok/assignees{/user}","branches_url":"https://api.github.com/repos/c-w/ok/branches{/branch}","tags_url":"https://api.github.com/repos/c-w/ok/tags","blobs_url":"https://api.github.com/repos/c-w/ok/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/c-w/ok/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/c-w/ok/git/refs{/sha}","trees_url":"https://api.github.com/repos/c-w/ok/git/trees{/sha}","statuses_url":"https://api.github.com/repos/c-w/ok/statuses/{sha}","languages_url":"https://api.github.com/repos/c-w/ok/languages","stargazers_url":"https://api.github.com/repos/c-w/ok/stargazers","contributors_url":"https://api.github.com/repos/c-w/ok/contributors","subscribers_url":"https://api.github.com/repos/c-w/ok/subscribers","subscription_url":"https://api.github.com/repos/c-w/ok/subscription","commits_url":"https://api.github.com/repos/c-w/ok/commits{/sha}","git_commits_url":"https://api.github.com/repos/c-w/ok/git/commits{/sha}","comments_url":"https://api.github.com/repos/c-w/ok/comments{/number}","issue_comment_url":"https://api.github.com/repos/c-w/ok/issues/comments{/number}","contents_url":"https://api.github.com/repos/c-w/ok/contents/{+path}","compare_url":"https://api.github.com/repos/c-w/ok/compare/{base}...{head}","merges_url":"https://api.github.com/repos/c-w/ok/merges","archive_url":"https://api.github.com/repos/c-w/ok/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/c-w/ok/downloads","issues_url":"https://api.github.com/repos/c-w/ok/issues{/number}","pulls_url":"https://api.github.com/repos/c-w/ok/pulls{/number}","milestones_url":"https://api.github.com/repos/c-w/ok/milestones{/number}","notifications_url":"https://api.github.com/repos/c-w/ok/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/c-w/ok/labels{/name}","releases_url":"https://api.github.com/repos/c-w/ok/releases{/id}","deployments_url":"https://api.github.com/repos/c-w/ok/deployments","created_at":"2018-04-16T14:42:56Z","updated_at":"2018-04-16T21:11:41Z","pushed_at":"2018-04-18T20:14:38Z","git_url":"git://github.com/c-w/ok.git","ssh_url":"git@github.com:c-w/ok.git","clone_url":"https://github.com/c-w/ok.git","svn_url":"https://github.com/c-w/ok","homepage":null,"size":20751,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"Cal-CS-61A-Staff:master","ref":"master","sha":"dc72cd67a174a41a2a0932ba54ef3fc0cb4c9d94","user":{"login":"Cal-CS-61A-Staff","id":7565272,"avatar_url":"https://avatars3.githubusercontent.com/u/7565272?v=4","gravatar_id":"","url":"https://api.github.com/users/Cal-CS-61A-Staff","html_url":"https://github.com/Cal-CS-61A-Staff","followers_url":"https://api.github.com/users/Cal-CS-61A-Staff/followers","following_url":"https://api.github.com/users/Cal-CS-61A-Staff/following{/other_user}","gists_url":"https://api.github.com/users/Cal-CS-61A-Staff/gists{/gist_id}","starred_url":"https://api.github.com/users/Cal-CS-61A-Staff/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Cal-CS-61A-Staff/subscriptions","organizations_url":"https://api.github.com/users/Cal-CS-61A-Staff/orgs","repos_url":"https://api.github.com/users/Cal-CS-61A-Staff/repos","events_url":"https://api.github.com/users/Cal-CS-61A-Staff/events{/privacy}","received_events_url":"https://api.github.com/users/Cal-CS-61A-Staff/received_events","type":"Organization","site_admin":false},"repo":{"id":19992691,"name":"ok","full_name":"Cal-CS-61A-Staff/ok","owner":{"login":"Cal-CS-61A-Staff","id":7565272,"avatar_url":"https://avatars3.githubusercontent.com/u/7565272?v=4","gravatar_id":"","url":"https://api.github.com/users/Cal-CS-61A-Staff","html_url":"https://github.com/Cal-CS-61A-Staff","followers_url":"https://api.github.com/users/Cal-CS-61A-Staff/followers","following_url":"https://api.github.com/users/Cal-CS-61A-Staff/following{/other_user}","gists_url":"https://api.github.com/users/Cal-CS-61A-Staff/gists{/gist_id}","starred_url":"https://api.github.com/users/Cal-CS-61A-Staff/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Cal-CS-61A-Staff/subscriptions","organizations_url":"https://api.github.com/users/Cal-CS-61A-Staff/orgs","repos_url":"https://api.github.com/users/Cal-CS-61A-Staff/repos","events_url":"https://api.github.com/users/Cal-CS-61A-Staff/events{/privacy}","received_events_url":"https://api.github.com/users/Cal-CS-61A-Staff/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/Cal-CS-61A-Staff/ok","description":"ok.py supports programming projects by running tests, tracking progress, and assisting in debugging.","fork":false,"url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok","forks_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/forks","keys_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/teams","hooks_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/hooks","issue_events_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/issues/events{/number}","events_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/events","assignees_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/assignees{/user}","branches_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/branches{/branch}","tags_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/tags","blobs_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/git/refs{/sha}","trees_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/statuses/{sha}","languages_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/languages","stargazers_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/stargazers","contributors_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/contributors","subscribers_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/subscribers","subscription_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/subscription","commits_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/commits{/sha}","git_commits_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/git/commits{/sha}","comments_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/comments{/number}","issue_comment_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/issues/comments{/number}","contents_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/contents/{+path}","compare_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/merges","archive_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/downloads","issues_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/issues{/number}","pulls_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/pulls{/number}","milestones_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/milestones{/number}","notifications_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/labels{/name}","releases_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/releases{/id}","deployments_url":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/deployments","created_at":"2014-05-20T18:36:23Z","updated_at":"2018-04-18T00:47:19Z","pushed_at":"2018-04-18T20:14:39Z","git_url":"git://github.com/Cal-CS-61A-Staff/ok.git","ssh_url":"git@github.com:Cal-CS-61A-Staff/ok.git","clone_url":"https://github.com/Cal-CS-61A-Staff/ok.git","svn_url":"https://github.com/Cal-CS-61A-Staff/ok","homepage":null,"size":20820,"stargazers_count":105,"watchers_count":105,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":42,"mirror_url":null,"archived":false,"open_issues_count":118,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":42,"open_issues":118,"watchers":105,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/pulls/1246"},"html":{"href":"https://github.com/Cal-CS-61A-Staff/ok/pull/1246"},"issue":{"href":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/issues/1246"},"comments":{"href":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/issues/1246/comments"},"review_comments":{"href":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/pulls/1246/comments"},"review_comment":{"href":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/pulls/1246/commits"},"statuses":{"href":"https://api.github.com/repos/Cal-CS-61A-Staff/ok/statuses/66dfffbf6014553ba07123ed40af7473bac8306d"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"c-w"}}