{"action":"created","issue":{"url":"https://api.github.com/repos/edx/edx-platform/issues/11286","repository_url":"https://api.github.com/repos/edx/edx-platform","labels_url":"https://api.github.com/repos/edx/edx-platform/issues/11286/labels{/name}","comments_url":"https://api.github.com/repos/edx/edx-platform/issues/11286/comments","events_url":"https://api.github.com/repos/edx/edx-platform/issues/11286/events","html_url":"https://github.com/edx/edx-platform/pull/11286","id":127837114,"number":11286,"title":"Add OpenStack Swift support","user":{"login":"omarkhan","id":793379,"avatar_url":"https://avatars.githubusercontent.com/u/793379?v=3","gravatar_id":"","url":"https://api.github.com/users/omarkhan","html_url":"https://github.com/omarkhan","followers_url":"https://api.github.com/users/omarkhan/followers","following_url":"https://api.github.com/users/omarkhan/following{/other_user}","gists_url":"https://api.github.com/users/omarkhan/gists{/gist_id}","starred_url":"https://api.github.com/users/omarkhan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/omarkhan/subscriptions","organizations_url":"https://api.github.com/users/omarkhan/orgs","repos_url":"https://api.github.com/users/omarkhan/repos","events_url":"https://api.github.com/users/omarkhan/events{/privacy}","received_events_url":"https://api.github.com/users/omarkhan/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/edx/edx-platform/labels/awaiting%20prioritization","name":"awaiting prioritization","color":"fad8c7"},{"url":"https://api.github.com/repos/edx/edx-platform/labels/open-source-contribution","name":"open-source-contribution","color":"bfe5bf"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":37,"created_at":"2016-01-21T03:27:48Z","updated_at":"2016-03-01T18:54:45Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/edx/edx-platform/pulls/11286","html_url":"https://github.com/edx/edx-platform/pull/11286","diff_url":"https://github.com/edx/edx-platform/pull/11286.diff","patch_url":"https://github.com/edx/edx-platform/pull/11286.patch"},"body":"**Background:** The edX platform includes a number of features that use S3 directly. These include:\r\n\r\n- Instructor CSV data exports\r\n- Video storage\r\n- xqueue\r\n- Student Identity Verification (photo uploads)\r\n- Analytics event dumps (via a logrotate script)\r\n\r\nFor deployments outside of AWS it would make sense to use a different cloud storage provider, or even the local filesystem.\r\n\r\nIn particular, OpenStack deployments will want to use an OpenStack Swift Object Store for user-uploaded content. Although Swift offers S3 API compatibility as a plugin (called \"Swift3\") in theory, in practice the compatibility is limited, often not installed on the OpenStack instance, and hard to get working.\r\n\r\nThis pull request is an attempt to solve this problem by replacing all S3-specific code with the [django storage API](https://docs.djangoproject.com/en/1.8/ref/files/storage/).\r\n\r\n**Studio updates:** None yet, although video uploads will need to be modified. I am currently holding off on this as it will require changes to VEDA and I don't have access to that repo. Once Greg Martin has made the necessary changes to VEDA I will update Studio in a separate PR.\r\n\r\n**LMS updates:** Instructor CSV exports and student photo uploads modified to use the django storage API. Added an `openstack.py` settings file.\r\n\r\n**Related PRs:**\r\n\r\n- https://github.com/edx/configuration/pull/2723\r\n- https://github.com/edx/xqueue/pull/103\r\n- https://github.com/edx/edx-platform/pull/11374\r\n\r\n**Testing:** Deploy a sandbox using https://github.com/edx/configuration/pull/2723. Use [these ansible vars](https://gist.github.com/82623f60243d3ae94152), completed with your openstack credentials.\r\n\r\n**To test the instructor CSV reports:**\r\n\r\n- Go to the instructor dashboard for a course\r\n- Open the 'data download' tab\r\n- Click 'generate grade report'\r\n- An entry for the report should appear in the list below\r\n- Check that the report CSV has been uploaded to the swift container\r\n- Click the link to download the report directly from swift\r\n\r\n**To test student identity verification:**\r\n\r\n- In the django admin, create a new `CourseMode` for an existing course. Set the mode to `verified`, and make sure the price is > 0 (https://github.com/edx/edx-platform/pull/11374)\r\n- In the LMS course list, there should now be an 'upgrade' button if you are already enrolled in the course. Click it\r\n- You will be presented with a payment page. We need to get past this to get to the ID verification step. Run `./manage.py lms --settings=openstack shell`, then:\r\n\r\n```python\r\nfrom django.contrib.auth.models import User\r\nfrom opaque_keys.edx.keys import CourseKey\r\nfrom student.models import CourseEnrollment\r\n\r\nuser = User.objects.get(username='<your_username>')\r\ncourse_key = CourseKey.from_string('course-v1:edX+DemoX+Demo_Course')\r\nenrollment = CourseEnrollment.get_or_create_enrollment(user, course_key)\r\nenrollment.update_enrollment(mode='verified')\r\n```\r\n\r\nAlternatively, you can simply update the `CourseEnrollment` in the django admin:\r\n\r\n<img width=\"366\" alt=\"screen shot 2016-02-01 at 18 44 59\" src=\"https://cloud.githubusercontent.com/assets/793379/12718009/3cc7d1e2-c91e-11e5-8bb0-ce1f1fdffabb.png\">\r\n\r\nThis fools the LMS into thinking that the course has already been paid for. Go back to the payment page, and you should see the webcam verification step. Note that Chrome will not allow access to the webcam over plain http unless you are running on localhost. Firefox doesn't seem to mind.\r\n\r\n**To test xqueue:**\r\n\r\n- In studio, add a blank problem and set its xml to:\r\n```xml\r\n<problem display_name=\"XQueue Test\">\r\n   <text>\r\n       <p>Upload a program that prints \"hello world\".</p>\r\n   </text>\r\n   <coderesponse queuename=\"edX-Open_DemoX\">\r\n       <filesubmission/>\r\n       <codeparam>\r\n           <initial_display>\r\n             # students please write your program here\r\n             print \"\"\r\n           </initial_display>\r\n           <answer_display>\r\n             print \"hello world\"\r\n           </answer_display>\r\n           <grader_payload>\r\n             {\"output\": \"hello world\"}\r\n           </grader_payload>\r\n       </codeparam>\r\n   </coderesponse>\r\n</problem>\r\n```\r\n- Go to that problem **in LMS** (xqueue does not work in studio), select a file and click the 'Check' button\r\n- You should see this:\r\n<img width=\"812\" alt=\"screen shot 2016-02-03 at 01 41 57\" src=\"https://cloud.githubusercontent.com/assets/793379/12760245/09dec876-ca18-11e5-8ec8-85f57a93669a.png\">\r\n- The submission should now be in the configured swift bucket\r\n\r\n**To test log sync to swift:**\r\n\r\nSSH into the sandbox, and run\r\n\r\n    sudo service supervisor stop\r\n\r\nThis should trigger a log sync, after which the tracker logs can be found in a swift container.\r\n\r\n**Future tasks:** write some tests for django-swift-storage. It doesn't have any yet."},"comment":{"url":"https://api.github.com/repos/edx/edx-platform/issues/comments/190851017","html_url":"https://github.com/edx/edx-platform/pull/11286#issuecomment-190851017","issue_url":"https://api.github.com/repos/edx/edx-platform/issues/11286","id":190851017,"user":{"login":"omarkhan","id":793379,"avatar_url":"https://avatars.githubusercontent.com/u/793379?v=3","gravatar_id":"","url":"https://api.github.com/users/omarkhan","html_url":"https://github.com/omarkhan","followers_url":"https://api.github.com/users/omarkhan/followers","following_url":"https://api.github.com/users/omarkhan/following{/other_user}","gists_url":"https://api.github.com/users/omarkhan/gists{/gist_id}","starred_url":"https://api.github.com/users/omarkhan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/omarkhan/subscriptions","organizations_url":"https://api.github.com/users/omarkhan/orgs","repos_url":"https://api.github.com/users/omarkhan/repos","events_url":"https://api.github.com/users/omarkhan/events{/privacy}","received_events_url":"https://api.github.com/users/omarkhan/received_events","type":"User","site_admin":false},"created_at":"2016-03-01T18:54:45Z","updated_at":"2016-03-01T18:54:45Z","body":"jenkins run all"}}