{"action":"created","comment":{"url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/pulls/comments/202461491","pull_request_review_id":137175789,"id":202461491,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIwMjQ2MTQ5MQ==","diff_hunk":"@@ -0,0 +1,49 @@\n+\"\"\"\n+run_aggregator_service management command.\n+\n+Usage:\n+\n+    ./manage.py run_aggregator_service\n+\n+Performs the actual aggregation.\n+\n+For continuous aggregation, set a cron job to run this task periodically.\n+\"\"\"\n+\n+from __future__ import absolute_import, division, print_function, unicode_literals\n+\n+import logging\n+\n+from django.core.management.base import BaseCommand\n+\n+from ...aggregator import perform_aggregation\n+\n+\n+class Command(BaseCommand):\n+    \"\"\"\n+    run_aggregator_service management command.\n+    \"\"\"\n+\n+    def handle(self, *args, **options):\n+        \"\"\"\n+        Run the aggregator service.\n+        \"\"\"\n+        self.set_logging(options['verbosity'])\n+        perform_aggregation()","path":"completion_aggregator/management/commands/run_aggregator_service.py","position":32,"original_position":32,"commit_id":"855cf45587d5cbe8b5bb6445ffdfa36795f7a453","original_commit_id":"855cf45587d5cbe8b5bb6445ffdfa36795f7a453","user":{"login":"jcdyer","id":77486,"node_id":"MDQ6VXNlcjc3NDg2","avatar_url":"https://avatars0.githubusercontent.com/u/77486?v=4","gravatar_id":"","url":"https://api.github.com/users/jcdyer","html_url":"https://github.com/jcdyer","followers_url":"https://api.github.com/users/jcdyer/followers","following_url":"https://api.github.com/users/jcdyer/following{/other_user}","gists_url":"https://api.github.com/users/jcdyer/gists{/gist_id}","starred_url":"https://api.github.com/users/jcdyer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jcdyer/subscriptions","organizations_url":"https://api.github.com/users/jcdyer/orgs","repos_url":"https://api.github.com/users/jcdyer/repos","events_url":"https://api.github.com/users/jcdyer/events{/privacy}","received_events_url":"https://api.github.com/users/jcdyer/received_events","type":"User","site_admin":false},"body":"It was just as easy to create a cleanup task.","created_at":"2018-07-13T20:16:37Z","updated_at":"2018-07-13T20:16:37Z","html_url":"https://github.com/open-craft/openedx-completion-aggregator/pull/10#discussion_r202461491","pull_request_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/pulls/10","author_association":"MEMBER","_links":{"self":{"href":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/pulls/comments/202461491"},"html":{"href":"https://github.com/open-craft/openedx-completion-aggregator/pull/10#discussion_r202461491"},"pull_request":{"href":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/pulls/10"}},"in_reply_to_id":201596193},"pull_request":{"url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/pulls/10","id":185265929,"node_id":"MDExOlB1bGxSZXF1ZXN0MTg1MjY1OTI5","html_url":"https://github.com/open-craft/openedx-completion-aggregator/pull/10","diff_url":"https://github.com/open-craft/openedx-completion-aggregator/pull/10.diff","patch_url":"https://github.com/open-craft/openedx-completion-aggregator/pull/10.patch","issue_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/issues/10","number":10,"state":"open","locked":false,"title":"Add service to periodically aggregate out of date completions.","user":{"login":"jcdyer","id":77486,"node_id":"MDQ6VXNlcjc3NDg2","avatar_url":"https://avatars0.githubusercontent.com/u/77486?v=4","gravatar_id":"","url":"https://api.github.com/users/jcdyer","html_url":"https://github.com/jcdyer","followers_url":"https://api.github.com/users/jcdyer/followers","following_url":"https://api.github.com/users/jcdyer/following{/other_user}","gists_url":"https://api.github.com/users/jcdyer/gists{/gist_id}","starred_url":"https://api.github.com/users/jcdyer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jcdyer/subscriptions","organizations_url":"https://api.github.com/users/jcdyer/orgs","repos_url":"https://api.github.com/users/jcdyer/repos","events_url":"https://api.github.com/users/jcdyer/events{/privacy}","received_events_url":"https://api.github.com/users/jcdyer/received_events","type":"User","site_admin":false},"body":"**Description:** \r\n\r\nThis provides a service that runs out-of-band and updates out-of-date completion aggregators.\r\nWhen a completion would previously have called the aggregator task directly, it now marks its BlockCompletions out-of-date in a StaleCompletion table.  When the service runs, it gets all the stale blocks from the table, runs one completion task for each enrollment that is out of date, and marks all its stale completions resolved.  Later, resolved completions are deleted from the table, but we don't do this immediately to reduce table churn, and to provide a little visibility into recent table activity.\r\n\r\n\r\n**JIRA:** OC-3968\r\n\r\n**Dependencies:** None\r\n\r\n**Merge deadline:** None\r\n\r\n**Installation instructions:** None.\r\n\r\n**Testing instructions:**\r\n1. Run the management command.  ./manage.py run_aggregator_service -v2\r\n2. Create some BlockCompletions, either by browsing a course, or from a ./manage.py shell\r\n3. Verify (from the logging or from the command line) that StaleCompletions are being created and resolved.  \r\n4. If running from edx-platform, verify that running a celery task worker causes the aggregators to actually be updated.  If running from the openedx-completion-aggregator repo, verify that running a celery task worker causes tasks to be taken off the queue, but fail, because the modulestore can't be accessed.\r\n\r\n**Reviewers:**\r\n- [ ] @pomegranited  \r\n\r\n**Merge checklist:**\r\n- [ ] All reviewers approved\r\n- [ ] CI build is green\r\n- [ ] Version bumped\r\n- [ ] Changelog record added\r\n- [ ] Documentation updated (not only docstrings)\r\n- [ ] Commits are squashed\r\n- [ ] PR author is listed in AUTHORS\r\n\r\n**Post merge:**\r\n- [ ] Delete working branch (if not needed anymore)\r\n\r\n**Author concerns:** We still need to make sure this service gets run from a cronjob when deployed to \r\nMcKinsey.  I'm not sure how we do that.  \r\n","created_at":"2018-05-01T20:20:12Z","updated_at":"2018-07-13T20:16:37Z","closed_at":null,"merged_at":null,"merge_commit_sha":"10521b6f1f22c84fc536401225fec8437806d20a","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/pulls/10/commits","review_comments_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/pulls/10/comments","review_comment_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/pulls/comments{/number}","comments_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/issues/10/comments","statuses_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/statuses/855cf45587d5cbe8b5bb6445ffdfa36795f7a453","head":{"label":"open-craft:periodic-update","ref":"periodic-update","sha":"855cf45587d5cbe8b5bb6445ffdfa36795f7a453","user":{"login":"open-craft","id":7414786,"node_id":"MDEyOk9yZ2FuaXphdGlvbjc0MTQ3ODY=","avatar_url":"https://avatars0.githubusercontent.com/u/7414786?v=4","gravatar_id":"","url":"https://api.github.com/users/open-craft","html_url":"https://github.com/open-craft","followers_url":"https://api.github.com/users/open-craft/followers","following_url":"https://api.github.com/users/open-craft/following{/other_user}","gists_url":"https://api.github.com/users/open-craft/gists{/gist_id}","starred_url":"https://api.github.com/users/open-craft/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/open-craft/subscriptions","organizations_url":"https://api.github.com/users/open-craft/orgs","repos_url":"https://api.github.com/users/open-craft/repos","events_url":"https://api.github.com/users/open-craft/events{/privacy}","received_events_url":"https://api.github.com/users/open-craft/received_events","type":"Organization","site_admin":false},"repo":{"id":116283379,"node_id":"MDEwOlJlcG9zaXRvcnkxMTYyODMzNzk=","name":"openedx-completion-aggregator","full_name":"open-craft/openedx-completion-aggregator","owner":{"login":"open-craft","id":7414786,"node_id":"MDEyOk9yZ2FuaXphdGlvbjc0MTQ3ODY=","avatar_url":"https://avatars0.githubusercontent.com/u/7414786?v=4","gravatar_id":"","url":"https://api.github.com/users/open-craft","html_url":"https://github.com/open-craft","followers_url":"https://api.github.com/users/open-craft/followers","following_url":"https://api.github.com/users/open-craft/following{/other_user}","gists_url":"https://api.github.com/users/open-craft/gists{/gist_id}","starred_url":"https://api.github.com/users/open-craft/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/open-craft/subscriptions","organizations_url":"https://api.github.com/users/open-craft/orgs","repos_url":"https://api.github.com/users/open-craft/repos","events_url":"https://api.github.com/users/open-craft/events{/privacy}","received_events_url":"https://api.github.com/users/open-craft/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/open-craft/openedx-completion-aggregator","description":"An Open edX Django app that aggregates block-level completion data to report on different block types.","fork":false,"url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator","forks_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/forks","keys_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/keys{/key_id}","collaborators_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/teams","hooks_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/hooks","issue_events_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/issues/events{/number}","events_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/events","assignees_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/assignees{/user}","branches_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/branches{/branch}","tags_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/tags","blobs_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/git/refs{/sha}","trees_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/git/trees{/sha}","statuses_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/statuses/{sha}","languages_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/languages","stargazers_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/stargazers","contributors_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/contributors","subscribers_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/subscribers","subscription_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/subscription","commits_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/commits{/sha}","git_commits_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/git/commits{/sha}","comments_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/comments{/number}","issue_comment_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/issues/comments{/number}","contents_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/contents/{+path}","compare_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/compare/{base}...{head}","merges_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/merges","archive_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/downloads","issues_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/issues{/number}","pulls_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/pulls{/number}","milestones_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/milestones{/number}","notifications_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/labels{/name}","releases_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/releases{/id}","deployments_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/deployments","created_at":"2018-01-04T16:38:20Z","updated_at":"2018-07-13T10:48:07Z","pushed_at":"2018-07-13T20:03:21Z","git_url":"git://github.com/open-craft/openedx-completion-aggregator.git","ssh_url":"git@github.com:open-craft/openedx-completion-aggregator.git","clone_url":"https://github.com/open-craft/openedx-completion-aggregator.git","svn_url":"https://github.com/open-craft/openedx-completion-aggregator","homepage":null,"size":186,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":4,"license":{"key":"agpl-3.0","name":"GNU Affero General Public License v3.0","spdx_id":"AGPL-3.0","url":"https://api.github.com/licenses/agpl-3.0","node_id":"MDc6TGljZW5zZTE="},"forks":0,"open_issues":4,"watchers":0,"default_branch":"master"}},"base":{"label":"open-craft:master","ref":"master","sha":"d3f835b30b6b69c08b1415e0ab69bd77d18bc267","user":{"login":"open-craft","id":7414786,"node_id":"MDEyOk9yZ2FuaXphdGlvbjc0MTQ3ODY=","avatar_url":"https://avatars0.githubusercontent.com/u/7414786?v=4","gravatar_id":"","url":"https://api.github.com/users/open-craft","html_url":"https://github.com/open-craft","followers_url":"https://api.github.com/users/open-craft/followers","following_url":"https://api.github.com/users/open-craft/following{/other_user}","gists_url":"https://api.github.com/users/open-craft/gists{/gist_id}","starred_url":"https://api.github.com/users/open-craft/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/open-craft/subscriptions","organizations_url":"https://api.github.com/users/open-craft/orgs","repos_url":"https://api.github.com/users/open-craft/repos","events_url":"https://api.github.com/users/open-craft/events{/privacy}","received_events_url":"https://api.github.com/users/open-craft/received_events","type":"Organization","site_admin":false},"repo":{"id":116283379,"node_id":"MDEwOlJlcG9zaXRvcnkxMTYyODMzNzk=","name":"openedx-completion-aggregator","full_name":"open-craft/openedx-completion-aggregator","owner":{"login":"open-craft","id":7414786,"node_id":"MDEyOk9yZ2FuaXphdGlvbjc0MTQ3ODY=","avatar_url":"https://avatars0.githubusercontent.com/u/7414786?v=4","gravatar_id":"","url":"https://api.github.com/users/open-craft","html_url":"https://github.com/open-craft","followers_url":"https://api.github.com/users/open-craft/followers","following_url":"https://api.github.com/users/open-craft/following{/other_user}","gists_url":"https://api.github.com/users/open-craft/gists{/gist_id}","starred_url":"https://api.github.com/users/open-craft/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/open-craft/subscriptions","organizations_url":"https://api.github.com/users/open-craft/orgs","repos_url":"https://api.github.com/users/open-craft/repos","events_url":"https://api.github.com/users/open-craft/events{/privacy}","received_events_url":"https://api.github.com/users/open-craft/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/open-craft/openedx-completion-aggregator","description":"An Open edX Django app that aggregates block-level completion data to report on different block types.","fork":false,"url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator","forks_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/forks","keys_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/keys{/key_id}","collaborators_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/teams","hooks_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/hooks","issue_events_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/issues/events{/number}","events_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/events","assignees_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/assignees{/user}","branches_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/branches{/branch}","tags_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/tags","blobs_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/git/refs{/sha}","trees_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/git/trees{/sha}","statuses_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/statuses/{sha}","languages_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/languages","stargazers_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/stargazers","contributors_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/contributors","subscribers_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/subscribers","subscription_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/subscription","commits_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/commits{/sha}","git_commits_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/git/commits{/sha}","comments_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/comments{/number}","issue_comment_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/issues/comments{/number}","contents_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/contents/{+path}","compare_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/compare/{base}...{head}","merges_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/merges","archive_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/downloads","issues_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/issues{/number}","pulls_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/pulls{/number}","milestones_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/milestones{/number}","notifications_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/labels{/name}","releases_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/releases{/id}","deployments_url":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/deployments","created_at":"2018-01-04T16:38:20Z","updated_at":"2018-07-13T10:48:07Z","pushed_at":"2018-07-13T20:03:21Z","git_url":"git://github.com/open-craft/openedx-completion-aggregator.git","ssh_url":"git@github.com:open-craft/openedx-completion-aggregator.git","clone_url":"https://github.com/open-craft/openedx-completion-aggregator.git","svn_url":"https://github.com/open-craft/openedx-completion-aggregator","homepage":null,"size":186,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":4,"license":{"key":"agpl-3.0","name":"GNU Affero General Public License v3.0","spdx_id":"AGPL-3.0","url":"https://api.github.com/licenses/agpl-3.0","node_id":"MDc6TGljZW5zZTE="},"forks":0,"open_issues":4,"watchers":0,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/pulls/10"},"html":{"href":"https://github.com/open-craft/openedx-completion-aggregator/pull/10"},"issue":{"href":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/issues/10"},"comments":{"href":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/issues/10/comments"},"review_comments":{"href":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/pulls/10/comments"},"review_comment":{"href":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/pulls/10/commits"},"statuses":{"href":"https://api.github.com/repos/open-craft/openedx-completion-aggregator/statuses/855cf45587d5cbe8b5bb6445ffdfa36795f7a453"}},"author_association":"MEMBER"}}
{"actor":{"display_login":"jcdyer"}}