{"action":"opened","number":3189,"pull_request":{"url":"https://api.github.com/repos/hypothesis/h/pulls/3189","id":65771023,"html_url":"https://github.com/hypothesis/h/pull/3189","diff_url":"https://github.com/hypothesis/h/pull/3189.diff","patch_url":"https://github.com/hypothesis/h/pull/3189.patch","issue_url":"https://api.github.com/repos/hypothesis/h/issues/3189","number":3189,"state":"open","locked":false,"title":"Replace homebrew background workers with Celery","user":{"login":"nickstenning","id":3602,"avatar_url":"https://avatars.githubusercontent.com/u/3602?v=3","gravatar_id":"","url":"https://api.github.com/users/nickstenning","html_url":"https://github.com/nickstenning","followers_url":"https://api.github.com/users/nickstenning/followers","following_url":"https://api.github.com/users/nickstenning/following{/other_user}","gists_url":"https://api.github.com/users/nickstenning/gists{/gist_id}","starred_url":"https://api.github.com/users/nickstenning/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nickstenning/subscriptions","organizations_url":"https://api.github.com/users/nickstenning/orgs","repos_url":"https://api.github.com/users/nickstenning/repos","events_url":"https://api.github.com/users/nickstenning/events{/privacy}","received_events_url":"https://api.github.com/users/nickstenning/received_events","type":"User","site_admin":false},"body":"Until now, we've used `h.worker`, a home-grown worker runner which reads tasks from NSQ. It has served its purpose, but is increasingly inflexible, as it currently assumes a process-per-job-type worker model which makes deploying new job types tricky, and consumes more resources as we add more job types, even if those jobs are infrequently used.\r\n\r\nRather than continuing to develop `h.worker`, it makes sense to switch to the well-known and actively developed Celery project to run background tasks. This allows us to configure tasks in a straightforward fashion (basically just functions) and dispatch to them easily from anywhere in the code.\r\n\r\nIn this pull request, Celery is configured to use Redis as a message broker. We already use Redis for session storage in production, and this PR configures Celery to use a different Redis \"database\" so that keys do not clash with those from `pyramid_redis_sessions`.\r\n\r\nIt's worth noting that Redis is *not* a message queue (it does not provide reliable persistence) but at the moment we only use background jobs for two things:\r\n\r\n1. Sending emails.\r\n2. Processing \"NIPSA\" jobs where we have to find all of a user's annotations and flag/unflag them.\r\n\r\nFor these tasks, it seemed (to me) to be reasonable to reuse Redis as a message broker rather than requiring that we set up RabbitMQ (the normal message broker used with Celery) in production. I think we should have a conversation about whether this is the right choice at this stage, because we have a number of options:\r\n\r\n1. Mandate (or at least document) the use of RabbitMQ in dev and prod. (Maximising dev/prod parity.)\r\n2. Use redis in dev (for now) and deploy RabbitMQ in production. (Maximising ease of setup in dev, but ensuring persistent messages in prod.)\r\n3. Use redis in both (for now). (Simplest deployment.)\r\n\r\n---\r\n\r\nHere's a quick overview in a little more detail of the changes in this branch:\r\n\r\n- `hypothesis-worker` is gone and replaced with `hypothesis-celery` which is a simple wrapper around the default Celery CLI.\r\n- `h.mailer` and `h.nipsa.worker` are rewritten to define Celery tasks which can be dispatched asynchronously from anywhere in the application.\r\n- `h.notification.worker` has been removed: responsibility for generating relevant notification emails has moved into the web worker process, and those emails are sent asynchronously by the `h.mailer` Celery task.\r\n- When a Celery worker boots, a fake request is created and attached to the Celery application, much as `h.worker` did.\r\n- Errors handling celery tasks are reported to Sentry.\r\n- Feature flag state is flushed before handling each task.\r\n- Task success results in the request transaction being committed, and task failure results in the request transaction being aborted.","created_at":"2016-04-08T11:12:02Z","updated_at":"2016-04-08T11:12:02Z","closed_at":null,"merged_at":null,"merge_commit_sha":null,"assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/hypothesis/h/pulls/3189/commits","review_comments_url":"https://api.github.com/repos/hypothesis/h/pulls/3189/comments","review_comment_url":"https://api.github.com/repos/hypothesis/h/pulls/comments{/number}","comments_url":"https://api.github.com/repos/hypothesis/h/issues/3189/comments","statuses_url":"https://api.github.com/repos/hypothesis/h/statuses/368c62fb24311af9541a06ed484b51e289aa2a94","head":{"label":"hypothesis:celery","ref":"celery","sha":"368c62fb24311af9541a06ed484b51e289aa2a94","user":{"login":"hypothesis","id":1243215,"avatar_url":"https://avatars.githubusercontent.com/u/1243215?v=3","gravatar_id":"","url":"https://api.github.com/users/hypothesis","html_url":"https://github.com/hypothesis","followers_url":"https://api.github.com/users/hypothesis/followers","following_url":"https://api.github.com/users/hypothesis/following{/other_user}","gists_url":"https://api.github.com/users/hypothesis/gists{/gist_id}","starred_url":"https://api.github.com/users/hypothesis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hypothesis/subscriptions","organizations_url":"https://api.github.com/users/hypothesis/orgs","repos_url":"https://api.github.com/users/hypothesis/repos","events_url":"https://api.github.com/users/hypothesis/events{/privacy}","received_events_url":"https://api.github.com/users/hypothesis/received_events","type":"Organization","site_admin":false},"repo":{"id":3910945,"name":"h","full_name":"hypothesis/h","owner":{"login":"hypothesis","id":1243215,"avatar_url":"https://avatars.githubusercontent.com/u/1243215?v=3","gravatar_id":"","url":"https://api.github.com/users/hypothesis","html_url":"https://github.com/hypothesis","followers_url":"https://api.github.com/users/hypothesis/followers","following_url":"https://api.github.com/users/hypothesis/following{/other_user}","gists_url":"https://api.github.com/users/hypothesis/gists{/gist_id}","starred_url":"https://api.github.com/users/hypothesis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hypothesis/subscriptions","organizations_url":"https://api.github.com/users/hypothesis/orgs","repos_url":"https://api.github.com/users/hypothesis/repos","events_url":"https://api.github.com/users/hypothesis/events{/privacy}","received_events_url":"https://api.github.com/users/hypothesis/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/hypothesis/h","description":"The Internet, peer reviewed.","fork":false,"url":"https://api.github.com/repos/hypothesis/h","forks_url":"https://api.github.com/repos/hypothesis/h/forks","keys_url":"https://api.github.com/repos/hypothesis/h/keys{/key_id}","collaborators_url":"https://api.github.com/repos/hypothesis/h/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/hypothesis/h/teams","hooks_url":"https://api.github.com/repos/hypothesis/h/hooks","issue_events_url":"https://api.github.com/repos/hypothesis/h/issues/events{/number}","events_url":"https://api.github.com/repos/hypothesis/h/events","assignees_url":"https://api.github.com/repos/hypothesis/h/assignees{/user}","branches_url":"https://api.github.com/repos/hypothesis/h/branches{/branch}","tags_url":"https://api.github.com/repos/hypothesis/h/tags","blobs_url":"https://api.github.com/repos/hypothesis/h/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/hypothesis/h/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/hypothesis/h/git/refs{/sha}","trees_url":"https://api.github.com/repos/hypothesis/h/git/trees{/sha}","statuses_url":"https://api.github.com/repos/hypothesis/h/statuses/{sha}","languages_url":"https://api.github.com/repos/hypothesis/h/languages","stargazers_url":"https://api.github.com/repos/hypothesis/h/stargazers","contributors_url":"https://api.github.com/repos/hypothesis/h/contributors","subscribers_url":"https://api.github.com/repos/hypothesis/h/subscribers","subscription_url":"https://api.github.com/repos/hypothesis/h/subscription","commits_url":"https://api.github.com/repos/hypothesis/h/commits{/sha}","git_commits_url":"https://api.github.com/repos/hypothesis/h/git/commits{/sha}","comments_url":"https://api.github.com/repos/hypothesis/h/comments{/number}","issue_comment_url":"https://api.github.com/repos/hypothesis/h/issues/comments{/number}","contents_url":"https://api.github.com/repos/hypothesis/h/contents/{+path}","compare_url":"https://api.github.com/repos/hypothesis/h/compare/{base}...{head}","merges_url":"https://api.github.com/repos/hypothesis/h/merges","archive_url":"https://api.github.com/repos/hypothesis/h/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/hypothesis/h/downloads","issues_url":"https://api.github.com/repos/hypothesis/h/issues{/number}","pulls_url":"https://api.github.com/repos/hypothesis/h/pulls{/number}","milestones_url":"https://api.github.com/repos/hypothesis/h/milestones{/number}","notifications_url":"https://api.github.com/repos/hypothesis/h/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/hypothesis/h/labels{/name}","releases_url":"https://api.github.com/repos/hypothesis/h/releases{/id}","deployments_url":"https://api.github.com/repos/hypothesis/h/deployments","created_at":"2012-04-02T19:56:59Z","updated_at":"2016-04-08T10:54:03Z","pushed_at":"2016-04-08T11:12:02Z","git_url":"git://github.com/hypothesis/h.git","ssh_url":"git@github.com:hypothesis/h.git","clone_url":"https://github.com/hypothesis/h.git","svn_url":"https://github.com/hypothesis/h","homepage":"https://hypothes.is/","size":29104,"stargazers_count":611,"watchers_count":611,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":145,"mirror_url":null,"open_issues_count":76,"forks":145,"open_issues":76,"watchers":611,"default_branch":"master"}},"base":{"label":"hypothesis:master","ref":"master","sha":"862c705d029bcd3b6c02ea50d8fa9c546eefa5f3","user":{"login":"hypothesis","id":1243215,"avatar_url":"https://avatars.githubusercontent.com/u/1243215?v=3","gravatar_id":"","url":"https://api.github.com/users/hypothesis","html_url":"https://github.com/hypothesis","followers_url":"https://api.github.com/users/hypothesis/followers","following_url":"https://api.github.com/users/hypothesis/following{/other_user}","gists_url":"https://api.github.com/users/hypothesis/gists{/gist_id}","starred_url":"https://api.github.com/users/hypothesis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hypothesis/subscriptions","organizations_url":"https://api.github.com/users/hypothesis/orgs","repos_url":"https://api.github.com/users/hypothesis/repos","events_url":"https://api.github.com/users/hypothesis/events{/privacy}","received_events_url":"https://api.github.com/users/hypothesis/received_events","type":"Organization","site_admin":false},"repo":{"id":3910945,"name":"h","full_name":"hypothesis/h","owner":{"login":"hypothesis","id":1243215,"avatar_url":"https://avatars.githubusercontent.com/u/1243215?v=3","gravatar_id":"","url":"https://api.github.com/users/hypothesis","html_url":"https://github.com/hypothesis","followers_url":"https://api.github.com/users/hypothesis/followers","following_url":"https://api.github.com/users/hypothesis/following{/other_user}","gists_url":"https://api.github.com/users/hypothesis/gists{/gist_id}","starred_url":"https://api.github.com/users/hypothesis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hypothesis/subscriptions","organizations_url":"https://api.github.com/users/hypothesis/orgs","repos_url":"https://api.github.com/users/hypothesis/repos","events_url":"https://api.github.com/users/hypothesis/events{/privacy}","received_events_url":"https://api.github.com/users/hypothesis/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/hypothesis/h","description":"The Internet, peer reviewed.","fork":false,"url":"https://api.github.com/repos/hypothesis/h","forks_url":"https://api.github.com/repos/hypothesis/h/forks","keys_url":"https://api.github.com/repos/hypothesis/h/keys{/key_id}","collaborators_url":"https://api.github.com/repos/hypothesis/h/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/hypothesis/h/teams","hooks_url":"https://api.github.com/repos/hypothesis/h/hooks","issue_events_url":"https://api.github.com/repos/hypothesis/h/issues/events{/number}","events_url":"https://api.github.com/repos/hypothesis/h/events","assignees_url":"https://api.github.com/repos/hypothesis/h/assignees{/user}","branches_url":"https://api.github.com/repos/hypothesis/h/branches{/branch}","tags_url":"https://api.github.com/repos/hypothesis/h/tags","blobs_url":"https://api.github.com/repos/hypothesis/h/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/hypothesis/h/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/hypothesis/h/git/refs{/sha}","trees_url":"https://api.github.com/repos/hypothesis/h/git/trees{/sha}","statuses_url":"https://api.github.com/repos/hypothesis/h/statuses/{sha}","languages_url":"https://api.github.com/repos/hypothesis/h/languages","stargazers_url":"https://api.github.com/repos/hypothesis/h/stargazers","contributors_url":"https://api.github.com/repos/hypothesis/h/contributors","subscribers_url":"https://api.github.com/repos/hypothesis/h/subscribers","subscription_url":"https://api.github.com/repos/hypothesis/h/subscription","commits_url":"https://api.github.com/repos/hypothesis/h/commits{/sha}","git_commits_url":"https://api.github.com/repos/hypothesis/h/git/commits{/sha}","comments_url":"https://api.github.com/repos/hypothesis/h/comments{/number}","issue_comment_url":"https://api.github.com/repos/hypothesis/h/issues/comments{/number}","contents_url":"https://api.github.com/repos/hypothesis/h/contents/{+path}","compare_url":"https://api.github.com/repos/hypothesis/h/compare/{base}...{head}","merges_url":"https://api.github.com/repos/hypothesis/h/merges","archive_url":"https://api.github.com/repos/hypothesis/h/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/hypothesis/h/downloads","issues_url":"https://api.github.com/repos/hypothesis/h/issues{/number}","pulls_url":"https://api.github.com/repos/hypothesis/h/pulls{/number}","milestones_url":"https://api.github.com/repos/hypothesis/h/milestones{/number}","notifications_url":"https://api.github.com/repos/hypothesis/h/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/hypothesis/h/labels{/name}","releases_url":"https://api.github.com/repos/hypothesis/h/releases{/id}","deployments_url":"https://api.github.com/repos/hypothesis/h/deployments","created_at":"2012-04-02T19:56:59Z","updated_at":"2016-04-08T10:54:03Z","pushed_at":"2016-04-08T11:12:02Z","git_url":"git://github.com/hypothesis/h.git","ssh_url":"git@github.com:hypothesis/h.git","clone_url":"https://github.com/hypothesis/h.git","svn_url":"https://github.com/hypothesis/h","homepage":"https://hypothes.is/","size":29104,"stargazers_count":611,"watchers_count":611,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":145,"mirror_url":null,"open_issues_count":76,"forks":145,"open_issues":76,"watchers":611,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/hypothesis/h/pulls/3189"},"html":{"href":"https://github.com/hypothesis/h/pull/3189"},"issue":{"href":"https://api.github.com/repos/hypothesis/h/issues/3189"},"comments":{"href":"https://api.github.com/repos/hypothesis/h/issues/3189/comments"},"review_comments":{"href":"https://api.github.com/repos/hypothesis/h/pulls/3189/comments"},"review_comment":{"href":"https://api.github.com/repos/hypothesis/h/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/hypothesis/h/pulls/3189/commits"},"statuses":{"href":"https://api.github.com/repos/hypothesis/h/statuses/368c62fb24311af9541a06ed484b51e289aa2a94"}},"merged":false,"mergeable":null,"mergeable_state":"unknown","merged_by":null,"comments":0,"review_comments":0,"commits":9,"additions":514,"deletions":361,"changed_files":18}}