{"action":"closed","number":10,"pull_request":{"url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/pulls/10","id":59969445,"html_url":"https://github.com/zooniverse/generate-subjects-from-planet-api/pull/10","diff_url":"https://github.com/zooniverse/generate-subjects-from-planet-api/pull/10.diff","patch_url":"https://github.com/zooniverse/generate-subjects-from-planet-api/pull/10.patch","issue_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/issues/10","number":10,"state":"closed","locked":false,"title":"- Add message queue via RSMQ","user":{"login":"itsravenous","id":1737376,"avatar_url":"https://avatars.githubusercontent.com/u/1737376?v=3","gravatar_id":"","url":"https://api.github.com/users/itsravenous","html_url":"https://github.com/itsravenous","followers_url":"https://api.github.com/users/itsravenous/followers","following_url":"https://api.github.com/users/itsravenous/following{/other_user}","gists_url":"https://api.github.com/users/itsravenous/gists{/gist_id}","starred_url":"https://api.github.com/users/itsravenous/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/itsravenous/subscriptions","organizations_url":"https://api.github.com/users/itsravenous/orgs","repos_url":"https://api.github.com/users/itsravenous/repos","events_url":"https://api.github.com/users/itsravenous/events{/privacy}","received_events_url":"https://api.github.com/users/itsravenous/received_events","type":"User","site_admin":false},"body":"- Separate job producer and consumer\r\n- Remove mosaic cache env var, add skip cache option on downloadFile method (currently not exposed)\r\n- Trim old test scripts from readme (we should implement tests for the various moving parts instead)\r\n\r\nThe advantage of using a message queue (in this case over Redis) with a separate producer and consumer (instead of having the uploader app directly spawn child workers) is that the job queue will be persisted even if the uploader crashed, and can easily be resumed later. Redis also offers various backup functionality which is nice to have.\r\n\r\nThis is moving towards the uploader app becoming the \"dashboard\" for the PRN process - once we've got sessions with oauth sorted, we can let users (or at least admins) monitor their running jobs from the interface (I realise this might have to be a later addition).\r\n\r\nThe separation of concerns should also help us with keeping the code tight.\r\n\r\nI've also trimmed the README to just show the running instructions for the main app. We should add some tests for the various moving parts. \r\n\r\nOh, and I got rid of `USE_MOSAIC_CACHE` as an env var and added `skip_cache` as [an option](https://github.com/zooniverse/generate-subjects-from-planet-api/blob/message_queue/modules/planet-api.js#L94-L97) on `downloadFile`. It's not settable via the command line yet, because I don't think we need it at the moment (and we don't currently have any other runtime options, so didn't want to add unnecessary parsing with yargs). If we decide we want to be able to disable the cache (because, e.g. the mosaics change over time) then we can always add it as a runtime options.\r\n\r\nNote: the message queue I chose is [rsmq](https://github.com/smrchy/rsmq) but we could easily swap it out for anything else that runs on Redis (or we could ditch Redis altogether). ","created_at":"2016-02-19T16:09:08Z","updated_at":"2016-02-19T18:11:36Z","closed_at":"2016-02-19T18:11:36Z","merged_at":"2016-02-19T18:11:36Z","merge_commit_sha":"e30731335d400936d5aabaa1ba4fac780790d498","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/pulls/10/commits","review_comments_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/pulls/10/comments","review_comment_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/pulls/comments{/number}","comments_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/issues/10/comments","statuses_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/statuses/008e4745f3c74388b8c4ab629cf2f9fdf4c22499","head":{"label":"zooniverse:message_queue","ref":"message_queue","sha":"008e4745f3c74388b8c4ab629cf2f9fdf4c22499","user":{"login":"zooniverse","id":106856,"avatar_url":"https://avatars.githubusercontent.com/u/106856?v=3","gravatar_id":"","url":"https://api.github.com/users/zooniverse","html_url":"https://github.com/zooniverse","followers_url":"https://api.github.com/users/zooniverse/followers","following_url":"https://api.github.com/users/zooniverse/following{/other_user}","gists_url":"https://api.github.com/users/zooniverse/gists{/gist_id}","starred_url":"https://api.github.com/users/zooniverse/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zooniverse/subscriptions","organizations_url":"https://api.github.com/users/zooniverse/orgs","repos_url":"https://api.github.com/users/zooniverse/repos","events_url":"https://api.github.com/users/zooniverse/events{/privacy}","received_events_url":"https://api.github.com/users/zooniverse/received_events","type":"Organization","site_admin":false},"repo":{"id":49604871,"name":"generate-subjects-from-planet-api","full_name":"zooniverse/generate-subjects-from-planet-api","owner":{"login":"zooniverse","id":106856,"avatar_url":"https://avatars.githubusercontent.com/u/106856?v=3","gravatar_id":"","url":"https://api.github.com/users/zooniverse","html_url":"https://github.com/zooniverse","followers_url":"https://api.github.com/users/zooniverse/followers","following_url":"https://api.github.com/users/zooniverse/following{/other_user}","gists_url":"https://api.github.com/users/zooniverse/gists{/gist_id}","starred_url":"https://api.github.com/users/zooniverse/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zooniverse/subscriptions","organizations_url":"https://api.github.com/users/zooniverse/orgs","repos_url":"https://api.github.com/users/zooniverse/repos","events_url":"https://api.github.com/users/zooniverse/events{/privacy}","received_events_url":"https://api.github.com/users/zooniverse/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/zooniverse/generate-subjects-from-planet-api","description":"Create Zooniverse subjects via Planet Labs API using area of interest (AOI) queries.","fork":false,"url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api","forks_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/forks","keys_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/keys{/key_id}","collaborators_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/teams","hooks_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/hooks","issue_events_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/issues/events{/number}","events_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/events","assignees_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/assignees{/user}","branches_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/branches{/branch}","tags_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/tags","blobs_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/git/refs{/sha}","trees_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/git/trees{/sha}","statuses_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/statuses/{sha}","languages_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/languages","stargazers_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/stargazers","contributors_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/contributors","subscribers_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/subscribers","subscription_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/subscription","commits_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/commits{/sha}","git_commits_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/git/commits{/sha}","comments_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/comments{/number}","issue_comment_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/issues/comments{/number}","contents_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/contents/{+path}","compare_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/compare/{base}...{head}","merges_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/merges","archive_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/downloads","issues_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/issues{/number}","pulls_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/pulls{/number}","milestones_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/milestones{/number}","notifications_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/labels{/name}","releases_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/releases{/id}","deployments_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/deployments","created_at":"2016-01-13T22:00:00Z","updated_at":"2016-01-28T21:56:10Z","pushed_at":"2016-02-19T18:11:36Z","git_url":"git://github.com/zooniverse/generate-subjects-from-planet-api.git","ssh_url":"git@github.com:zooniverse/generate-subjects-from-planet-api.git","clone_url":"https://github.com/zooniverse/generate-subjects-from-planet-api.git","svn_url":"https://github.com/zooniverse/generate-subjects-from-planet-api","homepage":null,"size":74319,"stargazers_count":0,"watchers_count":0,"language":"JavaScript","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":1,"forks":1,"open_issues":1,"watchers":0,"default_branch":"master"}},"base":{"label":"zooniverse:master","ref":"master","sha":"91c0aa14cea4121993c90e5b0095c40034d2f517","user":{"login":"zooniverse","id":106856,"avatar_url":"https://avatars.githubusercontent.com/u/106856?v=3","gravatar_id":"","url":"https://api.github.com/users/zooniverse","html_url":"https://github.com/zooniverse","followers_url":"https://api.github.com/users/zooniverse/followers","following_url":"https://api.github.com/users/zooniverse/following{/other_user}","gists_url":"https://api.github.com/users/zooniverse/gists{/gist_id}","starred_url":"https://api.github.com/users/zooniverse/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zooniverse/subscriptions","organizations_url":"https://api.github.com/users/zooniverse/orgs","repos_url":"https://api.github.com/users/zooniverse/repos","events_url":"https://api.github.com/users/zooniverse/events{/privacy}","received_events_url":"https://api.github.com/users/zooniverse/received_events","type":"Organization","site_admin":false},"repo":{"id":49604871,"name":"generate-subjects-from-planet-api","full_name":"zooniverse/generate-subjects-from-planet-api","owner":{"login":"zooniverse","id":106856,"avatar_url":"https://avatars.githubusercontent.com/u/106856?v=3","gravatar_id":"","url":"https://api.github.com/users/zooniverse","html_url":"https://github.com/zooniverse","followers_url":"https://api.github.com/users/zooniverse/followers","following_url":"https://api.github.com/users/zooniverse/following{/other_user}","gists_url":"https://api.github.com/users/zooniverse/gists{/gist_id}","starred_url":"https://api.github.com/users/zooniverse/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zooniverse/subscriptions","organizations_url":"https://api.github.com/users/zooniverse/orgs","repos_url":"https://api.github.com/users/zooniverse/repos","events_url":"https://api.github.com/users/zooniverse/events{/privacy}","received_events_url":"https://api.github.com/users/zooniverse/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/zooniverse/generate-subjects-from-planet-api","description":"Create Zooniverse subjects via Planet Labs API using area of interest (AOI) queries.","fork":false,"url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api","forks_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/forks","keys_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/keys{/key_id}","collaborators_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/teams","hooks_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/hooks","issue_events_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/issues/events{/number}","events_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/events","assignees_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/assignees{/user}","branches_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/branches{/branch}","tags_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/tags","blobs_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/git/refs{/sha}","trees_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/git/trees{/sha}","statuses_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/statuses/{sha}","languages_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/languages","stargazers_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/stargazers","contributors_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/contributors","subscribers_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/subscribers","subscription_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/subscription","commits_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/commits{/sha}","git_commits_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/git/commits{/sha}","comments_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/comments{/number}","issue_comment_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/issues/comments{/number}","contents_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/contents/{+path}","compare_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/compare/{base}...{head}","merges_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/merges","archive_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/downloads","issues_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/issues{/number}","pulls_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/pulls{/number}","milestones_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/milestones{/number}","notifications_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/labels{/name}","releases_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/releases{/id}","deployments_url":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/deployments","created_at":"2016-01-13T22:00:00Z","updated_at":"2016-01-28T21:56:10Z","pushed_at":"2016-02-19T18:11:36Z","git_url":"git://github.com/zooniverse/generate-subjects-from-planet-api.git","ssh_url":"git@github.com:zooniverse/generate-subjects-from-planet-api.git","clone_url":"https://github.com/zooniverse/generate-subjects-from-planet-api.git","svn_url":"https://github.com/zooniverse/generate-subjects-from-planet-api","homepage":null,"size":74319,"stargazers_count":0,"watchers_count":0,"language":"JavaScript","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":1,"forks":1,"open_issues":1,"watchers":0,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/pulls/10"},"html":{"href":"https://github.com/zooniverse/generate-subjects-from-planet-api/pull/10"},"issue":{"href":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/issues/10"},"comments":{"href":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/issues/10/comments"},"review_comments":{"href":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/pulls/10/comments"},"review_comment":{"href":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/pulls/10/commits"},"statuses":{"href":"https://api.github.com/repos/zooniverse/generate-subjects-from-planet-api/statuses/008e4745f3c74388b8c4ab629cf2f9fdf4c22499"}},"merged":true,"mergeable":null,"mergeable_state":"unknown","merged_by":{"login":"itsravenous","id":1737376,"avatar_url":"https://avatars.githubusercontent.com/u/1737376?v=3","gravatar_id":"","url":"https://api.github.com/users/itsravenous","html_url":"https://github.com/itsravenous","followers_url":"https://api.github.com/users/itsravenous/followers","following_url":"https://api.github.com/users/itsravenous/following{/other_user}","gists_url":"https://api.github.com/users/itsravenous/gists{/gist_id}","starred_url":"https://api.github.com/users/itsravenous/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/itsravenous/subscriptions","organizations_url":"https://api.github.com/users/itsravenous/orgs","repos_url":"https://api.github.com/users/itsravenous/repos","events_url":"https://api.github.com/users/itsravenous/events{/privacy}","received_events_url":"https://api.github.com/users/itsravenous/received_events","type":"User","site_admin":false},"comments":0,"review_comments":0,"commits":1,"additions":197,"deletions":74,"changed_files":14}}