{"action":"created","comment":{"url":"https://api.github.com/repos/cloudytimemachine/designdocs/pulls/comments/81176905","pull_request_review_id":2176096,"id":81176905,"diff_hunk":"@@ -14,19 +14,29 @@ We should make requests using the new REST API to archive the top X websites at\n * Automated tests\n \n # Background\n-*(terminology and history one needs to know to understand the design)*\n+N/A\n \n # High-Level Design\n-*(a brief, high-level description of the design; diagrams are helpful)*\n+The data_seeder will be a lightweight application that hits the REST API to create new snapshots at given intervals. The list of data to request and the schedule of requests shall be configurable via config file or command line.\n \n # Detailed Design\n-*(the design in detail; usually a top-down description)*\n+data_seeder is a node.js app running inside a temporary container in our cluster. A very basic starting point is the test branch of CTM/api that simply fires off a bunch of POST requests to the API with a wrapper around it to make it run regularly and with smarter logic.\n+\n+The app will read in a list of sites to archive at runtime from a config file. Node-cron will handle the scheduling.\n+\n+Major design consideration is throttling/scheduling the requests so the usability of the snapshot queue is never severely impacted by the data_seeder app. In this case, we'll throttle the requests by submitting them in small bunches at regular intervals so by the end of the day we have the full lot.\n \n # Alternatives Considered\n-*(alternatives considered and rejected)*\n+* Scheduling:\n+  - Kubernetes 1.4 Scheduled jobs are not yet an option. We're on 1.3.\n+  - Cron in a container is a possibility, but I think node-cron more explicitly internalizes the scheduling to the app.\n+* Throttling:\n+  - data_seeder could check the queue length before submitting bot jobs and if the queue has more than some threshold of jobs in it, retry the submission a little later. This solution is fine but I think submitting spaced out bunches of requests is likely simpler. If we run into issues with an overloaded queue, this is the next approach.\n+  - We could do a dual priority queue. High priority for human search and bots at another lower tier. This may be something to think about in the future, but it means changes all over the rest of code-base.\n \n # Security Concerns\n-*(security/privacy ramifications and how those concerns are mitigated)*\n+N/A\n \n # Open Issues\n-*(questions regarding requirements, mockups, etc)*\n+* Do we want to log the activities of data_seeder somewhere?","path":"api/data_seeding.md","position":34,"original_position":34,"commit_id":"f9c0d4b90aa0c019659117b63597c5cd77703635","original_commit_id":"f9c0d4b90aa0c019659117b63597c5cd77703635","user":{"login":"rosskukulinski","id":2746479,"avatar_url":"https://avatars.githubusercontent.com/u/2746479?v=3","gravatar_id":"","url":"https://api.github.com/users/rosskukulinski","html_url":"https://github.com/rosskukulinski","followers_url":"https://api.github.com/users/rosskukulinski/followers","following_url":"https://api.github.com/users/rosskukulinski/following{/other_user}","gists_url":"https://api.github.com/users/rosskukulinski/gists{/gist_id}","starred_url":"https://api.github.com/users/rosskukulinski/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rosskukulinski/subscriptions","organizations_url":"https://api.github.com/users/rosskukulinski/orgs","repos_url":"https://api.github.com/users/rosskukulinski/repos","events_url":"https://api.github.com/users/rosskukulinski/events{/privacy}","received_events_url":"https://api.github.com/users/rosskukulinski/received_events","type":"User","site_admin":false},"body":"All application logs should go to `stdout` where they are captured by Kubernetes (and optionally by a log aggregator that we haven't deployed yet)","created_at":"2016-09-29T16:23:38Z","updated_at":"2016-09-29T16:24:57Z","html_url":"https://github.com/cloudytimemachine/designdocs/pull/2#discussion_r81176905","pull_request_url":"https://api.github.com/repos/cloudytimemachine/designdocs/pulls/2","_links":{"self":{"href":"https://api.github.com/repos/cloudytimemachine/designdocs/pulls/comments/81176905"},"html":{"href":"https://github.com/cloudytimemachine/designdocs/pull/2#discussion_r81176905"},"pull_request":{"href":"https://api.github.com/repos/cloudytimemachine/designdocs/pulls/2"}}},"pull_request":{"url":"https://api.github.com/repos/cloudytimemachine/designdocs/pulls/2","id":87342984,"html_url":"https://github.com/cloudytimemachine/designdocs/pull/2","diff_url":"https://github.com/cloudytimemachine/designdocs/pull/2.diff","patch_url":"https://github.com/cloudytimemachine/designdocs/pull/2.patch","issue_url":"https://api.github.com/repos/cloudytimemachine/designdocs/issues/2","number":2,"state":"open","locked":false,"title":"Revised design doc for data_seeder.","user":{"login":"geojaz","id":9451328,"avatar_url":"https://avatars.githubusercontent.com/u/9451328?v=3","gravatar_id":"","url":"https://api.github.com/users/geojaz","html_url":"https://github.com/geojaz","followers_url":"https://api.github.com/users/geojaz/followers","following_url":"https://api.github.com/users/geojaz/following{/other_user}","gists_url":"https://api.github.com/users/geojaz/gists{/gist_id}","starred_url":"https://api.github.com/users/geojaz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/geojaz/subscriptions","organizations_url":"https://api.github.com/users/geojaz/orgs","repos_url":"https://api.github.com/users/geojaz/repos","events_url":"https://api.github.com/users/geojaz/events{/privacy}","received_events_url":"https://api.github.com/users/geojaz/received_events","type":"User","site_admin":false},"body":"Thanks","created_at":"2016-09-29T16:00:46Z","updated_at":"2016-09-29T16:24:57Z","closed_at":null,"merged_at":null,"merge_commit_sha":"27cd5318bacdda695839206cbe30805791cafa31","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/cloudytimemachine/designdocs/pulls/2/commits","review_comments_url":"https://api.github.com/repos/cloudytimemachine/designdocs/pulls/2/comments","review_comment_url":"https://api.github.com/repos/cloudytimemachine/designdocs/pulls/comments{/number}","comments_url":"https://api.github.com/repos/cloudytimemachine/designdocs/issues/2/comments","statuses_url":"https://api.github.com/repos/cloudytimemachine/designdocs/statuses/f9c0d4b90aa0c019659117b63597c5cd77703635","head":{"label":"cloudytimemachine:data-cron","ref":"data-cron","sha":"f9c0d4b90aa0c019659117b63597c5cd77703635","user":{"login":"cloudytimemachine","id":20557575,"avatar_url":"https://avatars.githubusercontent.com/u/20557575?v=3","gravatar_id":"","url":"https://api.github.com/users/cloudytimemachine","html_url":"https://github.com/cloudytimemachine","followers_url":"https://api.github.com/users/cloudytimemachine/followers","following_url":"https://api.github.com/users/cloudytimemachine/following{/other_user}","gists_url":"https://api.github.com/users/cloudytimemachine/gists{/gist_id}","starred_url":"https://api.github.com/users/cloudytimemachine/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cloudytimemachine/subscriptions","organizations_url":"https://api.github.com/users/cloudytimemachine/orgs","repos_url":"https://api.github.com/users/cloudytimemachine/repos","events_url":"https://api.github.com/users/cloudytimemachine/events{/privacy}","received_events_url":"https://api.github.com/users/cloudytimemachine/received_events","type":"Organization","site_admin":false},"repo":{"id":69414818,"name":"designdocs","full_name":"cloudytimemachine/designdocs","owner":{"login":"cloudytimemachine","id":20557575,"avatar_url":"https://avatars.githubusercontent.com/u/20557575?v=3","gravatar_id":"","url":"https://api.github.com/users/cloudytimemachine","html_url":"https://github.com/cloudytimemachine","followers_url":"https://api.github.com/users/cloudytimemachine/followers","following_url":"https://api.github.com/users/cloudytimemachine/following{/other_user}","gists_url":"https://api.github.com/users/cloudytimemachine/gists{/gist_id}","starred_url":"https://api.github.com/users/cloudytimemachine/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cloudytimemachine/subscriptions","organizations_url":"https://api.github.com/users/cloudytimemachine/orgs","repos_url":"https://api.github.com/users/cloudytimemachine/repos","events_url":"https://api.github.com/users/cloudytimemachine/events{/privacy}","received_events_url":"https://api.github.com/users/cloudytimemachine/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/cloudytimemachine/designdocs","description":"Repository for Design Documents (specifications) for features in development for Cloudy Time Machine","fork":false,"url":"https://api.github.com/repos/cloudytimemachine/designdocs","forks_url":"https://api.github.com/repos/cloudytimemachine/designdocs/forks","keys_url":"https://api.github.com/repos/cloudytimemachine/designdocs/keys{/key_id}","collaborators_url":"https://api.github.com/repos/cloudytimemachine/designdocs/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/cloudytimemachine/designdocs/teams","hooks_url":"https://api.github.com/repos/cloudytimemachine/designdocs/hooks","issue_events_url":"https://api.github.com/repos/cloudytimemachine/designdocs/issues/events{/number}","events_url":"https://api.github.com/repos/cloudytimemachine/designdocs/events","assignees_url":"https://api.github.com/repos/cloudytimemachine/designdocs/assignees{/user}","branches_url":"https://api.github.com/repos/cloudytimemachine/designdocs/branches{/branch}","tags_url":"https://api.github.com/repos/cloudytimemachine/designdocs/tags","blobs_url":"https://api.github.com/repos/cloudytimemachine/designdocs/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/cloudytimemachine/designdocs/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/cloudytimemachine/designdocs/git/refs{/sha}","trees_url":"https://api.github.com/repos/cloudytimemachine/designdocs/git/trees{/sha}","statuses_url":"https://api.github.com/repos/cloudytimemachine/designdocs/statuses/{sha}","languages_url":"https://api.github.com/repos/cloudytimemachine/designdocs/languages","stargazers_url":"https://api.github.com/repos/cloudytimemachine/designdocs/stargazers","contributors_url":"https://api.github.com/repos/cloudytimemachine/designdocs/contributors","subscribers_url":"https://api.github.com/repos/cloudytimemachine/designdocs/subscribers","subscription_url":"https://api.github.com/repos/cloudytimemachine/designdocs/subscription","commits_url":"https://api.github.com/repos/cloudytimemachine/designdocs/commits{/sha}","git_commits_url":"https://api.github.com/repos/cloudytimemachine/designdocs/git/commits{/sha}","comments_url":"https://api.github.com/repos/cloudytimemachine/designdocs/comments{/number}","issue_comment_url":"https://api.github.com/repos/cloudytimemachine/designdocs/issues/comments{/number}","contents_url":"https://api.github.com/repos/cloudytimemachine/designdocs/contents/{+path}","compare_url":"https://api.github.com/repos/cloudytimemachine/designdocs/compare/{base}...{head}","merges_url":"https://api.github.com/repos/cloudytimemachine/designdocs/merges","archive_url":"https://api.github.com/repos/cloudytimemachine/designdocs/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/cloudytimemachine/designdocs/downloads","issues_url":"https://api.github.com/repos/cloudytimemachine/designdocs/issues{/number}","pulls_url":"https://api.github.com/repos/cloudytimemachine/designdocs/pulls{/number}","milestones_url":"https://api.github.com/repos/cloudytimemachine/designdocs/milestones{/number}","notifications_url":"https://api.github.com/repos/cloudytimemachine/designdocs/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/cloudytimemachine/designdocs/labels{/name}","releases_url":"https://api.github.com/repos/cloudytimemachine/designdocs/releases{/id}","deployments_url":"https://api.github.com/repos/cloudytimemachine/designdocs/deployments","created_at":"2016-09-28T01:50:24Z","updated_at":"2016-09-28T01:50:24Z","pushed_at":"2016-09-29T16:00:47Z","git_url":"git://github.com/cloudytimemachine/designdocs.git","ssh_url":"git@github.com:cloudytimemachine/designdocs.git","clone_url":"https://github.com/cloudytimemachine/designdocs.git","svn_url":"https://github.com/cloudytimemachine/designdocs","homepage":null,"size":4,"stargazers_count":0,"watchers_count":0,"language":null,"has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":1,"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"base":{"label":"cloudytimemachine:master","ref":"master","sha":"f930970edee8146f822a4bde57f597e32794df2f","user":{"login":"cloudytimemachine","id":20557575,"avatar_url":"https://avatars.githubusercontent.com/u/20557575?v=3","gravatar_id":"","url":"https://api.github.com/users/cloudytimemachine","html_url":"https://github.com/cloudytimemachine","followers_url":"https://api.github.com/users/cloudytimemachine/followers","following_url":"https://api.github.com/users/cloudytimemachine/following{/other_user}","gists_url":"https://api.github.com/users/cloudytimemachine/gists{/gist_id}","starred_url":"https://api.github.com/users/cloudytimemachine/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cloudytimemachine/subscriptions","organizations_url":"https://api.github.com/users/cloudytimemachine/orgs","repos_url":"https://api.github.com/users/cloudytimemachine/repos","events_url":"https://api.github.com/users/cloudytimemachine/events{/privacy}","received_events_url":"https://api.github.com/users/cloudytimemachine/received_events","type":"Organization","site_admin":false},"repo":{"id":69414818,"name":"designdocs","full_name":"cloudytimemachine/designdocs","owner":{"login":"cloudytimemachine","id":20557575,"avatar_url":"https://avatars.githubusercontent.com/u/20557575?v=3","gravatar_id":"","url":"https://api.github.com/users/cloudytimemachine","html_url":"https://github.com/cloudytimemachine","followers_url":"https://api.github.com/users/cloudytimemachine/followers","following_url":"https://api.github.com/users/cloudytimemachine/following{/other_user}","gists_url":"https://api.github.com/users/cloudytimemachine/gists{/gist_id}","starred_url":"https://api.github.com/users/cloudytimemachine/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cloudytimemachine/subscriptions","organizations_url":"https://api.github.com/users/cloudytimemachine/orgs","repos_url":"https://api.github.com/users/cloudytimemachine/repos","events_url":"https://api.github.com/users/cloudytimemachine/events{/privacy}","received_events_url":"https://api.github.com/users/cloudytimemachine/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/cloudytimemachine/designdocs","description":"Repository for Design Documents (specifications) for features in development for Cloudy Time Machine","fork":false,"url":"https://api.github.com/repos/cloudytimemachine/designdocs","forks_url":"https://api.github.com/repos/cloudytimemachine/designdocs/forks","keys_url":"https://api.github.com/repos/cloudytimemachine/designdocs/keys{/key_id}","collaborators_url":"https://api.github.com/repos/cloudytimemachine/designdocs/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/cloudytimemachine/designdocs/teams","hooks_url":"https://api.github.com/repos/cloudytimemachine/designdocs/hooks","issue_events_url":"https://api.github.com/repos/cloudytimemachine/designdocs/issues/events{/number}","events_url":"https://api.github.com/repos/cloudytimemachine/designdocs/events","assignees_url":"https://api.github.com/repos/cloudytimemachine/designdocs/assignees{/user}","branches_url":"https://api.github.com/repos/cloudytimemachine/designdocs/branches{/branch}","tags_url":"https://api.github.com/repos/cloudytimemachine/designdocs/tags","blobs_url":"https://api.github.com/repos/cloudytimemachine/designdocs/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/cloudytimemachine/designdocs/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/cloudytimemachine/designdocs/git/refs{/sha}","trees_url":"https://api.github.com/repos/cloudytimemachine/designdocs/git/trees{/sha}","statuses_url":"https://api.github.com/repos/cloudytimemachine/designdocs/statuses/{sha}","languages_url":"https://api.github.com/repos/cloudytimemachine/designdocs/languages","stargazers_url":"https://api.github.com/repos/cloudytimemachine/designdocs/stargazers","contributors_url":"https://api.github.com/repos/cloudytimemachine/designdocs/contributors","subscribers_url":"https://api.github.com/repos/cloudytimemachine/designdocs/subscribers","subscription_url":"https://api.github.com/repos/cloudytimemachine/designdocs/subscription","commits_url":"https://api.github.com/repos/cloudytimemachine/designdocs/commits{/sha}","git_commits_url":"https://api.github.com/repos/cloudytimemachine/designdocs/git/commits{/sha}","comments_url":"https://api.github.com/repos/cloudytimemachine/designdocs/comments{/number}","issue_comment_url":"https://api.github.com/repos/cloudytimemachine/designdocs/issues/comments{/number}","contents_url":"https://api.github.com/repos/cloudytimemachine/designdocs/contents/{+path}","compare_url":"https://api.github.com/repos/cloudytimemachine/designdocs/compare/{base}...{head}","merges_url":"https://api.github.com/repos/cloudytimemachine/designdocs/merges","archive_url":"https://api.github.com/repos/cloudytimemachine/designdocs/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/cloudytimemachine/designdocs/downloads","issues_url":"https://api.github.com/repos/cloudytimemachine/designdocs/issues{/number}","pulls_url":"https://api.github.com/repos/cloudytimemachine/designdocs/pulls{/number}","milestones_url":"https://api.github.com/repos/cloudytimemachine/designdocs/milestones{/number}","notifications_url":"https://api.github.com/repos/cloudytimemachine/designdocs/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/cloudytimemachine/designdocs/labels{/name}","releases_url":"https://api.github.com/repos/cloudytimemachine/designdocs/releases{/id}","deployments_url":"https://api.github.com/repos/cloudytimemachine/designdocs/deployments","created_at":"2016-09-28T01:50:24Z","updated_at":"2016-09-28T01:50:24Z","pushed_at":"2016-09-29T16:00:47Z","git_url":"git://github.com/cloudytimemachine/designdocs.git","ssh_url":"git@github.com:cloudytimemachine/designdocs.git","clone_url":"https://github.com/cloudytimemachine/designdocs.git","svn_url":"https://github.com/cloudytimemachine/designdocs","homepage":null,"size":4,"stargazers_count":0,"watchers_count":0,"language":null,"has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":1,"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/cloudytimemachine/designdocs/pulls/2"},"html":{"href":"https://github.com/cloudytimemachine/designdocs/pull/2"},"issue":{"href":"https://api.github.com/repos/cloudytimemachine/designdocs/issues/2"},"comments":{"href":"https://api.github.com/repos/cloudytimemachine/designdocs/issues/2/comments"},"review_comments":{"href":"https://api.github.com/repos/cloudytimemachine/designdocs/pulls/2/comments"},"review_comment":{"href":"https://api.github.com/repos/cloudytimemachine/designdocs/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/cloudytimemachine/designdocs/pulls/2/commits"},"statuses":{"href":"https://api.github.com/repos/cloudytimemachine/designdocs/statuses/f9c0d4b90aa0c019659117b63597c5cd77703635"}}}}
{"actor":{"display_login":"rosskukulinski"}}