{"action":"created","comment":{"url":"https://api.github.com/repos/rubygems/rubygems.org/pulls/comments/53184950","id":53184950,"diff_hunk":"@@ -0,0 +1,4 @@\n+module Gemcutter\n+  # Feature flag for counting downloads through stats-update or FastlyLogProcessor\n+  ENABLE_FASTLY_LOG_PROCESSOR = Rails.env.test? || Rails.env.development?","path":"config/initializers/download_processor.rb","position":3,"original_position":3,"commit_id":"825ffd9abd96d9e9a6e005f7811c5c12d004042a","original_commit_id":"13f1babf20048815531d7ef9540bd0b7fe7f4abd","user":{"login":"ktheory","id":975,"avatar_url":"https://avatars.githubusercontent.com/u/975?v=3","gravatar_id":"","url":"https://api.github.com/users/ktheory","html_url":"https://github.com/ktheory","followers_url":"https://api.github.com/users/ktheory/followers","following_url":"https://api.github.com/users/ktheory/following{/other_user}","gists_url":"https://api.github.com/users/ktheory/gists{/gist_id}","starred_url":"https://api.github.com/users/ktheory/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ktheory/subscriptions","organizations_url":"https://api.github.com/users/ktheory/orgs","repos_url":"https://api.github.com/users/ktheory/repos","events_url":"https://api.github.com/users/ktheory/events{/privacy}","received_events_url":"https://api.github.com/users/ktheory/received_events","type":"User","site_admin":false},"body":"> why do we need this constant here? can we move this to the Download model or the log processor itself?\r\n\r\nWe don't need it *here*, but we need it somewhere. :smile:\r\n\r\nSince it's serving as a feature flag, it's more appropriate as config than as app or library code. My rule of thumb is that app code can talk to config code; and config code can use `Rails.env`. But app code should never directly check `Rails.env`.\r\n\r\n`app <--> config <--> environments`\r\n\r\nFWIW, this is a temporary feature flag while we dark-launch Fastly log processing. Once we're confident it's working well, we'd remove this.","created_at":"2016-02-17T15:58:51Z","updated_at":"2016-02-17T15:58:51Z","html_url":"https://github.com/rubygems/rubygems.org/pull/1176#discussion_r53184950","pull_request_url":"https://api.github.com/repos/rubygems/rubygems.org/pulls/1176","_links":{"self":{"href":"https://api.github.com/repos/rubygems/rubygems.org/pulls/comments/53184950"},"html":{"href":"https://github.com/rubygems/rubygems.org/pull/1176#discussion_r53184950"},"pull_request":{"href":"https://api.github.com/repos/rubygems/rubygems.org/pulls/1176"}}},"pull_request":{"url":"https://api.github.com/repos/rubygems/rubygems.org/pulls/1176","id":56232976,"html_url":"https://github.com/rubygems/rubygems.org/pull/1176","diff_url":"https://github.com/rubygems/rubygems.org/pull/1176.diff","patch_url":"https://github.com/rubygems/rubygems.org/pull/1176.patch","issue_url":"https://api.github.com/repos/rubygems/rubygems.org/issues/1176","number":1176,"state":"open","locked":false,"title":"Add Fastly log processor (work-in-progress)","user":{"login":"ktheory","id":975,"avatar_url":"https://avatars.githubusercontent.com/u/975?v=3","gravatar_id":"","url":"https://api.github.com/users/ktheory","html_url":"https://github.com/ktheory","followers_url":"https://api.github.com/users/ktheory/followers","following_url":"https://api.github.com/users/ktheory/following{/other_user}","gists_url":"https://api.github.com/users/ktheory/gists{/gist_id}","starred_url":"https://api.github.com/users/ktheory/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ktheory/subscriptions","organizations_url":"https://api.github.com/users/ktheory/orgs","repos_url":"https://api.github.com/users/ktheory/repos","events_url":"https://api.github.com/users/ktheory/events{/privacy}","received_events_url":"https://api.github.com/users/ktheory/received_events","type":"User","site_admin":false},"body":":wave: Happy Friday :ok_woman: \r\n\r\nThis implements much of rubygems/rubygems-infrastructure#35, allowing us to update Rubygem download counts in bulk using Fastly access logs rather than on each HTTP request.\r\n\r\nI still have some cleanup to do, but wanted to get some :eyes: earlier.\r\n\r\n### To Do\r\n- [x] add a feature flag to control whether to have FastlyLogProcessor actually update stats, or just log results. (It would only log initially.)\r\n- [ ] figure out how to disable the `stat-update` C library as part of the feature flag\r\n- [ ] Cleanup & test shoryuken config, preferably loading SQS queue names from ENV vars\r\n- [ ] Make another PR to run a `shoryuken` process on deploy once Chef provisions a runit script for it\r\n\r\n### Deploy plan\r\n\r\nNB that this code is disabled right now. We're not currently running a `shoryuken` worker, and so nothing would enqueue a `FastlyLogProcessor` job.\r\n\r\nOnce this is merged, the next step is to start running a `shoryuken` process, with `FastlyLogProcessor` only logging results & not updating redis.\r\n\r\nAfter that I'll toggle the feature flag so `FastlyLogProcessor` updates stats, and disable `stat-update` per request.\r\n\r\nFYI @dwradcliffe ","created_at":"2016-01-15T23:04:57Z","updated_at":"2016-02-17T15:58:51Z","closed_at":null,"merged_at":null,"merge_commit_sha":"3b06ff3047540083e199ecea6b24a9e282790982","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/rubygems/rubygems.org/pulls/1176/commits","review_comments_url":"https://api.github.com/repos/rubygems/rubygems.org/pulls/1176/comments","review_comment_url":"https://api.github.com/repos/rubygems/rubygems.org/pulls/comments{/number}","comments_url":"https://api.github.com/repos/rubygems/rubygems.org/issues/1176/comments","statuses_url":"https://api.github.com/repos/rubygems/rubygems.org/statuses/825ffd9abd96d9e9a6e005f7811c5c12d004042a","head":{"label":"ktheory:fastly-log-processor","ref":"fastly-log-processor","sha":"825ffd9abd96d9e9a6e005f7811c5c12d004042a","user":{"login":"ktheory","id":975,"avatar_url":"https://avatars.githubusercontent.com/u/975?v=3","gravatar_id":"","url":"https://api.github.com/users/ktheory","html_url":"https://github.com/ktheory","followers_url":"https://api.github.com/users/ktheory/followers","following_url":"https://api.github.com/users/ktheory/following{/other_user}","gists_url":"https://api.github.com/users/ktheory/gists{/gist_id}","starred_url":"https://api.github.com/users/ktheory/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ktheory/subscriptions","organizations_url":"https://api.github.com/users/ktheory/orgs","repos_url":"https://api.github.com/users/ktheory/repos","events_url":"https://api.github.com/users/ktheory/events{/privacy}","received_events_url":"https://api.github.com/users/ktheory/received_events","type":"User","site_admin":false},"repo":{"id":47340867,"name":"rubygems.org","full_name":"ktheory/rubygems.org","owner":{"login":"ktheory","id":975,"avatar_url":"https://avatars.githubusercontent.com/u/975?v=3","gravatar_id":"","url":"https://api.github.com/users/ktheory","html_url":"https://github.com/ktheory","followers_url":"https://api.github.com/users/ktheory/followers","following_url":"https://api.github.com/users/ktheory/following{/other_user}","gists_url":"https://api.github.com/users/ktheory/gists{/gist_id}","starred_url":"https://api.github.com/users/ktheory/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ktheory/subscriptions","organizations_url":"https://api.github.com/users/ktheory/orgs","repos_url":"https://api.github.com/users/ktheory/repos","events_url":"https://api.github.com/users/ktheory/events{/privacy}","received_events_url":"https://api.github.com/users/ktheory/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/ktheory/rubygems.org","description":"The Ruby community's gem hosting service.","fork":true,"url":"https://api.github.com/repos/ktheory/rubygems.org","forks_url":"https://api.github.com/repos/ktheory/rubygems.org/forks","keys_url":"https://api.github.com/repos/ktheory/rubygems.org/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ktheory/rubygems.org/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ktheory/rubygems.org/teams","hooks_url":"https://api.github.com/repos/ktheory/rubygems.org/hooks","issue_events_url":"https://api.github.com/repos/ktheory/rubygems.org/issues/events{/number}","events_url":"https://api.github.com/repos/ktheory/rubygems.org/events","assignees_url":"https://api.github.com/repos/ktheory/rubygems.org/assignees{/user}","branches_url":"https://api.github.com/repos/ktheory/rubygems.org/branches{/branch}","tags_url":"https://api.github.com/repos/ktheory/rubygems.org/tags","blobs_url":"https://api.github.com/repos/ktheory/rubygems.org/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ktheory/rubygems.org/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ktheory/rubygems.org/git/refs{/sha}","trees_url":"https://api.github.com/repos/ktheory/rubygems.org/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ktheory/rubygems.org/statuses/{sha}","languages_url":"https://api.github.com/repos/ktheory/rubygems.org/languages","stargazers_url":"https://api.github.com/repos/ktheory/rubygems.org/stargazers","contributors_url":"https://api.github.com/repos/ktheory/rubygems.org/contributors","subscribers_url":"https://api.github.com/repos/ktheory/rubygems.org/subscribers","subscription_url":"https://api.github.com/repos/ktheory/rubygems.org/subscription","commits_url":"https://api.github.com/repos/ktheory/rubygems.org/commits{/sha}","git_commits_url":"https://api.github.com/repos/ktheory/rubygems.org/git/commits{/sha}","comments_url":"https://api.github.com/repos/ktheory/rubygems.org/comments{/number}","issue_comment_url":"https://api.github.com/repos/ktheory/rubygems.org/issues/comments{/number}","contents_url":"https://api.github.com/repos/ktheory/rubygems.org/contents/{+path}","compare_url":"https://api.github.com/repos/ktheory/rubygems.org/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ktheory/rubygems.org/merges","archive_url":"https://api.github.com/repos/ktheory/rubygems.org/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ktheory/rubygems.org/downloads","issues_url":"https://api.github.com/repos/ktheory/rubygems.org/issues{/number}","pulls_url":"https://api.github.com/repos/ktheory/rubygems.org/pulls{/number}","milestones_url":"https://api.github.com/repos/ktheory/rubygems.org/milestones{/number}","notifications_url":"https://api.github.com/repos/ktheory/rubygems.org/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ktheory/rubygems.org/labels{/name}","releases_url":"https://api.github.com/repos/ktheory/rubygems.org/releases{/id}","deployments_url":"https://api.github.com/repos/ktheory/rubygems.org/deployments","created_at":"2015-12-03T15:30:22Z","updated_at":"2015-12-03T15:30:24Z","pushed_at":"2016-02-17T15:43:56Z","git_url":"git://github.com/ktheory/rubygems.org.git","ssh_url":"git@github.com:ktheory/rubygems.org.git","clone_url":"https://github.com/ktheory/rubygems.org.git","svn_url":"https://github.com/ktheory/rubygems.org","homepage":"https://rubygems.org","size":8607,"stargazers_count":0,"watchers_count":0,"language":"HTML","has_issues":false,"has_downloads":false,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"rubygems:master","ref":"master","sha":"e000c341c0dcd0126f6a1b88c60585ff87b5bb50","user":{"login":"rubygems","id":208761,"avatar_url":"https://avatars.githubusercontent.com/u/208761?v=3","gravatar_id":"","url":"https://api.github.com/users/rubygems","html_url":"https://github.com/rubygems","followers_url":"https://api.github.com/users/rubygems/followers","following_url":"https://api.github.com/users/rubygems/following{/other_user}","gists_url":"https://api.github.com/users/rubygems/gists{/gist_id}","starred_url":"https://api.github.com/users/rubygems/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rubygems/subscriptions","organizations_url":"https://api.github.com/users/rubygems/orgs","repos_url":"https://api.github.com/users/rubygems/repos","events_url":"https://api.github.com/users/rubygems/events{/privacy}","received_events_url":"https://api.github.com/users/rubygems/received_events","type":"Organization","site_admin":false},"repo":{"id":117646,"name":"rubygems.org","full_name":"rubygems/rubygems.org","owner":{"login":"rubygems","id":208761,"avatar_url":"https://avatars.githubusercontent.com/u/208761?v=3","gravatar_id":"","url":"https://api.github.com/users/rubygems","html_url":"https://github.com/rubygems","followers_url":"https://api.github.com/users/rubygems/followers","following_url":"https://api.github.com/users/rubygems/following{/other_user}","gists_url":"https://api.github.com/users/rubygems/gists{/gist_id}","starred_url":"https://api.github.com/users/rubygems/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rubygems/subscriptions","organizations_url":"https://api.github.com/users/rubygems/orgs","repos_url":"https://api.github.com/users/rubygems/repos","events_url":"https://api.github.com/users/rubygems/events{/privacy}","received_events_url":"https://api.github.com/users/rubygems/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/rubygems/rubygems.org","description":"The Ruby community's gem hosting service.","fork":false,"url":"https://api.github.com/repos/rubygems/rubygems.org","forks_url":"https://api.github.com/repos/rubygems/rubygems.org/forks","keys_url":"https://api.github.com/repos/rubygems/rubygems.org/keys{/key_id}","collaborators_url":"https://api.github.com/repos/rubygems/rubygems.org/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/rubygems/rubygems.org/teams","hooks_url":"https://api.github.com/repos/rubygems/rubygems.org/hooks","issue_events_url":"https://api.github.com/repos/rubygems/rubygems.org/issues/events{/number}","events_url":"https://api.github.com/repos/rubygems/rubygems.org/events","assignees_url":"https://api.github.com/repos/rubygems/rubygems.org/assignees{/user}","branches_url":"https://api.github.com/repos/rubygems/rubygems.org/branches{/branch}","tags_url":"https://api.github.com/repos/rubygems/rubygems.org/tags","blobs_url":"https://api.github.com/repos/rubygems/rubygems.org/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/rubygems/rubygems.org/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/rubygems/rubygems.org/git/refs{/sha}","trees_url":"https://api.github.com/repos/rubygems/rubygems.org/git/trees{/sha}","statuses_url":"https://api.github.com/repos/rubygems/rubygems.org/statuses/{sha}","languages_url":"https://api.github.com/repos/rubygems/rubygems.org/languages","stargazers_url":"https://api.github.com/repos/rubygems/rubygems.org/stargazers","contributors_url":"https://api.github.com/repos/rubygems/rubygems.org/contributors","subscribers_url":"https://api.github.com/repos/rubygems/rubygems.org/subscribers","subscription_url":"https://api.github.com/repos/rubygems/rubygems.org/subscription","commits_url":"https://api.github.com/repos/rubygems/rubygems.org/commits{/sha}","git_commits_url":"https://api.github.com/repos/rubygems/rubygems.org/git/commits{/sha}","comments_url":"https://api.github.com/repos/rubygems/rubygems.org/comments{/number}","issue_comment_url":"https://api.github.com/repos/rubygems/rubygems.org/issues/comments{/number}","contents_url":"https://api.github.com/repos/rubygems/rubygems.org/contents/{+path}","compare_url":"https://api.github.com/repos/rubygems/rubygems.org/compare/{base}...{head}","merges_url":"https://api.github.com/repos/rubygems/rubygems.org/merges","archive_url":"https://api.github.com/repos/rubygems/rubygems.org/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/rubygems/rubygems.org/downloads","issues_url":"https://api.github.com/repos/rubygems/rubygems.org/issues{/number}","pulls_url":"https://api.github.com/repos/rubygems/rubygems.org/pulls{/number}","milestones_url":"https://api.github.com/repos/rubygems/rubygems.org/milestones{/number}","notifications_url":"https://api.github.com/repos/rubygems/rubygems.org/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/rubygems/rubygems.org/labels{/name}","releases_url":"https://api.github.com/repos/rubygems/rubygems.org/releases{/id}","deployments_url":"https://api.github.com/repos/rubygems/rubygems.org/deployments","created_at":"2009-01-29T22:30:46Z","updated_at":"2016-02-16T10:56:23Z","pushed_at":"2016-02-17T15:43:57Z","git_url":"git://github.com/rubygems/rubygems.org.git","ssh_url":"git@github.com:rubygems/rubygems.org.git","clone_url":"https://github.com/rubygems/rubygems.org.git","svn_url":"https://github.com/rubygems/rubygems.org","homepage":"https://rubygems.org","size":635203,"stargazers_count":1338,"watchers_count":1338,"language":"HTML","has_issues":true,"has_downloads":false,"has_wiki":true,"has_pages":false,"forks_count":522,"mirror_url":null,"open_issues_count":52,"forks":522,"open_issues":52,"watchers":1338,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/rubygems/rubygems.org/pulls/1176"},"html":{"href":"https://github.com/rubygems/rubygems.org/pull/1176"},"issue":{"href":"https://api.github.com/repos/rubygems/rubygems.org/issues/1176"},"comments":{"href":"https://api.github.com/repos/rubygems/rubygems.org/issues/1176/comments"},"review_comments":{"href":"https://api.github.com/repos/rubygems/rubygems.org/pulls/1176/comments"},"review_comment":{"href":"https://api.github.com/repos/rubygems/rubygems.org/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/rubygems/rubygems.org/pulls/1176/commits"},"statuses":{"href":"https://api.github.com/repos/rubygems/rubygems.org/statuses/825ffd9abd96d9e9a6e005f7811c5c12d004042a"}}}}