{"action":"created","comment":{"url":"https://api.github.com/repos/mozilla/socorro/pulls/comments/78840196","id":78840196,"diff_hunk":"@@ -715,6 +718,16 @@ def path(*dirs):\n             'virtual_host': config('RABBITMQ_VIRTUAL_HOST', '/'),\n             'port': config('RABBITMQ_PORT', 5672),\n         },\n+        'boto': {\n+            'access_key': config('resource.boto.access_key', ''),\n+            'bucket_name': config(\n+                'resource.boto.bucket_name', 'crashstats'),\n+            'prefix': config('resource.boto.prefix', ''),\n+            'keybuilder_class': config(\n+                'resource.boto.keybuilder_class',\n+                'socorro.external.boto.connection_context.DatePrefixKeyBuilder'\n+            ),\n+        }","path":"webapp-django/crashstats/settings/base.py","position":23,"original_position":23,"commit_id":"b363fd507865cebc0e15828010e8be6090c6da73","original_commit_id":"b363fd507865cebc0e15828010e8be6090c6da73","user":{"login":"peterbe","id":26739,"avatar_url":"https://avatars.githubusercontent.com/u/26739?v=3","gravatar_id":"","url":"https://api.github.com/users/peterbe","html_url":"https://github.com/peterbe","followers_url":"https://api.github.com/users/peterbe/followers","following_url":"https://api.github.com/users/peterbe/following{/other_user}","gists_url":"https://api.github.com/users/peterbe/gists{/gist_id}","starred_url":"https://api.github.com/users/peterbe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/peterbe/subscriptions","organizations_url":"https://api.github.com/users/peterbe/orgs","repos_url":"https://api.github.com/users/peterbe/repos","events_url":"https://api.github.com/users/peterbe/events{/privacy}","received_events_url":"https://api.github.com/users/peterbe/received_events","type":"User","site_admin":false},"body":"Hmm... Now I'm getting tired but I thought my changes were so that nothing needs to be configured once this PR lands. ...because we use `socorro/common` as well as `socorro/webapp`. ","created_at":"2016-09-14T21:02:31Z","updated_at":"2016-09-14T21:02:31Z","html_url":"https://github.com/mozilla/socorro/pull/3452#discussion_r78840196","pull_request_url":"https://api.github.com/repos/mozilla/socorro/pulls/3452","_links":{"self":{"href":"https://api.github.com/repos/mozilla/socorro/pulls/comments/78840196"},"html":{"href":"https://github.com/mozilla/socorro/pull/3452#discussion_r78840196"},"pull_request":{"href":"https://api.github.com/repos/mozilla/socorro/pulls/3452"}}},"pull_request":{"url":"https://api.github.com/repos/mozilla/socorro/pulls/3452","id":83483698,"html_url":"https://github.com/mozilla/socorro/pull/3452","diff_url":"https://github.com/mozilla/socorro/pull/3452.diff","patch_url":"https://github.com/mozilla/socorro/pull/3452.patch","issue_url":"https://api.github.com/repos/mozilla/socorro/issues/3452","number":3452,"state":"open","locked":false,"title":"fixes bug 1262252 - Crash data pure implementation","user":{"login":"peterbe","id":26739,"avatar_url":"https://avatars.githubusercontent.com/u/26739?v=3","gravatar_id":"","url":"https://api.github.com/users/peterbe","html_url":"https://github.com/peterbe","followers_url":"https://api.github.com/users/peterbe/followers","following_url":"https://api.github.com/users/peterbe/following{/other_user}","gists_url":"https://api.github.com/users/peterbe/gists{/gist_id}","starred_url":"https://api.github.com/users/peterbe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/peterbe/subscriptions","organizations_url":"https://api.github.com/users/peterbe/orgs","repos_url":"https://api.github.com/users/peterbe/repos","events_url":"https://api.github.com/users/peterbe/events{/privacy}","received_events_url":"https://api.github.com/users/peterbe/received_events","type":"User","site_admin":false},"body":"Again, sorry for the size of this. I'll try to break it down. \r\n\r\n## The general gist\r\n\r\n### Implementations\r\nIt used to be that crash data got looked up via the middleware_app and ultimately it would be `socorro.external.crash_data_base.CrashDataBase` that would do the wrapping of the `S3BotoCrashStorage` to pull down raw dumps, raw crashes, processed crashes from S3. That class would also do too much. It would listen for boto StorageError exceptions and if that happens it would put the crash ID onto the priority jobs queue in rabbitmq.\r\n\r\nWhat I've done is break that apart. And simplified. Instead of re-writing `socorro.external.crash_data_base.CrashDataBase` I created a fresh new class called `socorro.external.boto.crash_data.SimplifiedCrashData`. First of all, it's put in `boto` folder which is very explicit. Second, if this works out and goes into production I might rename it back to plain `CrashData` once I've cleaned up old legacy code. \r\n\r\nThen there's a another brand new class that deals specifically with putting jobs into the priority queue. That's all it does. \r\n\r\n[Deleting more old legacy code will be done later](https://bugzilla.mozilla.org/show_bug.cgi?id=1299465).\r\n\r\n### Webapp\r\n[Before it used be weird](https://github.com/mozilla/socorro/blob/344979da66b185fc61ba660e9848591bf6a1ef14/webapp-django/crashstats/crashstats/views.py#L1359-L1378). When loading the report index it would first try to load the [unredacted] processed crash *then* later try to load the raw crash. With some magical juggling of types of exceptions from the middleware it would either display a page that says \"Crash Not Found\", \"Crash Too Old\" or \"Crash Pending, fetching it now\". \r\n\r\nI just vastly simplified that. So **first** it loads the raw crash. If that fails (e.g. `CrashIDNotFound`) it plain and simply says \"Crash Not Found\". Secondly, when it loads the processed crash, if it's not found it puts the crash ID on the priority job AND returns a \"Crash Pending\" template (error code 200) and that would load some JavaScript code that checks `/api/ProcessedCrash/?crash_id=:crash_id` repeatedly until it exists and then reloads the page. \r\n[I hope to improve the \"Crash Not Found\" error page later](https://bugzilla.mozilla.org/show_bug.cgi?id=1299460).\r\n\r\n## How to test it\r\nLoad up a report index page (using stage data). Go into the AWS management console and go to the `org.allizom.crash-stats.rhelmer` bucket and click into the [v1/processed_crash directory](https://console.aws.amazon.com/s3/home?region=us-west-2#&bucket=org.allizom.crash-stats.rhelmer-test.crashes&prefix=v1/processed_crash/). Look up the crash ID (that you loaded in the report index) and find it in AWS with the \"Search by prefix\" search. Once found, simply delete the processed crash json file. Then **restart** your webapp (to clear the LocMemCache) and refresh the report index URL you were on. (Make sure you have the Web Console open!) It should now load the \"pending template\" and it should just work :)\r\n\r\n## The mocking\r\nThe major reason this PR is so HUGE is because `crashstats/crashstats/tests/test_views.py` does a lot of tests on top of `report_index`. Every single line of mocking (implementation directly instead of middleware JSON) had to be re-written. As I was doing that I refactored a couple of places where it had a big hardcoded python dict and use the `_SAMPLE_META` or `_SAMPLE_UNREDACED` dicts. \r\n\r\n","created_at":"2016-08-31T16:15:01Z","updated_at":"2016-09-14T21:02:31Z","closed_at":null,"merged_at":null,"merge_commit_sha":"7c5854838fbd5be31e4066b4048bd0176b288042","assignee":{"login":"adngdb","id":328790,"avatar_url":"https://avatars.githubusercontent.com/u/328790?v=3","gravatar_id":"","url":"https://api.github.com/users/adngdb","html_url":"https://github.com/adngdb","followers_url":"https://api.github.com/users/adngdb/followers","following_url":"https://api.github.com/users/adngdb/following{/other_user}","gists_url":"https://api.github.com/users/adngdb/gists{/gist_id}","starred_url":"https://api.github.com/users/adngdb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adngdb/subscriptions","organizations_url":"https://api.github.com/users/adngdb/orgs","repos_url":"https://api.github.com/users/adngdb/repos","events_url":"https://api.github.com/users/adngdb/events{/privacy}","received_events_url":"https://api.github.com/users/adngdb/received_events","type":"User","site_admin":false},"assignees":[{"login":"adngdb","id":328790,"avatar_url":"https://avatars.githubusercontent.com/u/328790?v=3","gravatar_id":"","url":"https://api.github.com/users/adngdb","html_url":"https://github.com/adngdb","followers_url":"https://api.github.com/users/adngdb/followers","following_url":"https://api.github.com/users/adngdb/following{/other_user}","gists_url":"https://api.github.com/users/adngdb/gists{/gist_id}","starred_url":"https://api.github.com/users/adngdb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adngdb/subscriptions","organizations_url":"https://api.github.com/users/adngdb/orgs","repos_url":"https://api.github.com/users/adngdb/repos","events_url":"https://api.github.com/users/adngdb/events{/privacy}","received_events_url":"https://api.github.com/users/adngdb/received_events","type":"User","site_admin":false}],"milestone":null,"commits_url":"https://api.github.com/repos/mozilla/socorro/pulls/3452/commits","review_comments_url":"https://api.github.com/repos/mozilla/socorro/pulls/3452/comments","review_comment_url":"https://api.github.com/repos/mozilla/socorro/pulls/comments{/number}","comments_url":"https://api.github.com/repos/mozilla/socorro/issues/3452/comments","statuses_url":"https://api.github.com/repos/mozilla/socorro/statuses/b363fd507865cebc0e15828010e8be6090c6da73","head":{"label":"peterbe:bug-1262252-crash-data-pure-implementation","ref":"bug-1262252-crash-data-pure-implementation","sha":"b363fd507865cebc0e15828010e8be6090c6da73","user":{"login":"peterbe","id":26739,"avatar_url":"https://avatars.githubusercontent.com/u/26739?v=3","gravatar_id":"","url":"https://api.github.com/users/peterbe","html_url":"https://github.com/peterbe","followers_url":"https://api.github.com/users/peterbe/followers","following_url":"https://api.github.com/users/peterbe/following{/other_user}","gists_url":"https://api.github.com/users/peterbe/gists{/gist_id}","starred_url":"https://api.github.com/users/peterbe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/peterbe/subscriptions","organizations_url":"https://api.github.com/users/peterbe/orgs","repos_url":"https://api.github.com/users/peterbe/repos","events_url":"https://api.github.com/users/peterbe/events{/privacy}","received_events_url":"https://api.github.com/users/peterbe/received_events","type":"User","site_admin":false},"repo":{"id":2920665,"name":"socorro","full_name":"peterbe/socorro","owner":{"login":"peterbe","id":26739,"avatar_url":"https://avatars.githubusercontent.com/u/26739?v=3","gravatar_id":"","url":"https://api.github.com/users/peterbe","html_url":"https://github.com/peterbe","followers_url":"https://api.github.com/users/peterbe/followers","following_url":"https://api.github.com/users/peterbe/following{/other_user}","gists_url":"https://api.github.com/users/peterbe/gists{/gist_id}","starred_url":"https://api.github.com/users/peterbe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/peterbe/subscriptions","organizations_url":"https://api.github.com/users/peterbe/orgs","repos_url":"https://api.github.com/users/peterbe/repos","events_url":"https://api.github.com/users/peterbe/events{/privacy}","received_events_url":"https://api.github.com/users/peterbe/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/peterbe/socorro","description":"Socorro is a server to accept and process Breakpad crash reports. ","fork":true,"url":"https://api.github.com/repos/peterbe/socorro","forks_url":"https://api.github.com/repos/peterbe/socorro/forks","keys_url":"https://api.github.com/repos/peterbe/socorro/keys{/key_id}","collaborators_url":"https://api.github.com/repos/peterbe/socorro/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/peterbe/socorro/teams","hooks_url":"https://api.github.com/repos/peterbe/socorro/hooks","issue_events_url":"https://api.github.com/repos/peterbe/socorro/issues/events{/number}","events_url":"https://api.github.com/repos/peterbe/socorro/events","assignees_url":"https://api.github.com/repos/peterbe/socorro/assignees{/user}","branches_url":"https://api.github.com/repos/peterbe/socorro/branches{/branch}","tags_url":"https://api.github.com/repos/peterbe/socorro/tags","blobs_url":"https://api.github.com/repos/peterbe/socorro/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/peterbe/socorro/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/peterbe/socorro/git/refs{/sha}","trees_url":"https://api.github.com/repos/peterbe/socorro/git/trees{/sha}","statuses_url":"https://api.github.com/repos/peterbe/socorro/statuses/{sha}","languages_url":"https://api.github.com/repos/peterbe/socorro/languages","stargazers_url":"https://api.github.com/repos/peterbe/socorro/stargazers","contributors_url":"https://api.github.com/repos/peterbe/socorro/contributors","subscribers_url":"https://api.github.com/repos/peterbe/socorro/subscribers","subscription_url":"https://api.github.com/repos/peterbe/socorro/subscription","commits_url":"https://api.github.com/repos/peterbe/socorro/commits{/sha}","git_commits_url":"https://api.github.com/repos/peterbe/socorro/git/commits{/sha}","comments_url":"https://api.github.com/repos/peterbe/socorro/comments{/number}","issue_comment_url":"https://api.github.com/repos/peterbe/socorro/issues/comments{/number}","contents_url":"https://api.github.com/repos/peterbe/socorro/contents/{+path}","compare_url":"https://api.github.com/repos/peterbe/socorro/compare/{base}...{head}","merges_url":"https://api.github.com/repos/peterbe/socorro/merges","archive_url":"https://api.github.com/repos/peterbe/socorro/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/peterbe/socorro/downloads","issues_url":"https://api.github.com/repos/peterbe/socorro/issues{/number}","pulls_url":"https://api.github.com/repos/peterbe/socorro/pulls{/number}","milestones_url":"https://api.github.com/repos/peterbe/socorro/milestones{/number}","notifications_url":"https://api.github.com/repos/peterbe/socorro/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/peterbe/socorro/labels{/name}","releases_url":"https://api.github.com/repos/peterbe/socorro/releases{/id}","deployments_url":"https://api.github.com/repos/peterbe/socorro/deployments","created_at":"2011-12-05T23:07:30Z","updated_at":"2014-06-24T00:26:16Z","pushed_at":"2016-09-13T23:57:54Z","git_url":"git://github.com/peterbe/socorro.git","ssh_url":"git@github.com:peterbe/socorro.git","clone_url":"https://github.com/peterbe/socorro.git","svn_url":"https://github.com/peterbe/socorro","homepage":"https://wiki.mozilla.org/Socorro","size":169663,"stargazers_count":1,"watchers_count":1,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":1,"default_branch":"master"}},"base":{"label":"mozilla:master","ref":"master","sha":"344979da66b185fc61ba660e9848591bf6a1ef14","user":{"login":"mozilla","id":131524,"avatar_url":"https://avatars.githubusercontent.com/u/131524?v=3","gravatar_id":"","url":"https://api.github.com/users/mozilla","html_url":"https://github.com/mozilla","followers_url":"https://api.github.com/users/mozilla/followers","following_url":"https://api.github.com/users/mozilla/following{/other_user}","gists_url":"https://api.github.com/users/mozilla/gists{/gist_id}","starred_url":"https://api.github.com/users/mozilla/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mozilla/subscriptions","organizations_url":"https://api.github.com/users/mozilla/orgs","repos_url":"https://api.github.com/users/mozilla/repos","events_url":"https://api.github.com/users/mozilla/events{/privacy}","received_events_url":"https://api.github.com/users/mozilla/received_events","type":"Organization","site_admin":false},"repo":{"id":2263742,"name":"socorro","full_name":"mozilla/socorro","owner":{"login":"mozilla","id":131524,"avatar_url":"https://avatars.githubusercontent.com/u/131524?v=3","gravatar_id":"","url":"https://api.github.com/users/mozilla","html_url":"https://github.com/mozilla","followers_url":"https://api.github.com/users/mozilla/followers","following_url":"https://api.github.com/users/mozilla/following{/other_user}","gists_url":"https://api.github.com/users/mozilla/gists{/gist_id}","starred_url":"https://api.github.com/users/mozilla/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mozilla/subscriptions","organizations_url":"https://api.github.com/users/mozilla/orgs","repos_url":"https://api.github.com/users/mozilla/repos","events_url":"https://api.github.com/users/mozilla/events{/privacy}","received_events_url":"https://api.github.com/users/mozilla/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/mozilla/socorro","description":"Socorro is a server to accept and process Breakpad crash reports. ","fork":false,"url":"https://api.github.com/repos/mozilla/socorro","forks_url":"https://api.github.com/repos/mozilla/socorro/forks","keys_url":"https://api.github.com/repos/mozilla/socorro/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mozilla/socorro/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mozilla/socorro/teams","hooks_url":"https://api.github.com/repos/mozilla/socorro/hooks","issue_events_url":"https://api.github.com/repos/mozilla/socorro/issues/events{/number}","events_url":"https://api.github.com/repos/mozilla/socorro/events","assignees_url":"https://api.github.com/repos/mozilla/socorro/assignees{/user}","branches_url":"https://api.github.com/repos/mozilla/socorro/branches{/branch}","tags_url":"https://api.github.com/repos/mozilla/socorro/tags","blobs_url":"https://api.github.com/repos/mozilla/socorro/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mozilla/socorro/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mozilla/socorro/git/refs{/sha}","trees_url":"https://api.github.com/repos/mozilla/socorro/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mozilla/socorro/statuses/{sha}","languages_url":"https://api.github.com/repos/mozilla/socorro/languages","stargazers_url":"https://api.github.com/repos/mozilla/socorro/stargazers","contributors_url":"https://api.github.com/repos/mozilla/socorro/contributors","subscribers_url":"https://api.github.com/repos/mozilla/socorro/subscribers","subscription_url":"https://api.github.com/repos/mozilla/socorro/subscription","commits_url":"https://api.github.com/repos/mozilla/socorro/commits{/sha}","git_commits_url":"https://api.github.com/repos/mozilla/socorro/git/commits{/sha}","comments_url":"https://api.github.com/repos/mozilla/socorro/comments{/number}","issue_comment_url":"https://api.github.com/repos/mozilla/socorro/issues/comments{/number}","contents_url":"https://api.github.com/repos/mozilla/socorro/contents/{+path}","compare_url":"https://api.github.com/repos/mozilla/socorro/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mozilla/socorro/merges","archive_url":"https://api.github.com/repos/mozilla/socorro/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mozilla/socorro/downloads","issues_url":"https://api.github.com/repos/mozilla/socorro/issues{/number}","pulls_url":"https://api.github.com/repos/mozilla/socorro/pulls{/number}","milestones_url":"https://api.github.com/repos/mozilla/socorro/milestones{/number}","notifications_url":"https://api.github.com/repos/mozilla/socorro/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mozilla/socorro/labels{/name}","releases_url":"https://api.github.com/repos/mozilla/socorro/releases{/id}","deployments_url":"https://api.github.com/repos/mozilla/socorro/deployments","created_at":"2011-08-24T19:21:49Z","updated_at":"2016-09-08T01:11:21Z","pushed_at":"2016-09-14T16:42:04Z","git_url":"git://github.com/mozilla/socorro.git","ssh_url":"git@github.com:mozilla/socorro.git","clone_url":"https://github.com/mozilla/socorro.git","svn_url":"https://github.com/mozilla/socorro","homepage":"https://wiki.mozilla.org/Socorro","size":178155,"stargazers_count":314,"watchers_count":314,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":161,"mirror_url":null,"open_issues_count":6,"forks":161,"open_issues":6,"watchers":314,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/mozilla/socorro/pulls/3452"},"html":{"href":"https://github.com/mozilla/socorro/pull/3452"},"issue":{"href":"https://api.github.com/repos/mozilla/socorro/issues/3452"},"comments":{"href":"https://api.github.com/repos/mozilla/socorro/issues/3452/comments"},"review_comments":{"href":"https://api.github.com/repos/mozilla/socorro/pulls/3452/comments"},"review_comment":{"href":"https://api.github.com/repos/mozilla/socorro/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/mozilla/socorro/pulls/3452/commits"},"statuses":{"href":"https://api.github.com/repos/mozilla/socorro/statuses/b363fd507865cebc0e15828010e8be6090c6da73"}}}}
{"actor":{"display_login":"peterbe"}}