{"action":"created","comment":{"url":"https://api.github.com/repos/airbnb/streamalert/pulls/comments/167655498","pull_request_review_id":95903650,"id":167655498,"diff_hunk":"@@ -418,13 +434,24 @@ def __init__(self, config):\n \n         self.setup()\n \n+    @property\n+    def queue_name(self):\n+        \"\"\"Return the name of the sqs queue to use. This can be overridden in the config\"\"\"\n+        queue = self.config['lambda']['athena_partition_refresh_config'].get('queue_name', '')\n+        queue = queue.replace(' ', '') # strip any spaces which are invalid queue names\n+        if queue == '':","path":"stream_alert/athena_partition_refresh/main.py","position":127,"original_position":127,"commit_id":"6afe6249cc49989bdcf4bef29fdcbed0bd15e0a1","original_commit_id":"6afe6249cc49989bdcf4bef29fdcbed0bd15e0a1","user":{"login":"chunyong-lin","id":21151436,"avatar_url":"https://avatars1.githubusercontent.com/u/21151436?v=4","gravatar_id":"","url":"https://api.github.com/users/chunyong-lin","html_url":"https://github.com/chunyong-lin","followers_url":"https://api.github.com/users/chunyong-lin/followers","following_url":"https://api.github.com/users/chunyong-lin/following{/other_user}","gists_url":"https://api.github.com/users/chunyong-lin/gists{/gist_id}","starred_url":"https://api.github.com/users/chunyong-lin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chunyong-lin/subscriptions","organizations_url":"https://api.github.com/users/chunyong-lin/orgs","repos_url":"https://api.github.com/users/chunyong-lin/repos","events_url":"https://api.github.com/users/chunyong-lin/events{/privacy}","received_events_url":"https://api.github.com/users/chunyong-lin/received_events","type":"User","site_admin":false},"body":"`if not queue:`","created_at":"2018-02-12T19:07:54Z","updated_at":"2018-02-12T19:17:23Z","html_url":"https://github.com/airbnb/streamalert/pull/592#discussion_r167655498","pull_request_url":"https://api.github.com/repos/airbnb/streamalert/pulls/592","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/airbnb/streamalert/pulls/comments/167655498"},"html":{"href":"https://github.com/airbnb/streamalert/pull/592#discussion_r167655498"},"pull_request":{"href":"https://api.github.com/repos/airbnb/streamalert/pulls/592"}}},"pull_request":{"url":"https://api.github.com/repos/airbnb/streamalert/pulls/592","id":168374121,"html_url":"https://github.com/airbnb/streamalert/pull/592","diff_url":"https://github.com/airbnb/streamalert/pull/592.diff","patch_url":"https://github.com/airbnb/streamalert/pull/592.patch","issue_url":"https://api.github.com/repos/airbnb/streamalert/issues/592","number":592,"state":"open","locked":false,"title":"[athena] changes to make athena part of a default deployment","user":{"login":"ryandeivert","id":13773789,"avatar_url":"https://avatars3.githubusercontent.com/u/13773789?v=4","gravatar_id":"","url":"https://api.github.com/users/ryandeivert","html_url":"https://github.com/ryandeivert","followers_url":"https://api.github.com/users/ryandeivert/followers","following_url":"https://api.github.com/users/ryandeivert/following{/other_user}","gists_url":"https://api.github.com/users/ryandeivert/gists{/gist_id}","starred_url":"https://api.github.com/users/ryandeivert/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ryandeivert/subscriptions","organizations_url":"https://api.github.com/users/ryandeivert/orgs","repos_url":"https://api.github.com/users/ryandeivert/repos","events_url":"https://api.github.com/users/ryandeivert/events{/privacy}","received_events_url":"https://api.github.com/users/ryandeivert/received_events","type":"User","site_admin":false},"body":"to: @austinbyers or @chunyong-lin \r\ncc: @airbnb/streamalert-maintainers, @jacknagz \r\nsize: medium\r\nresolves #N/A\r\n\r\n## Background\r\n\r\n* In order to support some upcoming features, StreamAlert must always have alerts sent to S3 to allow for Athena querying. Therefore, the Athena partition refresh function is no longer optional and should be created upon StreamAlert initialization.\r\n\r\n## Changes\r\n\r\n* Updating cli to handle deploying athena on init\r\n  * This removes some now unnecessary `manage.py` athena subcommands like `init`, `create-db`, etc.\r\n  * Athena database creation is now handled via Terraform.\r\n* The athena AWS resources are now name-spaced to the user-specified `prefix` as to not conflict if multiple deploys exist in one AWS account.\r\n* Updating athena terraform generate code and updating default local config for athena.\r\n* Three additional options related to the athena function are now configurable by the user (within the `athena_partition_refresh_config` in `lambda.json`:\r\n  * `database_name`: the name of the Athena database to use. (default is: `<prefix>_streamalert`)\r\n  * `results_bucket `: the S3 bucket to use for Athena query results and metastore storage (default is: `<prefix>.streamalert.athena-results`)\r\n  * `queue_name`: the name of the sqs queue to use for bucket notifications (default is: `<prefix>_streamalert_athena_data_bucket_notifications`)\r\n\r\n\r\n### Other Changes\r\n\r\n* Preliminary doc updates to go with changes. Another wave will follow.\r\n* Syncing carbon black logs schema updates\r\n\r\n## Testing\r\n\r\n* Updating some unit tests for changes.\r\n* Deployed in test account end-to-end and tested to make sure alerts searchable in S3.\r\n","created_at":"2018-02-10T00:40:01Z","updated_at":"2018-02-12T19:17:23Z","closed_at":null,"merged_at":null,"merge_commit_sha":"51e949438881e17ddc1b650e2ee845e522799229","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[{"id":528374571,"url":"https://api.github.com/repos/airbnb/streamalert/labels/cli","name":"cli","color":"006b75","default":false},{"id":528375916,"url":"https://api.github.com/repos/airbnb/streamalert/labels/documentation","name":"documentation","color":"c5def5","default":false},{"id":658809993,"url":"https://api.github.com/repos/airbnb/streamalert/labels/historical%20search","name":"historical search","color":"bfdadc","default":false},{"id":528375308,"url":"https://api.github.com/repos/airbnb/streamalert/labels/terraform","name":"terraform","color":"1d76db","default":false}],"milestone":{"url":"https://api.github.com/repos/airbnb/streamalert/milestones/6","html_url":"https://github.com/airbnb/streamalert/milestone/6","labels_url":"https://api.github.com/repos/airbnb/streamalert/milestones/6/labels","id":3103420,"number":6,"title":"2.0.0","description":"","creator":{"login":"ryandeivert","id":13773789,"avatar_url":"https://avatars3.githubusercontent.com/u/13773789?v=4","gravatar_id":"","url":"https://api.github.com/users/ryandeivert","html_url":"https://github.com/ryandeivert","followers_url":"https://api.github.com/users/ryandeivert/followers","following_url":"https://api.github.com/users/ryandeivert/following{/other_user}","gists_url":"https://api.github.com/users/ryandeivert/gists{/gist_id}","starred_url":"https://api.github.com/users/ryandeivert/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ryandeivert/subscriptions","organizations_url":"https://api.github.com/users/ryandeivert/orgs","repos_url":"https://api.github.com/users/ryandeivert/repos","events_url":"https://api.github.com/users/ryandeivert/events{/privacy}","received_events_url":"https://api.github.com/users/ryandeivert/received_events","type":"User","site_admin":false},"open_issues":1,"closed_issues":0,"state":"open","created_at":"2018-02-09T18:18:51Z","updated_at":"2018-02-10T00:56:50Z","due_on":"2018-03-30T07:00:00Z","closed_at":null},"commits_url":"https://api.github.com/repos/airbnb/streamalert/pulls/592/commits","review_comments_url":"https://api.github.com/repos/airbnb/streamalert/pulls/592/comments","review_comment_url":"https://api.github.com/repos/airbnb/streamalert/pulls/comments{/number}","comments_url":"https://api.github.com/repos/airbnb/streamalert/issues/592/comments","statuses_url":"https://api.github.com/repos/airbnb/streamalert/statuses/6afe6249cc49989bdcf4bef29fdcbed0bd15e0a1","head":{"label":"airbnb:ryandeivert-athena-default-output","ref":"ryandeivert-athena-default-output","sha":"6afe6249cc49989bdcf4bef29fdcbed0bd15e0a1","user":{"login":"airbnb","id":698437,"avatar_url":"https://avatars3.githubusercontent.com/u/698437?v=4","gravatar_id":"","url":"https://api.github.com/users/airbnb","html_url":"https://github.com/airbnb","followers_url":"https://api.github.com/users/airbnb/followers","following_url":"https://api.github.com/users/airbnb/following{/other_user}","gists_url":"https://api.github.com/users/airbnb/gists{/gist_id}","starred_url":"https://api.github.com/users/airbnb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/airbnb/subscriptions","organizations_url":"https://api.github.com/users/airbnb/orgs","repos_url":"https://api.github.com/users/airbnb/repos","events_url":"https://api.github.com/users/airbnb/events{/privacy}","received_events_url":"https://api.github.com/users/airbnb/received_events","type":"Organization","site_admin":false},"repo":{"id":79683936,"name":"streamalert","full_name":"airbnb/streamalert","owner":{"login":"airbnb","id":698437,"avatar_url":"https://avatars3.githubusercontent.com/u/698437?v=4","gravatar_id":"","url":"https://api.github.com/users/airbnb","html_url":"https://github.com/airbnb","followers_url":"https://api.github.com/users/airbnb/followers","following_url":"https://api.github.com/users/airbnb/following{/other_user}","gists_url":"https://api.github.com/users/airbnb/gists{/gist_id}","starred_url":"https://api.github.com/users/airbnb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/airbnb/subscriptions","organizations_url":"https://api.github.com/users/airbnb/orgs","repos_url":"https://api.github.com/users/airbnb/repos","events_url":"https://api.github.com/users/airbnb/events{/privacy}","received_events_url":"https://api.github.com/users/airbnb/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/airbnb/streamalert","description":"StreamAlert is a serverless, realtime data analysis framework which empowers you to ingest, analyze, and alert on data from any environment, using datasources and alerting logic you define.","fork":false,"url":"https://api.github.com/repos/airbnb/streamalert","forks_url":"https://api.github.com/repos/airbnb/streamalert/forks","keys_url":"https://api.github.com/repos/airbnb/streamalert/keys{/key_id}","collaborators_url":"https://api.github.com/repos/airbnb/streamalert/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/airbnb/streamalert/teams","hooks_url":"https://api.github.com/repos/airbnb/streamalert/hooks","issue_events_url":"https://api.github.com/repos/airbnb/streamalert/issues/events{/number}","events_url":"https://api.github.com/repos/airbnb/streamalert/events","assignees_url":"https://api.github.com/repos/airbnb/streamalert/assignees{/user}","branches_url":"https://api.github.com/repos/airbnb/streamalert/branches{/branch}","tags_url":"https://api.github.com/repos/airbnb/streamalert/tags","blobs_url":"https://api.github.com/repos/airbnb/streamalert/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/airbnb/streamalert/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/airbnb/streamalert/git/refs{/sha}","trees_url":"https://api.github.com/repos/airbnb/streamalert/git/trees{/sha}","statuses_url":"https://api.github.com/repos/airbnb/streamalert/statuses/{sha}","languages_url":"https://api.github.com/repos/airbnb/streamalert/languages","stargazers_url":"https://api.github.com/repos/airbnb/streamalert/stargazers","contributors_url":"https://api.github.com/repos/airbnb/streamalert/contributors","subscribers_url":"https://api.github.com/repos/airbnb/streamalert/subscribers","subscription_url":"https://api.github.com/repos/airbnb/streamalert/subscription","commits_url":"https://api.github.com/repos/airbnb/streamalert/commits{/sha}","git_commits_url":"https://api.github.com/repos/airbnb/streamalert/git/commits{/sha}","comments_url":"https://api.github.com/repos/airbnb/streamalert/comments{/number}","issue_comment_url":"https://api.github.com/repos/airbnb/streamalert/issues/comments{/number}","contents_url":"https://api.github.com/repos/airbnb/streamalert/contents/{+path}","compare_url":"https://api.github.com/repos/airbnb/streamalert/compare/{base}...{head}","merges_url":"https://api.github.com/repos/airbnb/streamalert/merges","archive_url":"https://api.github.com/repos/airbnb/streamalert/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/airbnb/streamalert/downloads","issues_url":"https://api.github.com/repos/airbnb/streamalert/issues{/number}","pulls_url":"https://api.github.com/repos/airbnb/streamalert/pulls{/number}","milestones_url":"https://api.github.com/repos/airbnb/streamalert/milestones{/number}","notifications_url":"https://api.github.com/repos/airbnb/streamalert/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/airbnb/streamalert/labels{/name}","releases_url":"https://api.github.com/repos/airbnb/streamalert/releases{/id}","deployments_url":"https://api.github.com/repos/airbnb/streamalert/deployments","created_at":"2017-01-22T01:10:56Z","updated_at":"2018-02-12T00:18:32Z","pushed_at":"2018-02-10T01:03:48Z","git_url":"git://github.com/airbnb/streamalert.git","ssh_url":"git@github.com:airbnb/streamalert.git","clone_url":"https://github.com/airbnb/streamalert.git","svn_url":"https://github.com/airbnb/streamalert","homepage":"","size":7623,"stargazers_count":1103,"watchers_count":1103,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":110,"mirror_url":null,"archived":false,"open_issues_count":63,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":110,"open_issues":63,"watchers":1103,"default_branch":"master"}},"base":{"label":"airbnb:master","ref":"master","sha":"2529d8bf34448ac45b0913b599c1c994c4343729","user":{"login":"airbnb","id":698437,"avatar_url":"https://avatars3.githubusercontent.com/u/698437?v=4","gravatar_id":"","url":"https://api.github.com/users/airbnb","html_url":"https://github.com/airbnb","followers_url":"https://api.github.com/users/airbnb/followers","following_url":"https://api.github.com/users/airbnb/following{/other_user}","gists_url":"https://api.github.com/users/airbnb/gists{/gist_id}","starred_url":"https://api.github.com/users/airbnb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/airbnb/subscriptions","organizations_url":"https://api.github.com/users/airbnb/orgs","repos_url":"https://api.github.com/users/airbnb/repos","events_url":"https://api.github.com/users/airbnb/events{/privacy}","received_events_url":"https://api.github.com/users/airbnb/received_events","type":"Organization","site_admin":false},"repo":{"id":79683936,"name":"streamalert","full_name":"airbnb/streamalert","owner":{"login":"airbnb","id":698437,"avatar_url":"https://avatars3.githubusercontent.com/u/698437?v=4","gravatar_id":"","url":"https://api.github.com/users/airbnb","html_url":"https://github.com/airbnb","followers_url":"https://api.github.com/users/airbnb/followers","following_url":"https://api.github.com/users/airbnb/following{/other_user}","gists_url":"https://api.github.com/users/airbnb/gists{/gist_id}","starred_url":"https://api.github.com/users/airbnb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/airbnb/subscriptions","organizations_url":"https://api.github.com/users/airbnb/orgs","repos_url":"https://api.github.com/users/airbnb/repos","events_url":"https://api.github.com/users/airbnb/events{/privacy}","received_events_url":"https://api.github.com/users/airbnb/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/airbnb/streamalert","description":"StreamAlert is a serverless, realtime data analysis framework which empowers you to ingest, analyze, and alert on data from any environment, using datasources and alerting logic you define.","fork":false,"url":"https://api.github.com/repos/airbnb/streamalert","forks_url":"https://api.github.com/repos/airbnb/streamalert/forks","keys_url":"https://api.github.com/repos/airbnb/streamalert/keys{/key_id}","collaborators_url":"https://api.github.com/repos/airbnb/streamalert/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/airbnb/streamalert/teams","hooks_url":"https://api.github.com/repos/airbnb/streamalert/hooks","issue_events_url":"https://api.github.com/repos/airbnb/streamalert/issues/events{/number}","events_url":"https://api.github.com/repos/airbnb/streamalert/events","assignees_url":"https://api.github.com/repos/airbnb/streamalert/assignees{/user}","branches_url":"https://api.github.com/repos/airbnb/streamalert/branches{/branch}","tags_url":"https://api.github.com/repos/airbnb/streamalert/tags","blobs_url":"https://api.github.com/repos/airbnb/streamalert/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/airbnb/streamalert/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/airbnb/streamalert/git/refs{/sha}","trees_url":"https://api.github.com/repos/airbnb/streamalert/git/trees{/sha}","statuses_url":"https://api.github.com/repos/airbnb/streamalert/statuses/{sha}","languages_url":"https://api.github.com/repos/airbnb/streamalert/languages","stargazers_url":"https://api.github.com/repos/airbnb/streamalert/stargazers","contributors_url":"https://api.github.com/repos/airbnb/streamalert/contributors","subscribers_url":"https://api.github.com/repos/airbnb/streamalert/subscribers","subscription_url":"https://api.github.com/repos/airbnb/streamalert/subscription","commits_url":"https://api.github.com/repos/airbnb/streamalert/commits{/sha}","git_commits_url":"https://api.github.com/repos/airbnb/streamalert/git/commits{/sha}","comments_url":"https://api.github.com/repos/airbnb/streamalert/comments{/number}","issue_comment_url":"https://api.github.com/repos/airbnb/streamalert/issues/comments{/number}","contents_url":"https://api.github.com/repos/airbnb/streamalert/contents/{+path}","compare_url":"https://api.github.com/repos/airbnb/streamalert/compare/{base}...{head}","merges_url":"https://api.github.com/repos/airbnb/streamalert/merges","archive_url":"https://api.github.com/repos/airbnb/streamalert/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/airbnb/streamalert/downloads","issues_url":"https://api.github.com/repos/airbnb/streamalert/issues{/number}","pulls_url":"https://api.github.com/repos/airbnb/streamalert/pulls{/number}","milestones_url":"https://api.github.com/repos/airbnb/streamalert/milestones{/number}","notifications_url":"https://api.github.com/repos/airbnb/streamalert/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/airbnb/streamalert/labels{/name}","releases_url":"https://api.github.com/repos/airbnb/streamalert/releases{/id}","deployments_url":"https://api.github.com/repos/airbnb/streamalert/deployments","created_at":"2017-01-22T01:10:56Z","updated_at":"2018-02-12T00:18:32Z","pushed_at":"2018-02-10T01:03:48Z","git_url":"git://github.com/airbnb/streamalert.git","ssh_url":"git@github.com:airbnb/streamalert.git","clone_url":"https://github.com/airbnb/streamalert.git","svn_url":"https://github.com/airbnb/streamalert","homepage":"","size":7623,"stargazers_count":1103,"watchers_count":1103,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":110,"mirror_url":null,"archived":false,"open_issues_count":63,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":110,"open_issues":63,"watchers":1103,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/airbnb/streamalert/pulls/592"},"html":{"href":"https://github.com/airbnb/streamalert/pull/592"},"issue":{"href":"https://api.github.com/repos/airbnb/streamalert/issues/592"},"comments":{"href":"https://api.github.com/repos/airbnb/streamalert/issues/592/comments"},"review_comments":{"href":"https://api.github.com/repos/airbnb/streamalert/pulls/592/comments"},"review_comment":{"href":"https://api.github.com/repos/airbnb/streamalert/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/airbnb/streamalert/pulls/592/commits"},"statuses":{"href":"https://api.github.com/repos/airbnb/streamalert/statuses/6afe6249cc49989bdcf4bef29fdcbed0bd15e0a1"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"chunyong-lin"}}