{"action":"closed","number":612,"pull_request":{"url":"https://api.github.com/repos/airbnb/streamalert/pulls/612","id":171809816,"html_url":"https://github.com/airbnb/streamalert/pull/612","diff_url":"https://github.com/airbnb/streamalert/pull/612.diff","patch_url":"https://github.com/airbnb/streamalert/pull/612.patch","issue_url":"https://api.github.com/repos/airbnb/streamalert/issues/612","number":612,"state":"closed","locked":false,"title":"Fork alerts to a Dynamo table","user":{"login":"austinbyers","id":3608925,"avatar_url":"https://avatars0.githubusercontent.com/u/3608925?v=4","gravatar_id":"","url":"https://api.github.com/users/austinbyers","html_url":"https://github.com/austinbyers","followers_url":"https://api.github.com/users/austinbyers/followers","following_url":"https://api.github.com/users/austinbyers/following{/other_user}","gists_url":"https://api.github.com/users/austinbyers/gists{/gist_id}","starred_url":"https://api.github.com/users/austinbyers/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/austinbyers/subscriptions","organizations_url":"https://api.github.com/users/austinbyers/orgs","repos_url":"https://api.github.com/users/austinbyers/repos","events_url":"https://api.github.com/users/austinbyers/events{/privacy}","received_events_url":"https://api.github.com/users/austinbyers/received_events","type":"User","site_admin":false},"body":"to: @jacknagz and @ryandeivert \r\ncc: @airbnb/streamalert-maintainers \r\nsize: medium\r\n\r\n## Background\r\n\r\nIn order to support alert merging (and more reliable alert delivery), we need to start keeping state about the alerts. We will do this with a Dynamo table that sits in between the Rule Processor and Alert Processor.\r\n\r\n## Changes\r\nThis PR creates the new alerts table and forks all alerts to the table *in addition to the alert processor*, so we can test the alerts table in production. We will sever the rule processor - alert processor connection in a subsequent pull request after we verify that writing to the table is working as expected.\r\n\r\n* Fixes a bug in the `tf_alert_processor_iam` module when the output lists are empty\r\n* Replaces the `LogsAndMetrics` custom policy in the `tf_lambda` module with the managed `AWSLambdaBasicExecutionRole` policy\r\n* Adds the alerts table to the `tf_globals` module, with capacity configurable in `conf/global.json`\r\n* Changes and renames `sink.py` in the rule processor to write alerts in batches to Dynamo after sending them to Lambda.\r\n\r\n## Testing\r\n\r\nDeployed in a test account, alerts are sent to both the alert processor and Dynamo as expected. The Dynamo entry looks like this:\r\n\r\n![screen shot 2018-02-28 at 2 58 31 pm](https://user-images.githubusercontent.com/3608925/36817966-f04aca8e-1c97-11e8-9347-ffc54d186563.png)\r\n","created_at":"2018-02-27T22:59:29Z","updated_at":"2018-03-01T22:08:35Z","closed_at":"2018-03-01T22:08:35Z","merged_at":"2018-03-01T22:08:35Z","merge_commit_sha":"0f6a8f98a50549ebd021adadfe787398e1f47a28","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/airbnb/streamalert/pulls/612/commits","review_comments_url":"https://api.github.com/repos/airbnb/streamalert/pulls/612/comments","review_comment_url":"https://api.github.com/repos/airbnb/streamalert/pulls/comments{/number}","comments_url":"https://api.github.com/repos/airbnb/streamalert/issues/612/comments","statuses_url":"https://api.github.com/repos/airbnb/streamalert/statuses/35e6cfb5e28ca3ae393e03722647dec9ea266084","head":{"label":"airbnb:austin-dynamo","ref":"austin-dynamo","sha":"35e6cfb5e28ca3ae393e03722647dec9ea266084","user":{"login":"airbnb","id":698437,"avatar_url":"https://avatars3.githubusercontent.com/u/698437?v=4","gravatar_id":"","url":"https://api.github.com/users/airbnb","html_url":"https://github.com/airbnb","followers_url":"https://api.github.com/users/airbnb/followers","following_url":"https://api.github.com/users/airbnb/following{/other_user}","gists_url":"https://api.github.com/users/airbnb/gists{/gist_id}","starred_url":"https://api.github.com/users/airbnb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/airbnb/subscriptions","organizations_url":"https://api.github.com/users/airbnb/orgs","repos_url":"https://api.github.com/users/airbnb/repos","events_url":"https://api.github.com/users/airbnb/events{/privacy}","received_events_url":"https://api.github.com/users/airbnb/received_events","type":"Organization","site_admin":false},"repo":{"id":79683936,"name":"streamalert","full_name":"airbnb/streamalert","owner":{"login":"airbnb","id":698437,"avatar_url":"https://avatars3.githubusercontent.com/u/698437?v=4","gravatar_id":"","url":"https://api.github.com/users/airbnb","html_url":"https://github.com/airbnb","followers_url":"https://api.github.com/users/airbnb/followers","following_url":"https://api.github.com/users/airbnb/following{/other_user}","gists_url":"https://api.github.com/users/airbnb/gists{/gist_id}","starred_url":"https://api.github.com/users/airbnb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/airbnb/subscriptions","organizations_url":"https://api.github.com/users/airbnb/orgs","repos_url":"https://api.github.com/users/airbnb/repos","events_url":"https://api.github.com/users/airbnb/events{/privacy}","received_events_url":"https://api.github.com/users/airbnb/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/airbnb/streamalert","description":"StreamAlert is a serverless, realtime data analysis framework which empowers you to ingest, analyze, and alert on data from any environment, using datasources and alerting logic you define.","fork":false,"url":"https://api.github.com/repos/airbnb/streamalert","forks_url":"https://api.github.com/repos/airbnb/streamalert/forks","keys_url":"https://api.github.com/repos/airbnb/streamalert/keys{/key_id}","collaborators_url":"https://api.github.com/repos/airbnb/streamalert/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/airbnb/streamalert/teams","hooks_url":"https://api.github.com/repos/airbnb/streamalert/hooks","issue_events_url":"https://api.github.com/repos/airbnb/streamalert/issues/events{/number}","events_url":"https://api.github.com/repos/airbnb/streamalert/events","assignees_url":"https://api.github.com/repos/airbnb/streamalert/assignees{/user}","branches_url":"https://api.github.com/repos/airbnb/streamalert/branches{/branch}","tags_url":"https://api.github.com/repos/airbnb/streamalert/tags","blobs_url":"https://api.github.com/repos/airbnb/streamalert/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/airbnb/streamalert/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/airbnb/streamalert/git/refs{/sha}","trees_url":"https://api.github.com/repos/airbnb/streamalert/git/trees{/sha}","statuses_url":"https://api.github.com/repos/airbnb/streamalert/statuses/{sha}","languages_url":"https://api.github.com/repos/airbnb/streamalert/languages","stargazers_url":"https://api.github.com/repos/airbnb/streamalert/stargazers","contributors_url":"https://api.github.com/repos/airbnb/streamalert/contributors","subscribers_url":"https://api.github.com/repos/airbnb/streamalert/subscribers","subscription_url":"https://api.github.com/repos/airbnb/streamalert/subscription","commits_url":"https://api.github.com/repos/airbnb/streamalert/commits{/sha}","git_commits_url":"https://api.github.com/repos/airbnb/streamalert/git/commits{/sha}","comments_url":"https://api.github.com/repos/airbnb/streamalert/comments{/number}","issue_comment_url":"https://api.github.com/repos/airbnb/streamalert/issues/comments{/number}","contents_url":"https://api.github.com/repos/airbnb/streamalert/contents/{+path}","compare_url":"https://api.github.com/repos/airbnb/streamalert/compare/{base}...{head}","merges_url":"https://api.github.com/repos/airbnb/streamalert/merges","archive_url":"https://api.github.com/repos/airbnb/streamalert/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/airbnb/streamalert/downloads","issues_url":"https://api.github.com/repos/airbnb/streamalert/issues{/number}","pulls_url":"https://api.github.com/repos/airbnb/streamalert/pulls{/number}","milestones_url":"https://api.github.com/repos/airbnb/streamalert/milestones{/number}","notifications_url":"https://api.github.com/repos/airbnb/streamalert/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/airbnb/streamalert/labels{/name}","releases_url":"https://api.github.com/repos/airbnb/streamalert/releases{/id}","deployments_url":"https://api.github.com/repos/airbnb/streamalert/deployments","created_at":"2017-01-22T01:10:56Z","updated_at":"2018-03-01T16:35:48Z","pushed_at":"2018-03-01T22:08:35Z","git_url":"git://github.com/airbnb/streamalert.git","ssh_url":"git@github.com:airbnb/streamalert.git","clone_url":"https://github.com/airbnb/streamalert.git","svn_url":"https://github.com/airbnb/streamalert","homepage":"","size":7779,"stargazers_count":1147,"watchers_count":1147,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":116,"mirror_url":null,"archived":false,"open_issues_count":59,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":116,"open_issues":59,"watchers":1147,"default_branch":"master"}},"base":{"label":"airbnb:release-2.0.0","ref":"release-2.0.0","sha":"282d893dfcd4c13ebe2b3c93a0ba8452e3c51f82","user":{"login":"airbnb","id":698437,"avatar_url":"https://avatars3.githubusercontent.com/u/698437?v=4","gravatar_id":"","url":"https://api.github.com/users/airbnb","html_url":"https://github.com/airbnb","followers_url":"https://api.github.com/users/airbnb/followers","following_url":"https://api.github.com/users/airbnb/following{/other_user}","gists_url":"https://api.github.com/users/airbnb/gists{/gist_id}","starred_url":"https://api.github.com/users/airbnb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/airbnb/subscriptions","organizations_url":"https://api.github.com/users/airbnb/orgs","repos_url":"https://api.github.com/users/airbnb/repos","events_url":"https://api.github.com/users/airbnb/events{/privacy}","received_events_url":"https://api.github.com/users/airbnb/received_events","type":"Organization","site_admin":false},"repo":{"id":79683936,"name":"streamalert","full_name":"airbnb/streamalert","owner":{"login":"airbnb","id":698437,"avatar_url":"https://avatars3.githubusercontent.com/u/698437?v=4","gravatar_id":"","url":"https://api.github.com/users/airbnb","html_url":"https://github.com/airbnb","followers_url":"https://api.github.com/users/airbnb/followers","following_url":"https://api.github.com/users/airbnb/following{/other_user}","gists_url":"https://api.github.com/users/airbnb/gists{/gist_id}","starred_url":"https://api.github.com/users/airbnb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/airbnb/subscriptions","organizations_url":"https://api.github.com/users/airbnb/orgs","repos_url":"https://api.github.com/users/airbnb/repos","events_url":"https://api.github.com/users/airbnb/events{/privacy}","received_events_url":"https://api.github.com/users/airbnb/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/airbnb/streamalert","description":"StreamAlert is a serverless, realtime data analysis framework which empowers you to ingest, analyze, and alert on data from any environment, using datasources and alerting logic you define.","fork":false,"url":"https://api.github.com/repos/airbnb/streamalert","forks_url":"https://api.github.com/repos/airbnb/streamalert/forks","keys_url":"https://api.github.com/repos/airbnb/streamalert/keys{/key_id}","collaborators_url":"https://api.github.com/repos/airbnb/streamalert/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/airbnb/streamalert/teams","hooks_url":"https://api.github.com/repos/airbnb/streamalert/hooks","issue_events_url":"https://api.github.com/repos/airbnb/streamalert/issues/events{/number}","events_url":"https://api.github.com/repos/airbnb/streamalert/events","assignees_url":"https://api.github.com/repos/airbnb/streamalert/assignees{/user}","branches_url":"https://api.github.com/repos/airbnb/streamalert/branches{/branch}","tags_url":"https://api.github.com/repos/airbnb/streamalert/tags","blobs_url":"https://api.github.com/repos/airbnb/streamalert/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/airbnb/streamalert/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/airbnb/streamalert/git/refs{/sha}","trees_url":"https://api.github.com/repos/airbnb/streamalert/git/trees{/sha}","statuses_url":"https://api.github.com/repos/airbnb/streamalert/statuses/{sha}","languages_url":"https://api.github.com/repos/airbnb/streamalert/languages","stargazers_url":"https://api.github.com/repos/airbnb/streamalert/stargazers","contributors_url":"https://api.github.com/repos/airbnb/streamalert/contributors","subscribers_url":"https://api.github.com/repos/airbnb/streamalert/subscribers","subscription_url":"https://api.github.com/repos/airbnb/streamalert/subscription","commits_url":"https://api.github.com/repos/airbnb/streamalert/commits{/sha}","git_commits_url":"https://api.github.com/repos/airbnb/streamalert/git/commits{/sha}","comments_url":"https://api.github.com/repos/airbnb/streamalert/comments{/number}","issue_comment_url":"https://api.github.com/repos/airbnb/streamalert/issues/comments{/number}","contents_url":"https://api.github.com/repos/airbnb/streamalert/contents/{+path}","compare_url":"https://api.github.com/repos/airbnb/streamalert/compare/{base}...{head}","merges_url":"https://api.github.com/repos/airbnb/streamalert/merges","archive_url":"https://api.github.com/repos/airbnb/streamalert/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/airbnb/streamalert/downloads","issues_url":"https://api.github.com/repos/airbnb/streamalert/issues{/number}","pulls_url":"https://api.github.com/repos/airbnb/streamalert/pulls{/number}","milestones_url":"https://api.github.com/repos/airbnb/streamalert/milestones{/number}","notifications_url":"https://api.github.com/repos/airbnb/streamalert/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/airbnb/streamalert/labels{/name}","releases_url":"https://api.github.com/repos/airbnb/streamalert/releases{/id}","deployments_url":"https://api.github.com/repos/airbnb/streamalert/deployments","created_at":"2017-01-22T01:10:56Z","updated_at":"2018-03-01T16:35:48Z","pushed_at":"2018-03-01T22:08:35Z","git_url":"git://github.com/airbnb/streamalert.git","ssh_url":"git@github.com:airbnb/streamalert.git","clone_url":"https://github.com/airbnb/streamalert.git","svn_url":"https://github.com/airbnb/streamalert","homepage":"","size":7779,"stargazers_count":1147,"watchers_count":1147,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":116,"mirror_url":null,"archived":false,"open_issues_count":59,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":116,"open_issues":59,"watchers":1147,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/airbnb/streamalert/pulls/612"},"html":{"href":"https://github.com/airbnb/streamalert/pull/612"},"issue":{"href":"https://api.github.com/repos/airbnb/streamalert/issues/612"},"comments":{"href":"https://api.github.com/repos/airbnb/streamalert/issues/612/comments"},"review_comments":{"href":"https://api.github.com/repos/airbnb/streamalert/pulls/612/comments"},"review_comment":{"href":"https://api.github.com/repos/airbnb/streamalert/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/airbnb/streamalert/pulls/612/commits"},"statuses":{"href":"https://api.github.com/repos/airbnb/streamalert/statuses/35e6cfb5e28ca3ae393e03722647dec9ea266084"}},"author_association":"CONTRIBUTOR","merged":true,"mergeable":null,"rebaseable":null,"mergeable_state":"unknown","merged_by":{"login":"austinbyers","id":3608925,"avatar_url":"https://avatars0.githubusercontent.com/u/3608925?v=4","gravatar_id":"","url":"https://api.github.com/users/austinbyers","html_url":"https://github.com/austinbyers","followers_url":"https://api.github.com/users/austinbyers/followers","following_url":"https://api.github.com/users/austinbyers/following{/other_user}","gists_url":"https://api.github.com/users/austinbyers/gists{/gist_id}","starred_url":"https://api.github.com/users/austinbyers/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/austinbyers/subscriptions","organizations_url":"https://api.github.com/users/austinbyers/orgs","repos_url":"https://api.github.com/users/austinbyers/repos","events_url":"https://api.github.com/users/austinbyers/events{/privacy}","received_events_url":"https://api.github.com/users/austinbyers/received_events","type":"User","site_admin":false},"comments":4,"review_comments":46,"maintainer_can_modify":false,"commits":10,"additions":520,"deletions":250,"changed_files":18}}
{"actor":{"display_login":"austinbyers"}}