{"action":"created","issue":{"url":"https://api.github.com/repos/airbnb/streamalert/issues/720","repository_url":"https://api.github.com/repos/airbnb/streamalert","labels_url":"https://api.github.com/repos/airbnb/streamalert/issues/720/labels{/name}","comments_url":"https://api.github.com/repos/airbnb/streamalert/issues/720/comments","events_url":"https://api.github.com/repos/airbnb/streamalert/issues/720/events","html_url":"https://github.com/airbnb/streamalert/pull/720","id":319740706,"number":720,"title":"Support merging on nested and nonexistent keys","user":{"login":"austinbyers","id":3608925,"avatar_url":"https://avatars0.githubusercontent.com/u/3608925?v=4","gravatar_id":"","url":"https://api.github.com/users/austinbyers","html_url":"https://github.com/austinbyers","followers_url":"https://api.github.com/users/austinbyers/followers","following_url":"https://api.github.com/users/austinbyers/following{/other_user}","gists_url":"https://api.github.com/users/austinbyers/gists{/gist_id}","starred_url":"https://api.github.com/users/austinbyers/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/austinbyers/subscriptions","organizations_url":"https://api.github.com/users/austinbyers/orgs","repos_url":"https://api.github.com/users/austinbyers/repos","events_url":"https://api.github.com/users/austinbyers/events{/privacy}","received_events_url":"https://api.github.com/users/austinbyers/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-05-02T23:03:07Z","updated_at":"2018-05-02T23:07:36Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/airbnb/streamalert/pulls/720","html_url":"https://github.com/airbnb/streamalert/pull/720","diff_url":"https://github.com/airbnb/streamalert/pull/720.diff","patch_url":"https://github.com/airbnb/streamalert/pull/720.patch"},"body":"to: @jacknagz or @ryandeivert \r\ncc: @fusionrace @mime-frame @jbussing @airbnb/streamalert-maintainers\r\nsize: medium\r\n\r\n## Background\r\n\r\nRight now, alert merging is only supported for top-level keys. Moreover, there is a bug where merged alerts cannot be created if the specified merge key doesn't exist in the record.\r\n\r\nThis PR enables merging for any key anywhere (or nowhere) in the record\r\n\r\n## Changes\r\n\r\n* The alert merger uses the existing `get_first_key` method to recursively search the entire record for the specified merge key. \r\n* A number of functions from the `helpers` folder (including `get_first_key`) have been migrated to `stream_alert/shared/utils.py` because they are imported by the StreamAlert core (Lambda functions). Rules which use these helpers will work exactly as before, because `helpers/base.py` imports them from `stream_alert/shared/utils.py`\r\n    * A consequence of this change is that `netaddr` is now required for the alert processor and alert merger, since they import `utils.py` which includes `netaddr` logic\r\n\r\n## Testing\r\n\r\n* Deploy to test account\r\n* Configure `binaryalert_yara_match` rule with `merge_by_keys=['nested']`\r\n* Send 3 BinaryAlert records of the form\r\n\r\n```python\r\n{\r\n    'NumMatchedRules': random.randint(1, 100),\r\n    'FileInfo': {\r\n        'there': None,\r\n        'nested': [1, 2, 'hi']\r\n    },\r\n    'MatchedRules': {\r\n        'Rule1': 'Stuff'\r\n    }\r\n}\r\n```\r\n\r\n* Received the following Slack message:\r\n\r\n![screen shot 2018-05-02 at 3 59 08 pm](https://user-images.githubusercontent.com/3608925/39553303-d5dcbb06-4e21-11e8-9f22-08be7cd507ee.png)\r\n\r\n* Testing again, this time eliding `'nested'` from the records:\r\n\r\n![screen shot 2018-05-02 at 4 01 40 pm](https://user-images.githubusercontent.com/3608925/39553355-20e1f8b4-4e22-11e8-8ff9-50e0f0d5a401.png)\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/airbnb/streamalert/issues/comments/386148553","html_url":"https://github.com/airbnb/streamalert/pull/720#issuecomment-386148553","issue_url":"https://api.github.com/repos/airbnb/streamalert/issues/720","id":386148553,"user":{"login":"coveralls","id":2354108,"avatar_url":"https://avatars1.githubusercontent.com/u/2354108?v=4","gravatar_id":"","url":"https://api.github.com/users/coveralls","html_url":"https://github.com/coveralls","followers_url":"https://api.github.com/users/coveralls/followers","following_url":"https://api.github.com/users/coveralls/following{/other_user}","gists_url":"https://api.github.com/users/coveralls/gists{/gist_id}","starred_url":"https://api.github.com/users/coveralls/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/coveralls/subscriptions","organizations_url":"https://api.github.com/users/coveralls/orgs","repos_url":"https://api.github.com/users/coveralls/repos","events_url":"https://api.github.com/users/coveralls/events{/privacy}","received_events_url":"https://api.github.com/users/coveralls/received_events","type":"User","site_admin":false},"created_at":"2018-05-02T23:07:36Z","updated_at":"2018-05-02T23:07:36Z","author_association":"NONE","body":"\n[![Coverage Status](https://coveralls.io/builds/16807719/badge)](https://coveralls.io/builds/16807719)\n\nCoverage increased (+0.1%) to 97.211% when pulling **6cddedc81a177d333f41a42ae68eee2d744488b3 on austin-better-merging** into **0578f3ce1b831b469c465b1f6446eb5693887b76 on master**.\n"}}
{"actor":{"display_login":"coveralls"}}