{"action":"created","issue":{"url":"https://api.github.com/repos/airbnb/streamalert/issues/821","repository_url":"https://api.github.com/repos/airbnb/streamalert","labels_url":"https://api.github.com/repos/airbnb/streamalert/issues/821/labels{/name}","comments_url":"https://api.github.com/repos/airbnb/streamalert/issues/821/comments","events_url":"https://api.github.com/repos/airbnb/streamalert/issues/821/events","html_url":"https://github.com/airbnb/streamalert/pull/821","id":365111150,"node_id":"MDExOlB1bGxSZXF1ZXN0MjE5MTM0NTY4","number":821,"title":"function split part 1, migration and refactor of payload and parsers","user":{"login":"ryandeivert","id":13773789,"node_id":"MDQ6VXNlcjEzNzczNzg5","avatar_url":"https://avatars3.githubusercontent.com/u/13773789?v=4","gravatar_id":"","url":"https://api.github.com/users/ryandeivert","html_url":"https://github.com/ryandeivert","followers_url":"https://api.github.com/users/ryandeivert/followers","following_url":"https://api.github.com/users/ryandeivert/following{/other_user}","gists_url":"https://api.github.com/users/ryandeivert/gists{/gist_id}","starred_url":"https://api.github.com/users/ryandeivert/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ryandeivert/subscriptions","organizations_url":"https://api.github.com/users/ryandeivert/orgs","repos_url":"https://api.github.com/users/ryandeivert/repos","events_url":"https://api.github.com/users/ryandeivert/events{/privacy}","received_events_url":"https://api.github.com/users/ryandeivert/received_events","type":"User","site_admin":false},"labels":[{"id":528833485,"node_id":"MDU6TGFiZWw1Mjg4MzM0ODU=","url":"https://api.github.com/repos/airbnb/streamalert/labels/improvement","name":"improvement","color":"5319e7","default":false},{"id":539574195,"node_id":"MDU6TGFiZWw1Mzk1NzQxOTU=","url":"https://api.github.com/repos/airbnb/streamalert/labels/log%20parsers","name":"log parsers","color":"006b75","default":false},{"id":1073398671,"node_id":"MDU6TGFiZWwxMDczMzk4Njcx","url":"https://api.github.com/repos/airbnb/streamalert/labels/xlarge","name":"xlarge","color":"b60205","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/airbnb/streamalert/milestones/9","html_url":"https://github.com/airbnb/streamalert/milestone/9","labels_url":"https://api.github.com/repos/airbnb/streamalert/milestones/9/labels","id":3698097,"node_id":"MDk6TWlsZXN0b25lMzY5ODA5Nw==","number":9,"title":"2.1.0","description":"","creator":{"login":"ryandeivert","id":13773789,"node_id":"MDQ6VXNlcjEzNzczNzg5","avatar_url":"https://avatars3.githubusercontent.com/u/13773789?v=4","gravatar_id":"","url":"https://api.github.com/users/ryandeivert","html_url":"https://github.com/ryandeivert","followers_url":"https://api.github.com/users/ryandeivert/followers","following_url":"https://api.github.com/users/ryandeivert/following{/other_user}","gists_url":"https://api.github.com/users/ryandeivert/gists{/gist_id}","starred_url":"https://api.github.com/users/ryandeivert/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ryandeivert/subscriptions","organizations_url":"https://api.github.com/users/ryandeivert/orgs","repos_url":"https://api.github.com/users/ryandeivert/repos","events_url":"https://api.github.com/users/ryandeivert/events{/privacy}","received_events_url":"https://api.github.com/users/ryandeivert/received_events","type":"User","site_admin":false},"open_issues":1,"closed_issues":0,"state":"open","created_at":"2018-09-29T08:07:16Z","updated_at":"2018-09-29T08:28:16Z","due_on":null,"closed_at":null},"comments":0,"created_at":"2018-09-29T08:28:16Z","updated_at":"2018-09-29T08:33:34Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/airbnb/streamalert/pulls/821","html_url":"https://github.com/airbnb/streamalert/pull/821","diff_url":"https://github.com/airbnb/streamalert/pull/821.diff","patch_url":"https://github.com/airbnb/streamalert/pull/821.patch"},"body":"to: @chunyong-lin , @strcrzy \r\ncc: @airbnb/streamalert-maintainers\r\n\r\n## Background\r\n\r\nIn the near future, StreamAlert's rule process will be split into 2 functions - one for classification and one for rule processing. This is the start of that migration.\r\n\r\n## Changes\r\n\r\n* Slight refactor to the `StreamPayload` class to better support loading of data from incoming records. This class now encapsulates all logic, instead of the logic being in multiple disparate places/tied to the 'classifier'.\r\n* Updates to the `S3Payload` class\r\n  * Adding support for various types of input. Now, any combination of the following will work:\r\n    * Newline-delimited json data (one json object per line)\r\n    * Formatted json data (not newline-delimited)\r\n    * Lines of raw data (non-json, aka csv)\r\n    * Gzipped data (of all above types)\r\n    * Non-gzipped data (of all above types)\r\n  * The above change now uses `jsonlines` to iterate over lines in newline-delimited json data. This is faster/more memory efficient then the alternative `ndjson` package, which loads the entire list of lines into memory and _then_ iterates over them. `jsonlines` does this upon read, so will use as little memory possible.\r\n  * Utilizing `tempfile.TemporaryFile` to handle reading/garbage collection better.\r\n* Complete overhaul of the `ParserBase` class to be far more clear and concise.\r\n  * This change includes updates to support never missing unparsed data. Prior to this, [data could be deleted](https://github.com/airbnb/streamalert/blob/master/stream_alert/rule_processor/parsers.py#L219-L226) during parsing and we would never actually know if this occurred.\r\n  * The parser will now track what records have successfully parsed with a given schema, along with ones that have failed.\r\n  * No successful parses with a schema now means a complete failure, which will skip to trying the next schema.\r\n* Removing usage of the `jsonpath_rw` library in parsing, since it is very slow. Replacing it with the simpler/faster `jmespath` library.\r\n* Updates to requirements.txt and various other fixes.\r\n* Fixes to various other bugs discovered during the rewrite.\r\n\r\n\r\n## Testing\r\n\r\n* Completely re-wrote unit tests for comprehensive coverage.\r\n\r\n### Notes for reviewers\r\n\r\nI apologize in advance that this is so large. This will be merged into an ongoing work-in-progress feature branch before merging into master."},"comment":{"url":"https://api.github.com/repos/airbnb/streamalert/issues/comments/425627542","html_url":"https://github.com/airbnb/streamalert/pull/821#issuecomment-425627542","issue_url":"https://api.github.com/repos/airbnb/streamalert/issues/821","id":425627542,"node_id":"MDEyOklzc3VlQ29tbWVudDQyNTYyNzU0Mg==","user":{"login":"coveralls","id":2354108,"node_id":"MDQ6VXNlcjIzNTQxMDg=","avatar_url":"https://avatars1.githubusercontent.com/u/2354108?v=4","gravatar_id":"","url":"https://api.github.com/users/coveralls","html_url":"https://github.com/coveralls","followers_url":"https://api.github.com/users/coveralls/followers","following_url":"https://api.github.com/users/coveralls/following{/other_user}","gists_url":"https://api.github.com/users/coveralls/gists{/gist_id}","starred_url":"https://api.github.com/users/coveralls/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/coveralls/subscriptions","organizations_url":"https://api.github.com/users/coveralls/orgs","repos_url":"https://api.github.com/users/coveralls/repos","events_url":"https://api.github.com/users/coveralls/events{/privacy}","received_events_url":"https://api.github.com/users/coveralls/received_events","type":"User","site_admin":false},"created_at":"2018-09-29T08:33:34Z","updated_at":"2018-09-29T08:33:34Z","author_association":"NONE","body":"\n[![Coverage Status](https://coveralls.io/builds/19260632/badge)](https://coveralls.io/builds/19260632)\n\nCoverage decreased (-0.1%) to 97.164% when pulling **f57e27d0437f0bcfdc81e0a6329418aee9cb300d on ryandeivert-func-split-pt-1** into **557fb3f604661cdd9bd36486cccc8ce3a34bd1f1 on ryandeivert-function-split**.\n"}}
{"actor":{"display_login":"coveralls"}}