{"action":"created","issue":{"url":"https://api.github.com/repos/airbnb/streamalert/issues/716","repository_url":"https://api.github.com/repos/airbnb/streamalert","labels_url":"https://api.github.com/repos/airbnb/streamalert/issues/716/labels{/name}","comments_url":"https://api.github.com/repos/airbnb/streamalert/issues/716/comments","events_url":"https://api.github.com/repos/airbnb/streamalert/issues/716/events","html_url":"https://github.com/airbnb/streamalert/pull/716","id":319410361,"number":716,"title":"updates to make athena client portable for use in other packages","user":{"login":"ryandeivert","id":13773789,"avatar_url":"https://avatars3.githubusercontent.com/u/13773789?v=4","gravatar_id":"","url":"https://api.github.com/users/ryandeivert","html_url":"https://github.com/ryandeivert","followers_url":"https://api.github.com/users/ryandeivert/followers","following_url":"https://api.github.com/users/ryandeivert/following{/other_user}","gists_url":"https://api.github.com/users/ryandeivert/gists{/gist_id}","starred_url":"https://api.github.com/users/ryandeivert/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ryandeivert/subscriptions","organizations_url":"https://api.github.com/users/ryandeivert/orgs","repos_url":"https://api.github.com/users/ryandeivert/repos","events_url":"https://api.github.com/users/ryandeivert/events{/privacy}","received_events_url":"https://api.github.com/users/ryandeivert/received_events","type":"User","site_admin":false},"labels":[{"id":911711961,"url":"https://api.github.com/repos/airbnb/streamalert/labels/rule%20staging","name":"rule staging","color":"e2c012","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/airbnb/streamalert/milestones/6","html_url":"https://github.com/airbnb/streamalert/milestone/6","labels_url":"https://api.github.com/repos/airbnb/streamalert/milestones/6/labels","id":3103420,"number":6,"title":"2.0.0","description":"","creator":{"login":"ryandeivert","id":13773789,"avatar_url":"https://avatars3.githubusercontent.com/u/13773789?v=4","gravatar_id":"","url":"https://api.github.com/users/ryandeivert","html_url":"https://github.com/ryandeivert","followers_url":"https://api.github.com/users/ryandeivert/followers","following_url":"https://api.github.com/users/ryandeivert/following{/other_user}","gists_url":"https://api.github.com/users/ryandeivert/gists{/gist_id}","starred_url":"https://api.github.com/users/ryandeivert/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ryandeivert/subscriptions","organizations_url":"https://api.github.com/users/ryandeivert/orgs","repos_url":"https://api.github.com/users/ryandeivert/repos","events_url":"https://api.github.com/users/ryandeivert/events{/privacy}","received_events_url":"https://api.github.com/users/ryandeivert/received_events","type":"User","site_admin":false},"open_issues":2,"closed_issues":71,"state":"open","created_at":"2018-02-09T18:18:51Z","updated_at":"2018-05-02T05:00:20Z","due_on":"2018-03-30T07:00:00Z","closed_at":null},"comments":0,"created_at":"2018-05-02T05:00:20Z","updated_at":"2018-05-02T05:05:36Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/airbnb/streamalert/pulls/716","html_url":"https://github.com/airbnb/streamalert/pull/716","diff_url":"https://github.com/airbnb/streamalert/pull/716.diff","patch_url":"https://github.com/airbnb/streamalert/pull/716.patch"},"body":"to: @austinbyers \r\ncc: @airbnb/streamalert-maintainers\r\nsize: large\r\n\r\n## Background\r\n\r\nThe upcoming rule promotion lambda function is going to have to perform various queries against athena in order to generate alert statistics, etc. Instead of writing all of logic again, we're adapting the `StreamAlertAthenaClient` to be more flexible/portable.\r\n\r\n## Note to Reviewers\r\n\r\n* The `StreamAlertSQSClient` was migrated from the `stream_alert/athena_partition_refresh/main.py` file to the `clients.py` file without many code changes at all. **Please** do not review this too heavily as this code is mostly not mine. The exception is some small updates and bug fixes that were made during the migration/testing phase. See majority of them in [this commit](https://github.com/airbnb/streamalert/commit/354e827d4def35590d41ee15f5153f54af946dd3).\r\n\r\n## Changes\r\n\r\n* Migrating the `StreamAlertAthenaClient` from `stream_alert/athena_partition_refresh/main.py` to `stream_alert/shared/athena.py`.\r\n\r\n### AthenaRefresher\r\n\r\nAdding an `AthenaRefresher` class that is now responsible for:\r\n* 1) polling sqs via the sqs client for new event notifications\r\n* 2) using the new `StreamAlertAthenaClient` to update the partitions in athena given the buckets/keys from sqs.\r\n* Removing a previous `add_partitions` method from the `StreamAlertAthenaClient` and adding this as a method to the `AthenaRefresher` class (since the logic embedded in this function was not related to athena directly ).\r\n* Breaking the `add_partitions` method into two functions: `_get_partitions_from_keys` and `_add_partitions`. This helped with testing quite a bit.\r\n\r\n### StreamAlertAthenaClient\r\n* The client now supports a number of new operations:\r\n  * The `query_result_paginator` returns a generator and allows for paginating over the results of a query that is being executed. Prior to this only one page could be retrieved for any query ran.\r\n  * Simple standard sql operations like dropping tables, showing partitions, etc are now implemented as methods to improve ease of use (see: `drop_table`, `drop_all_tables`, and `get_table_partitions`)\r\n  * New query operation that allows for executing and verifying the execution was successful (useful for things like dropping tables, etc where there is no actual results returned).\r\n* Moving the helper function of `_unique_values_from_query` to be a static method of this class.\r\n* Now catching `ClientErrors` that could be the result of a bad query string, etc.\r\n\r\n### CLI Changes\r\n\r\n* Updating the `stream_alert_cli/athena/handler.py` file to use the new athena client along with its new functionality.\r\n\r\n## Testing\r\n\r\n* Unit tests: migrating and updating some/adding new to cover all new code.\r\n* Deployed in testing account to ensure all systems go. Alerts and data are available in athena as expected.\r\n"},"comment":{"url":"https://api.github.com/repos/airbnb/streamalert/issues/comments/385865981","html_url":"https://github.com/airbnb/streamalert/pull/716#issuecomment-385865981","issue_url":"https://api.github.com/repos/airbnb/streamalert/issues/716","id":385865981,"user":{"login":"coveralls","id":2354108,"avatar_url":"https://avatars1.githubusercontent.com/u/2354108?v=4","gravatar_id":"","url":"https://api.github.com/users/coveralls","html_url":"https://github.com/coveralls","followers_url":"https://api.github.com/users/coveralls/followers","following_url":"https://api.github.com/users/coveralls/following{/other_user}","gists_url":"https://api.github.com/users/coveralls/gists{/gist_id}","starred_url":"https://api.github.com/users/coveralls/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/coveralls/subscriptions","organizations_url":"https://api.github.com/users/coveralls/orgs","repos_url":"https://api.github.com/users/coveralls/repos","events_url":"https://api.github.com/users/coveralls/events{/privacy}","received_events_url":"https://api.github.com/users/coveralls/received_events","type":"User","site_admin":false},"created_at":"2018-05-02T05:05:36Z","updated_at":"2018-05-02T05:05:36Z","author_association":"NONE","body":"\n[![Coverage Status](https://coveralls.io/builds/16790301/badge)](https://coveralls.io/builds/16790301)\n\nCoverage increased (+0.3%) to 97.092% when pulling **fccfa2a0834b9646289377b9e2cef9f10e2f25c6 on ryandeivert-athena-shared-client** into **d4ec8d6a4b783de91d1eb0e3b8517f5e2fb384d5 on master**.\n"}}
{"actor":{"display_login":"coveralls"}}