{"action":"created","issue":{"url":"https://api.github.com/repos/Yelp/elastalert/issues/1887","repository_url":"https://api.github.com/repos/Yelp/elastalert","labels_url":"https://api.github.com/repos/Yelp/elastalert/issues/1887/labels{/name}","comments_url":"https://api.github.com/repos/Yelp/elastalert/issues/1887/comments","events_url":"https://api.github.com/repos/Yelp/elastalert/issues/1887/events","html_url":"https://github.com/Yelp/elastalert/issues/1887","id":357153191,"node_id":"MDU6SXNzdWUzNTcxNTMxOTE=","number":1887,"title":"Cardinality count issue","user":{"login":"Yv-o-rL","id":42407721,"node_id":"MDQ6VXNlcjQyNDA3NzIx","avatar_url":"https://avatars0.githubusercontent.com/u/42407721?v=4","gravatar_id":"","url":"https://api.github.com/users/Yv-o-rL","html_url":"https://github.com/Yv-o-rL","followers_url":"https://api.github.com/users/Yv-o-rL/followers","following_url":"https://api.github.com/users/Yv-o-rL/following{/other_user}","gists_url":"https://api.github.com/users/Yv-o-rL/gists{/gist_id}","starred_url":"https://api.github.com/users/Yv-o-rL/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Yv-o-rL/subscriptions","organizations_url":"https://api.github.com/users/Yv-o-rL/orgs","repos_url":"https://api.github.com/users/Yv-o-rL/repos","events_url":"https://api.github.com/users/Yv-o-rL/events{/privacy}","received_events_url":"https://api.github.com/users/Yv-o-rL/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-09-05T09:55:49Z","updated_at":"2018-09-07T22:18:27Z","closed_at":null,"author_association":"NONE","body":"Hi!\r\n\r\nI'm trying to set up an alert for domains where unique IP addresses surpass a limit within 4 hours. Since there are several sites on a node, I need to group the result per domain. The main config is set up to run every 30 minutes and has a 60 minute buffer time. I intend to lower the \"run_time\" setting to every 10 minutes because some alerts have to pop up ASAP.\r\nThe configuration for the alert in question:\r\n\r\n```\r\nindex: XXXXXX-%Y.%m.%d\r\nname: XXXXXX\r\ntype: cardinality\r\ncardinality_field: \"ip_address\"\r\nmax_cardinality: 10000\r\ntimeframe:\r\n  hours: 4\r\nfilter:\r\n- query_string:\r\n   query: \"XXXXXX\"\r\nquery_key: \"domain\"\r\nsummary_table_fields: [node,domain]\r\n\r\nalert:\r\n- \"slack\"\r\nalert_text_type: aggregation_summary_only\r\nslack:\r\nslack_webhook_url: \"XXXXXX\"\r\n```\r\n\r\nI thought this would result in a query every 30 minutes for the past 4 hours and send me the grouped results with the match count. However, I don't get all the matched items (compared to a Kibana request), and the count is only showing \"1\":\r\n\r\n> Aggregation resulted in the following data for summary_table_fields ==> ['node', 'domain', 'count']:\r\n> +-------+--------------------+-------+\r\n> |  node   |         domain         | count |\r\n> +====+============+====+\r\n> | XXXXX | www.XXXXXXX.io |     1     |\r\n> +-------+--------------------+-------+\r\n> \r\n> Aggregation resulted in the following data for summary_table_fields ==> ['node', 'domain', 'count']:\r\n> +-------+--------------------+-------+\r\n> |  node   |         domain         | count |\r\n> +====+============+====+\r\n> | XXXXX | www.YYYYYYYY.io |     1     |\r\n> +-------+--------------------+-------+\r\n\r\nMy intention would be to get the list of the domains where the unique count of the \"ip_address\" in the past 4 hours surpassed the maximum cardinality. The list should contain every affected domain and sent every hour. Reoccurring items don't cause any issue for me.\r\n\r\nI really appreciate your help!\r\nThank you!\r\nYvorl"},"comment":{"url":"https://api.github.com/repos/Yelp/elastalert/issues/comments/419580999","html_url":"https://github.com/Yelp/elastalert/issues/1887#issuecomment-419580999","issue_url":"https://api.github.com/repos/Yelp/elastalert/issues/1887","id":419580999,"node_id":"MDEyOklzc3VlQ29tbWVudDQxOTU4MDk5OQ==","user":{"login":"Qmando","id":3246990,"node_id":"MDQ6VXNlcjMyNDY5OTA=","avatar_url":"https://avatars3.githubusercontent.com/u/3246990?v=4","gravatar_id":"","url":"https://api.github.com/users/Qmando","html_url":"https://github.com/Qmando","followers_url":"https://api.github.com/users/Qmando/followers","following_url":"https://api.github.com/users/Qmando/following{/other_user}","gists_url":"https://api.github.com/users/Qmando/gists{/gist_id}","starred_url":"https://api.github.com/users/Qmando/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Qmando/subscriptions","organizations_url":"https://api.github.com/users/Qmando/orgs","repos_url":"https://api.github.com/users/Qmando/repos","events_url":"https://api.github.com/users/Qmando/events{/privacy}","received_events_url":"https://api.github.com/users/Qmando/received_events","type":"User","site_admin":false},"created_at":"2018-09-07T22:18:27Z","updated_at":"2018-09-07T22:18:27Z","author_association":"MEMBER","body":"You don't need a high buffer time. The default is probably fine. It only has to be larger than `run_every`. Whether it's 15 minutes or 30 mintues or 4 hours, it generally won't affect the outcome.\r\n\r\nI'm very confused as to why you are getting things on the 46th minute.. That shouldn't be happening, unless you have a very large run_every setting, or the query is taking a really long time to process. When you get `next alert at 2018-09-06 15:20` message, the first time elastalert runs after that time period it should send it. There could be a bug here too.\r\n\r\nRegarding why you get different results than Kibana: It could be because the time frames are not lining up as you expect. When you start elastalert, it will try to find the most recent query made for each rule and start from there, otherwise it starts from the present time. And then each four hour time frame only starts when the first value for some domain is hit, so some domains could alert until slightly later than what you might expect.\r\n\r\nOne interesting thing you could do if you want to try and debug the internals further:\r\n```\r\n--- a/elastalert/ruletypes.py\r\n+++ b/elastalert/ruletypes.py\r\n@@ -940,6 +940,7 @@ class CardinalityRule(RuleType):\r\n\r\n     def garbage_collect(self, timestamp):\r\n         \"\"\" Remove all occurrence data that is beyond the timeframe away \"\"\"\r\n+        print \"Cardinality values at\", timestamp, [key, len(values) for key, values in self.cardinality_cache.items()]\r\n```"}}
{"actor":{"display_login":"Qmando"}}