{"action":"created","issue":{"url":"https://api.github.com/repos/guardian/alerta/issues/189","repository_url":"https://api.github.com/repos/guardian/alerta","labels_url":"https://api.github.com/repos/guardian/alerta/issues/189/labels{/name}","comments_url":"https://api.github.com/repos/guardian/alerta/issues/189/comments","events_url":"https://api.github.com/repos/guardian/alerta/issues/189/events","html_url":"https://github.com/guardian/alerta/issues/189","id":151063935,"number":189,"title":"Strange behaviour with prometheus/alertmanager alert transitions","user":{"login":"hasso","id":1933532,"avatar_url":"https://avatars.githubusercontent.com/u/1933532?v=3","gravatar_id":"","url":"https://api.github.com/users/hasso","html_url":"https://github.com/hasso","followers_url":"https://api.github.com/users/hasso/followers","following_url":"https://api.github.com/users/hasso/following{/other_user}","gists_url":"https://api.github.com/users/hasso/gists{/gist_id}","starred_url":"https://api.github.com/users/hasso/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hasso/subscriptions","organizations_url":"https://api.github.com/users/hasso/orgs","repos_url":"https://api.github.com/users/hasso/repos","events_url":"https://api.github.com/users/hasso/events{/privacy}","received_events_url":"https://api.github.com/users/hasso/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/guardian/alerta/labels/question","name":"question","color":"cc317c"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2016-04-26T07:32:49Z","updated_at":"2016-08-19T07:37:46Z","closed_at":null,"body":"Here is important snippets from my configurations. Prometheus.rules:\r\n\r\n```\r\nALERT alertName\r\n    IF (metric > 1000) FOR 1m\r\n    LABELS { severity=\"critical\" }\r\n    ANNOTATIONS {\r\n    }\r\n\r\nALERT alertName\r\n    IF (metric > 100) FOR 1m\r\n    LABELS { severity=\"major\" }\r\n    ANNOTATIONS {\r\n    }\r\n```\r\n\r\nAlertmanager.yml:\r\n\r\n```\r\nroute:\r\n  receiver:        'alerta'\r\n  group_wait:      10s\r\n  group_interval:  10s\r\n  repeat_interval: 60s\r\n  group_by:        ['alertname', 'host']\r\n\r\ninhibit_rules:\r\n- source_match:\r\n    severity: 'critical'\r\n  target_match:\r\n    severity: 'major'\r\n  equal: ['alertname', 'host']\r\n\r\nreceivers:\r\n- name: 'alerta'\r\n  webhook_configs:\r\n  - url: 'http://localhost:8080/webhooks/prometheus?api-key=key'\r\n    send_resolved: true\r\n```\r\n\r\nFrom alertmanager seems that all is working as expected, but there is a strange behaviour in Alerta.\r\n * If metric rises over 100, the major alarm is firing in alertmanager. Alerta receives the alert correctly.\r\n * If metric rises over 1000 , the critical alarm is firing in alertmanager and the major is inhibited. Alerta updates the alert correctly.\r\n * If metric falls under 1000 again (but is still over 100) alertmnager removes the critical one and the major one is firing again. But Alerta gets confused for some reason - number of duplicates is zeroed, it's sometimes unclear whether its major or critical now, and duplicate updates (different ID, but same datetime etc) will appear under alert data."},"comment":{"url":"https://api.github.com/repos/guardian/alerta/issues/comments/240949704","html_url":"https://github.com/guardian/alerta/issues/189#issuecomment-240949704","issue_url":"https://api.github.com/repos/guardian/alerta/issues/189","id":240949704,"user":{"login":"satterly","id":615057,"avatar_url":"https://avatars.githubusercontent.com/u/615057?v=3","gravatar_id":"","url":"https://api.github.com/users/satterly","html_url":"https://github.com/satterly","followers_url":"https://api.github.com/users/satterly/followers","following_url":"https://api.github.com/users/satterly/following{/other_user}","gists_url":"https://api.github.com/users/satterly/gists{/gist_id}","starred_url":"https://api.github.com/users/satterly/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/satterly/subscriptions","organizations_url":"https://api.github.com/users/satterly/orgs","repos_url":"https://api.github.com/users/satterly/repos","events_url":"https://api.github.com/users/satterly/events{/privacy}","received_events_url":"https://api.github.com/users/satterly/received_events","type":"User","site_admin":false},"created_at":"2016-08-19T07:37:46Z","updated_at":"2016-08-19T07:37:46Z","body":"This issue could have been resolved by the fix #234 . Any chance you could update to the latest alerta server version to test and let me know?"}}
{"actor":{"display_login":"satterly"}}