{"action":"created","issue":{"url":"https://api.github.com/repos/bosun-monitor/bosun/issues/1690","repository_url":"https://api.github.com/repos/bosun-monitor/bosun","labels_url":"https://api.github.com/repos/bosun-monitor/bosun/issues/1690/labels{/name}","comments_url":"https://api.github.com/repos/bosun-monitor/bosun/issues/1690/comments","events_url":"https://api.github.com/repos/bosun-monitor/bosun/issues/1690/events","html_url":"https://github.com/bosun-monitor/bosun/issues/1690","id":145118320,"number":1690,"title":"Bosun sending notifications for closed and inactive alerts","user":{"login":"angadsingh","id":1623026,"avatar_url":"https://avatars.githubusercontent.com/u/1623026?v=3","gravatar_id":"","url":"https://api.github.com/users/angadsingh","html_url":"https://github.com/angadsingh","followers_url":"https://api.github.com/users/angadsingh/followers","following_url":"https://api.github.com/users/angadsingh/following{/other_user}","gists_url":"https://api.github.com/users/angadsingh/gists{/gist_id}","starred_url":"https://api.github.com/users/angadsingh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/angadsingh/subscriptions","organizations_url":"https://api.github.com/users/angadsingh/orgs","repos_url":"https://api.github.com/users/angadsingh/repos","events_url":"https://api.github.com/users/angadsingh/events{/privacy}","received_events_url":"https://api.github.com/users/angadsingh/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/bosun-monitor/bosun/labels/bosun","name":"bosun","color":"fef2c0"},{"url":"https://api.github.com/repos/bosun-monitor/bosun/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":{"login":"captncraig","id":192540,"avatar_url":"https://avatars.githubusercontent.com/u/192540?v=3","gravatar_id":"","url":"https://api.github.com/users/captncraig","html_url":"https://github.com/captncraig","followers_url":"https://api.github.com/users/captncraig/followers","following_url":"https://api.github.com/users/captncraig/following{/other_user}","gists_url":"https://api.github.com/users/captncraig/gists{/gist_id}","starred_url":"https://api.github.com/users/captncraig/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/captncraig/subscriptions","organizations_url":"https://api.github.com/users/captncraig/orgs","repos_url":"https://api.github.com/users/captncraig/repos","events_url":"https://api.github.com/users/captncraig/events{/privacy}","received_events_url":"https://api.github.com/users/captncraig/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/bosun-monitor/bosun/milestones/1","html_url":"https://github.com/bosun-monitor/bosun/milestones/0.5.0","labels_url":"https://api.github.com/repos/bosun-monitor/bosun/milestones/1/labels","id":1646178,"number":1,"title":"0.5.0","description":"Things to do before cutting final release.","creator":{"login":"kylebrandt","id":1692624,"avatar_url":"https://avatars.githubusercontent.com/u/1692624?v=3","gravatar_id":"","url":"https://api.github.com/users/kylebrandt","html_url":"https://github.com/kylebrandt","followers_url":"https://api.github.com/users/kylebrandt/followers","following_url":"https://api.github.com/users/kylebrandt/following{/other_user}","gists_url":"https://api.github.com/users/kylebrandt/gists{/gist_id}","starred_url":"https://api.github.com/users/kylebrandt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kylebrandt/subscriptions","organizations_url":"https://api.github.com/users/kylebrandt/orgs","repos_url":"https://api.github.com/users/kylebrandt/repos","events_url":"https://api.github.com/users/kylebrandt/events{/privacy}","received_events_url":"https://api.github.com/users/kylebrandt/received_events","type":"User","site_admin":false},"open_issues":5,"closed_issues":3,"state":"open","created_at":"2016-03-15T17:23:23Z","updated_at":"2016-04-14T13:33:13Z","due_on":null,"closed_at":null},"comments":12,"created_at":"2016-04-01T08:21:48Z","updated_at":"2016-04-19T07:53:36Z","closed_at":null,"body":"We have a very simple rule file, with 3 `notifications` (http post to PD and slack, and email) and a bunch of `alert` rules which trigger them. We are facing a weird issue wherein, the following happens:\r\n\r\n* `alert` triggers, sends `notifications`\r\n* a human `acks` the alert\r\n* human solves problem, alert becomes `inactive`\r\n* human `closes` the alert\r\n* **notification still keeps triggering (alert is no where to be seen in the bosun UI/api) - forever!**\r\n\r\nto explain it through logs, quite literally this is what we're seeing:\r\n\r\n> 2016/04/01 07:56:37 info: check.go:513: check alert masked.masked.write.rate.too.low start\r\n> 2016/04/01 07:26:38 info: check.go:537: check alert masked.masked.write.rate.too.low done (1.378029647s): **0 crits, 0 warns, 0 unevaluated, 0 unknown**\r\n> 2016/04/01 07:26:38 info: alertRunner.go:55: runHistory on masked.masked.write.rate.too.low took 54.852815ms\r\n> 2016/04/01 07:26:39 info: search.go:205: Backing up last data to redis\r\n> 2016/04/01 07:28:20 info: notify.go:57: [bosun] critical: component xyz write rate too low: 0.00 records/minute in {adaptor=masked-masked-masked,colo=xyz,stream=writeAttributeToKafka}\r\n> 2016/04/01 07:28:20 info: notify.go:57: [bosun] critical: component xyz write rate too low: 0.00 records/minute in {adaptor=masked-masked-masked,colo=xyz,stream=writeActivityToKafka}\r\n> 2016/04/01 07:28:20 info: notify.go:57: [bosun] critical: component xyz write rate too low: 0.00 records/minute in {adaptor=masked-masked-masked,colo=xyz,stream=writeAttributeToKafka}\r\n> 2016/04/01 07:28:20 info: notify.go:57: [bosun] critical: component xyz write rate too low: 0.00 records/minute in {adaptor=masked-masked-masked,colo=xyz,stream=writeActivityToKafka}\r\n> 2016/04/01 07:28:20 info: notify.go:57: [bosun] critical: component xyz write rate too low: 0.00 records/minute in {adaptor=masked-masked-masked,colo=xyz,stream=writeAttributeToKafka}\r\n> 2016/04/01 07:28:20 info: notify.go:57: [bosun] critical: component xyz write rate too low: 0.00 records/minute in {adaptor=masked-masked-masked,colo=xyz,stream=writeActivityToKafka}\r\n> 2016/04/01 07:28:20 info: notify.go:115: relayed alert masked.masked.write.rate.too.low{adaptor=masked-masked-masked,colo=xyz,stream=writeAttributeToKafka} to [alert-group@xyz.com] sucessfully. Subject: 148 bytes. Body: 3500 bytes.\r\n> 2016/04/01 07:28:20 info: notify.go:115: relayed alert masked.masked.write.rate.too.low{adaptor=masked-masked-masked,colo=xyz,stream=writeActivityToKafka} to [alert-group@xyz.com] sucessfully. Subject: 147 bytes. Body: 3497 bytes.\r\n> 2016/04/01 07:28:20 info: notify.go:80: post notification successful for alert masked.masked.write.rate.too.low{adaptor=masked-masked-masked,colo=xyz,stream=writeAttributeToKafka}. Response code 200.\r\n> 2016/04/01 07:28:20 info: notify.go:80: post notification successful for alert masked.masked.write.rate.too.low{adaptor=masked-masked-masked,colo=xyz,stream=writeActivityToKafka}. Response code 200.\r\n> 2016/04/01 07:28:20 info: notify.go:80: post notification successful for alert masked.masked.write.rate.too.low{adaptor=masked-masked-masked,colo=xyz,stream=writeAttributeToKafka}. Response code 200.\r\n> 2016/04/01 07:28:20 info: notify.go:80: post notification successful for alert masked.masked.write.rate.too.low{adaptor=masked-masked-masked,colo=xyz,stream=writeActivityToKafka}. Response code 200."},"comment":{"url":"https://api.github.com/repos/bosun-monitor/bosun/issues/comments/211781117","html_url":"https://github.com/bosun-monitor/bosun/issues/1690#issuecomment-211781117","issue_url":"https://api.github.com/repos/bosun-monitor/bosun/issues/1690","id":211781117,"user":{"login":"lukaspustina","id":398967,"avatar_url":"https://avatars.githubusercontent.com/u/398967?v=3","gravatar_id":"","url":"https://api.github.com/users/lukaspustina","html_url":"https://github.com/lukaspustina","followers_url":"https://api.github.com/users/lukaspustina/followers","following_url":"https://api.github.com/users/lukaspustina/following{/other_user}","gists_url":"https://api.github.com/users/lukaspustina/gists{/gist_id}","starred_url":"https://api.github.com/users/lukaspustina/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lukaspustina/subscriptions","organizations_url":"https://api.github.com/users/lukaspustina/orgs","repos_url":"https://api.github.com/users/lukaspustina/repos","events_url":"https://api.github.com/users/lukaspustina/events{/privacy}","received_events_url":"https://api.github.com/users/lukaspustina/received_events","type":"User","site_admin":false},"created_at":"2016-04-19T07:53:36Z","updated_at":"2016-04-19T07:53:36Z","body":"After upgrading several of our Bosun installations including the migration of the old state file to ledis, we always observed the same behavior: For a few days we see repeated notifications for already acked or even closed alarms. This behavior fades away after a few days."}}