{"action":"created","issue":{"url":"https://api.github.com/repos/prometheus/alertmanager/issues/347","repository_url":"https://api.github.com/repos/prometheus/alertmanager","labels_url":"https://api.github.com/repos/prometheus/alertmanager/issues/347/labels{/name}","comments_url":"https://api.github.com/repos/prometheus/alertmanager/issues/347/comments","events_url":"https://api.github.com/repos/prometheus/alertmanager/issues/347/events","html_url":"https://github.com/prometheus/alertmanager/pull/347","id":154055940,"number":347,"title":"[WIP] check availability of templates when loading the config","user":{"login":"cherti","id":6452205,"avatar_url":"https://avatars.githubusercontent.com/u/6452205?v=3","gravatar_id":"","url":"https://api.github.com/users/cherti","html_url":"https://github.com/cherti","followers_url":"https://api.github.com/users/cherti/followers","following_url":"https://api.github.com/users/cherti/following{/other_user}","gists_url":"https://api.github.com/users/cherti/gists{/gist_id}","starred_url":"https://api.github.com/users/cherti/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cherti/subscriptions","organizations_url":"https://api.github.com/users/cherti/orgs","repos_url":"https://api.github.com/users/cherti/repos","events_url":"https://api.github.com/users/cherti/events{/privacy}","received_events_url":"https://api.github.com/users/cherti/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":14,"created_at":"2016-05-10T16:56:07Z","updated_at":"2016-05-10T22:44:32Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/prometheus/alertmanager/pulls/347","html_url":"https://github.com/prometheus/alertmanager/pull/347","diff_url":"https://github.com/prometheus/alertmanager/pull/347.diff","patch_url":"https://github.com/prometheus/alertmanager/pull/347.patch"},"body":"### Work in progress, do not merge yet!\r\n\r\nThis is proof-of-concept-code to be discussed with upstream.\r\nFixes #311 when finished and merged.\r\n@fabxc \r\n\r\n## Problem diagnosis\r\nWhen sending out alerts, Prometheus tries sending the alert and if that fails with an error, it tries again with exponential backoff. Unfortunately, an error that can cause this is, besides others, the use of a nonexistent template, which (at least for mail) results in AM flooding the receivers inbox by repeatedly trying (and succeeding) sending mails. This has primarily been tested for mail, but might occur as well for other notification types.\r\n\r\n## Basic idea for fixing\r\nThere are two options to fix this: \r\n1) fall back onto the default template when a template is not found\r\n2) Check upon loading of the config that all templates in use are actually found\r\n\r\nin the current structure, `1)` would require internal silencing of errors due to the mechanics of the used notification module when falling back onto the default, which could be done, but is highly unelegant.\r\nSo we went for `2)` and verify the integrity of the entire config set by checking it upon loading and make config-loading throw an error for inconsistent configurations.\r\n\r\n## State of this PR\r\nThis PR implements a general Proof-of-concept of fixing the issue mentioned above to be discussed with upstream, to find the optimal solution for this before carrying out work that might end up being unnecessary and wasting time. For this reason, currently only the `html`-field of the mail-configs is checked for invalid templates. For other notification methods the code will be fairly similar.\r\n\r\n## Further plans to be discussed besides the obvious\r\n\r\n* Currently the significant part of the fix lives directly in `main.go`, it might be a reasonable choice to move it into the `config`-package and just call that package from within `main.go`.\r\n\r\n\r\nFeedback is highly appreciated so that we can finish fixing this issue. :)\r\n\r\nShoutout to @joker234 with whom I debugged and fixed this!\r\n"},"comment":{"url":"https://api.github.com/repos/prometheus/alertmanager/issues/comments/218314037","html_url":"https://github.com/prometheus/alertmanager/pull/347#issuecomment-218314037","issue_url":"https://api.github.com/repos/prometheus/alertmanager/issues/347","id":218314037,"user":{"login":"brian-brazil","id":7115638,"avatar_url":"https://avatars.githubusercontent.com/u/7115638?v=3","gravatar_id":"","url":"https://api.github.com/users/brian-brazil","html_url":"https://github.com/brian-brazil","followers_url":"https://api.github.com/users/brian-brazil/followers","following_url":"https://api.github.com/users/brian-brazil/following{/other_user}","gists_url":"https://api.github.com/users/brian-brazil/gists{/gist_id}","starred_url":"https://api.github.com/users/brian-brazil/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brian-brazil/subscriptions","organizations_url":"https://api.github.com/users/brian-brazil/orgs","repos_url":"https://api.github.com/users/brian-brazil/repos","events_url":"https://api.github.com/users/brian-brazil/events{/privacy}","received_events_url":"https://api.github.com/users/brian-brazil/received_events","type":"User","site_admin":false},"created_at":"2016-05-10T22:44:32Z","updated_at":"2016-05-10T22:44:32Z","body":"How are you getting the name of the template?"}}