{"action":"created","issue":{"url":"https://api.github.com/repos/netdata/netdata/issues/2673","repository_url":"https://api.github.com/repos/netdata/netdata","labels_url":"https://api.github.com/repos/netdata/netdata/issues/2673/labels{/name}","comments_url":"https://api.github.com/repos/netdata/netdata/issues/2673/comments","events_url":"https://api.github.com/repos/netdata/netdata/issues/2673/events","html_url":"https://github.com/netdata/netdata/issues/2673","id":254862100,"node_id":"MDU6SXNzdWUyNTQ4NjIxMDA=","number":2673,"title":"Suppressing alerts programatically","user":{"login":"candlerb","id":44789,"node_id":"MDQ6VXNlcjQ0Nzg5","avatar_url":"https://avatars2.githubusercontent.com/u/44789?v=4","gravatar_id":"","url":"https://api.github.com/users/candlerb","html_url":"https://github.com/candlerb","followers_url":"https://api.github.com/users/candlerb/followers","following_url":"https://api.github.com/users/candlerb/following{/other_user}","gists_url":"https://api.github.com/users/candlerb/gists{/gist_id}","starred_url":"https://api.github.com/users/candlerb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/candlerb/subscriptions","organizations_url":"https://api.github.com/users/candlerb/orgs","repos_url":"https://api.github.com/users/candlerb/repos","events_url":"https://api.github.com/users/candlerb/events{/privacy}","received_events_url":"https://api.github.com/users/candlerb/received_events","type":"User","site_admin":false},"labels":[{"id":1065927064,"node_id":"MDU6TGFiZWwxMDY1OTI3MDY0","url":"https://api.github.com/repos/netdata/netdata/labels/area/health","name":"area/health","color":"0f34c6","default":false},{"id":1065918744,"node_id":"MDU6TGFiZWwxMDY1OTE4NzQ0","url":"https://api.github.com/repos/netdata/netdata/labels/feature%20request","name":"feature request","color":"9beff2","default":false}],"state":"open","locked":false,"assignee":{"login":"cakrit","id":43294513,"node_id":"MDQ6VXNlcjQzMjk0NTEz","avatar_url":"https://avatars0.githubusercontent.com/u/43294513?v=4","gravatar_id":"","url":"https://api.github.com/users/cakrit","html_url":"https://github.com/cakrit","followers_url":"https://api.github.com/users/cakrit/followers","following_url":"https://api.github.com/users/cakrit/following{/other_user}","gists_url":"https://api.github.com/users/cakrit/gists{/gist_id}","starred_url":"https://api.github.com/users/cakrit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cakrit/subscriptions","organizations_url":"https://api.github.com/users/cakrit/orgs","repos_url":"https://api.github.com/users/cakrit/repos","events_url":"https://api.github.com/users/cakrit/events{/privacy}","received_events_url":"https://api.github.com/users/cakrit/received_events","type":"User","site_admin":false},"assignees":[{"login":"cakrit","id":43294513,"node_id":"MDQ6VXNlcjQzMjk0NTEz","avatar_url":"https://avatars0.githubusercontent.com/u/43294513?v=4","gravatar_id":"","url":"https://api.github.com/users/cakrit","html_url":"https://github.com/cakrit","followers_url":"https://api.github.com/users/cakrit/followers","following_url":"https://api.github.com/users/cakrit/following{/other_user}","gists_url":"https://api.github.com/users/cakrit/gists{/gist_id}","starred_url":"https://api.github.com/users/cakrit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cakrit/subscriptions","organizations_url":"https://api.github.com/users/cakrit/orgs","repos_url":"https://api.github.com/users/cakrit/repos","events_url":"https://api.github.com/users/cakrit/events{/privacy}","received_events_url":"https://api.github.com/users/cakrit/received_events","type":"User","site_admin":false}],"milestone":null,"comments":3,"created_at":"2017-09-03T08:52:08Z","updated_at":"2018-11-23T20:12:54Z","closed_at":null,"author_association":"NONE","body":"I got alerts overnight of high disk utilisation and backlog from about 1am to 4am last night.\r\n\r\nIt was more than 1 hour later that I looked at them, but I think I have found the culprit: it's the monthly mdadm RAID scrub, which starts at 00:57 on the first Sunday of the month.\r\n\r\n~~~\r\nroot@wrn-mon2:~# cat /etc/cron.d/mdadm\r\n#\r\n# cron.d/mdadm -- schedules periodic redundancy checks of MD devices\r\n#\r\n# Copyright © martin f. krafft <madduck@madduck.net>\r\n# distributed under the terms of the Artistic Licence 2.0\r\n#\r\n\r\n# By default, run at 00:57 on every Sunday, but do nothing unless the day of\r\n# the month is less than or equal to 7. Thus, only run on the first Sunday of\r\n# each month. crontab(5) sucks, unfortunately, in this regard; therefore this\r\n# hack (see #380425).\r\n57 0 * * 0 root if [ -x /usr/share/mdadm/checkarray ] && [ $(date +\\%d) -le 7 ]; then /usr/share/mdadm/checkarray --cron --all --idle --quiet; fi\r\n~~~\r\n\r\nWhat I would like to do is have some way to automatically silence/suppress the alert here.  That is, modify the cronjob to:\r\n\r\n* temporarily silence specific alerts\r\n* do the checkarray\r\n* unsilence those alerts\r\n\r\nSo while disks.conf defines template `10min_disk_utilization`, I just want to disable specific alarm instances `disk_util.sda.10min_disk_utilization` and `disk_util.sdb.10min_disk_utilization`\r\n\r\nI [see](https://github.com/firehol/netdata/wiki/health-configuration-reference) that it should be possible:\r\n\r\n> netdata supports overriding templates with alarms. For example, when a template is defined for a set of charts, an alarm with exactly the same name attached to the same chart the template matches, will have higher precedence (i.e. netdata will use the alarm on this chart and prevent the template from being applied to it).\r\n\r\nAs far as I can see, the [health API](https://github.com/firehol/netdata/wiki/health-API-calls) only allows querying alarms, not controlling them, so I think I have to drop a file under `/opt/netdata/etc/netdata/health.d/` and then [send a SIGUSR2](https://github.com/firehol/netdata/wiki/health-configuration-reference).  Is that correct?\r\n\r\nWhat order are these files read in - e.g. alphabetically - and does it matter here?  That is, if netdata reads an 'alarm' definition before the corresponding 'template', will it still do the right thing?\r\n\r\nAnd what's the best way to make a \"null alarm\" which does nothing, to override an alarm from a template?\r\n\r\nThanks,\r\n\r\nBrian."},"comment":{"url":"https://api.github.com/repos/netdata/netdata/issues/comments/441312366","html_url":"https://github.com/netdata/netdata/issues/2673#issuecomment-441312366","issue_url":"https://api.github.com/repos/netdata/netdata/issues/2673","id":441312366,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0MTMxMjM2Ng==","user":{"login":"cakrit","id":43294513,"node_id":"MDQ6VXNlcjQzMjk0NTEz","avatar_url":"https://avatars0.githubusercontent.com/u/43294513?v=4","gravatar_id":"","url":"https://api.github.com/users/cakrit","html_url":"https://github.com/cakrit","followers_url":"https://api.github.com/users/cakrit/followers","following_url":"https://api.github.com/users/cakrit/following{/other_user}","gists_url":"https://api.github.com/users/cakrit/gists{/gist_id}","starred_url":"https://api.github.com/users/cakrit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cakrit/subscriptions","organizations_url":"https://api.github.com/users/cakrit/orgs","repos_url":"https://api.github.com/users/cakrit/repos","events_url":"https://api.github.com/users/cakrit/events{/privacy}","received_events_url":"https://api.github.com/users/cakrit/received_events","type":"User","site_admin":false},"created_at":"2018-11-23T20:12:54Z","updated_at":"2018-11-23T20:12:54Z","author_association":"CONTRIBUTOR","body":"Related to #3187 "}}
{"actor":{"display_login":"cakrit"}}