{"action":"created","issue":{"url":"https://api.github.com/repos/prometheus/alertmanager/issues/1546","repository_url":"https://api.github.com/repos/prometheus/alertmanager","labels_url":"https://api.github.com/repos/prometheus/alertmanager/issues/1546/labels{/name}","comments_url":"https://api.github.com/repos/prometheus/alertmanager/issues/1546/comments","events_url":"https://api.github.com/repos/prometheus/alertmanager/issues/1546/events","html_url":"https://github.com/prometheus/alertmanager/issues/1546","id":359294891,"node_id":"MDU6SXNzdWUzNTkyOTQ4OTE=","number":1546,"title":"Alert manager goes from Firing->Resolved->Firing","user":{"login":"justinjk007","id":12916795,"node_id":"MDQ6VXNlcjEyOTE2Nzk1","avatar_url":"https://avatars2.githubusercontent.com/u/12916795?v=4","gravatar_id":"","url":"https://api.github.com/users/justinjk007","html_url":"https://github.com/justinjk007","followers_url":"https://api.github.com/users/justinjk007/followers","following_url":"https://api.github.com/users/justinjk007/following{/other_user}","gists_url":"https://api.github.com/users/justinjk007/gists{/gist_id}","starred_url":"https://api.github.com/users/justinjk007/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/justinjk007/subscriptions","organizations_url":"https://api.github.com/users/justinjk007/orgs","repos_url":"https://api.github.com/users/justinjk007/repos","events_url":"https://api.github.com/users/justinjk007/events{/privacy}","received_events_url":"https://api.github.com/users/justinjk007/received_events","type":"User","site_admin":false},"labels":[{"id":365566474,"node_id":"MDU6TGFiZWwzNjU1NjY0NzQ=","url":"https://api.github.com/repos/prometheus/alertmanager/labels/component/notify","name":"component/notify","color":"0052cc","default":false},{"id":1012399309,"node_id":"MDU6TGFiZWwxMDEyMzk5MzA5","url":"https://api.github.com/repos/prometheus/alertmanager/labels/kind/more-info-needed","name":"kind/more-info-needed","color":"c7def8","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-09-12T02:47:20Z","updated_at":"2018-09-17T15:31:41Z","closed_at":null,"author_association":"NONE","body":"**What did you do?**\r\n\r\nStarted the prometheus followed by the alert manger. \r\n\r\n**What did you expect to see?**\r\n\r\nExpect to see alerts go off once everyday unless they are resolved for some time and then became firing again.\r\n\r\n**What did you see instead? Under which circumstances?**\r\n\r\nI only use alert manager(for now) to monitor if machine is down/up and if any disk locations I monitor goes above threshold . At first I only had disk usage being alerted. At which point, I was getting notifications all the time during the day, my disk space issue is resolved , but then couple of minutes later, it became firing again. I thought this is because the usage was very close to the threshold level.  And it might have resolved for some time. But then I started monitoring machine up/down status as well, I started getting alerts saying machines being down resolved. But these are machines that does not even have the prometheus exporter running. So they were never up/resolved. \r\n\r\nSo the problem is alert manager goes from  Firing->Resolved->Firing often. I think it has to do with alert manager itself.\r\n\r\n**Environment**\r\n\r\n* System information:\r\n\r\n\tinsert output of `uname -srm` here: `Linux 4.15.0-20-generic x86_64`\r\n\r\n* Alertmanager version:\r\n\r\n``` sh\r\nalertmanager, version 0.15.2 (branch: HEAD, revision: d19fae3bae451940b8470abb680cfdd59bfa7cfa)\r\n  build user:       root@3101e5b68a55\r\n  build date:       20180814-10:53:39\r\n  go version:       go1.10.3\r\n```\r\n\r\n* Prometheus version:\r\n```\r\nprometheus, version 2.2.1 (branch: HEAD, revision: bc6058c81272a8d938c05e75607371284236aadc)\r\n  build user:       root@149e5b3f0829\r\n  build date:       20180314-14:15:45\r\n  go version:       go1.10\r\n```\r\n\r\n* Alertmanager configuration file:\r\n```yml\r\nglobal:\r\n  slack_api_url: 'https://hooks.slack.com/services/notTheRealDealBoi'\r\ntemplates:\r\n- '/etc/alertmanager/template/*.tmpl'\r\n- './slack_title.tmpl' # This is the custom template we wrote\r\n\r\n# The root route on which each incoming alert enters.\r\nroute:\r\n  # The labels by which incoming alerts are grouped together. For example,\r\n  # multiple alerts coming in for cluster=A and alertname=LatencyHigh would\r\n  # be batched into a single group.\r\n  group_by: ['instance', 'alertname']\r\n\r\n  # When a new group of alerts is created by an incoming alert, wait at\r\n  # least 'group_wait' to send the initial notification.\r\n  # This way ensures that you get multiple alerts for the same group that start\r\n  # firing shortly after another are batched together on the first\r\n  # notification.\r\n  #group_wait: 30s\r\n\r\n  # When the first notification was sent, wait 'group_interval' to send a batch\r\n  # of new alerts that started firing for that group.\r\n  group_interval: 1m\r\n\r\n  # If an alert has successfully been sent, wait 'repeat_interval' to\r\n  # resend them.\r\n  repeat_interval: 24h\r\n\r\n  receiver: machine-status-slack-notifications # Default receiver\r\n\r\n  routes:\r\n  - match:\r\n      alertname: gsa-fs-threshold\r\n    receiver: gsa-disk-usage-slack-notifications\r\n  - match:\r\n      alertname: fs-threshold\r\n    receiver: disk-usage-slack-notifications\r\n  - match:\r\n      alertname: machine-down-in-canlab\r\n    receiver: machine-status-slack-notifications\r\n  - match:\r\n      alertname: machine-down-not-in-canlab\r\n    receiver: machine-status-slack-notifications\r\n\r\nreceivers:\r\n- name: 'disk-usage-slack-notifications'\r\n  slack_configs:\r\n   - channel: '#prometheus_alerts'\r\n     title: '{{ template \"slack.mytitle.disk.text\" . }}'\r\n     title_link: 'http://foo.different.comcom:3000/dashboard/db/machine-stats-single-page-build-disk-test?var-machine={{ .GroupLabels.instance }}'\r\n     text: '{{ template \"slack.desc.disk.text\" . }}'\r\n     send_resolved: true\r\n- name: 'gsa-disk-usage-slack-notifications'\r\n  slack_configs:\r\n   - channel: '#prometheus_alerts'\r\n     title: '{{ template \"slack.mytitle.gsa.text\" . }}'\r\n     title_link: 'http://foo.different.comcom:3000/dashboard/db/machine-stats-single-page-build-disk-test?var-machine={{ .GroupLabels.instance }}'\r\n     text: '{{ template \"slack.desc.gsa.text\" . }}'\r\n     send_resolved: true\r\n- name: 'machine-status-slack-notifications'\r\n  slack_configs:\r\n   - channel: '#prometheus_alerts'\r\n     title: '{{ template \"slack.mytitle.text\" . }}'\r\n     title_link: 'http://foo.different.com:3000/dashboard/db/machine-stats-single-page-build-disk-test?var-machine={{ .GroupLabels.instance }}'\r\n     text: '{{ template \"slack.desc.text\" . }}'\r\n     send_resolved: true\r\n  \r\n\r\n# Inhibition rules allow to mute a set of alerts given that another alert is\r\n# firing.\r\n# We use this to mute any warning-level notifications if the same alert is\r\n# already critical.\r\n#inhibit_rules:\r\n#- source_match:\r\n#    severity: 'critical'\r\n#  target_match:\r\n#    severity: 'warning'\r\n  # Apply inhibition if the alertname is the same.\r\n  #  equal: ['alertname']\r\n```\r\n\r\n* Prometheus configuration file:\r\n```yml\r\ni# my global config\r\nglobal:\r\n  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.\r\n  evaluation_interval: 5m # Evaluate rules every 15 seconds. The default is every 1 minute.\r\n  # scrape_timeout is set to the global default (10s).\r\n\r\n# Alertmanager configuration\r\nalerting:\r\n  alertmanagers:\r\n  - static_configs:\r\n    - targets:\r\n      - localhost:9093\r\n\r\n# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.\r\nrule_files:\r\n  - \"alerts.yml\"\r\n  # - \"second_rules.yml\"\r\n\r\n# A scrape configuration containing exactly one endpoint to scrape:\r\n# Here it's Prometheus itself.\r\nscrape_configs:\r\n  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.\r\n  #\r\n\r\n  # metrics_path defaults to '/metrics'\r\n  # scheme defaults to 'http'.\r\n\r\n  - job_name: 'prometheus'\r\n    static_configs:\r\n      - targets: ['localhost:9090']\r\n\r\n  - job_name: 'general'\r\n    static_configs:\r\n      - targets: ['lorem.ipsum.com:1337']\r\n      - targets: ['bar.foo.bar:1337']\r\n      - targets: ['foo.bar.com:1337']\r\n      - targets: ['localhost:1337']\r\n\r\n```\r\n\r\n* Logs:\r\n```\r\nAlertmanager logs.\r\n\r\nlevel=info ts=2018-09-07T20:22:14.013605369Z caller=main.go:174 msg=\"Starting Alertmanager\" version=\"(version=0.15.2, branch=HEAD, revision=d19fae3bae451940b8470abb680cfdd59bfa7cfa)\"\r\nlevel=info ts=2018-09-07T20:22:14.013710082Z caller=main.go:175 build_context=\"(go=go1.10.3, user=root@3101e5b68a55, date=20180814-10:53:39)\"\r\nlevel=info ts=2018-09-07T20:22:14.047249055Z caller=cluster.go:155 component=cluster msg=\"setting advertise address explicitly\" addr=10.31.9.74 port=9094\r\nlevel=info ts=2018-09-07T20:22:14.051230665Z caller=cluster.go:570 component=cluster msg=\"Waiting for gossip to settle...\" interval=2s\r\nlevel=info ts=2018-09-07T20:22:14.051244997Z caller=main.go:322 msg=\"Loading configuration file\" file=simple_new.yml\r\nlevel=info ts=2018-09-07T20:22:14.059479976Z caller=main.go:398 msg=Listening address=:9093\r\nlevel=info ts=2018-09-07T20:22:16.053277255Z caller=cluster.go:595 component=cluster msg=\"gossip not settled\" polls=0 before=0 now=1 elapsed=2.00019006s\r\nlevel=info ts=2018-09-07T20:22:24.058948881Z caller=cluster.go:587 component=cluster msg=\"gossip settled; proceeding\" elapsed=10.005876805s\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/prometheus/alertmanager/issues/comments/422063520","html_url":"https://github.com/prometheus/alertmanager/issues/1546#issuecomment-422063520","issue_url":"https://api.github.com/repos/prometheus/alertmanager/issues/1546","id":422063520,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMjA2MzUyMA==","user":{"login":"justinjk007","id":12916795,"node_id":"MDQ6VXNlcjEyOTE2Nzk1","avatar_url":"https://avatars2.githubusercontent.com/u/12916795?v=4","gravatar_id":"","url":"https://api.github.com/users/justinjk007","html_url":"https://github.com/justinjk007","followers_url":"https://api.github.com/users/justinjk007/followers","following_url":"https://api.github.com/users/justinjk007/following{/other_user}","gists_url":"https://api.github.com/users/justinjk007/gists{/gist_id}","starred_url":"https://api.github.com/users/justinjk007/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/justinjk007/subscriptions","organizations_url":"https://api.github.com/users/justinjk007/orgs","repos_url":"https://api.github.com/users/justinjk007/repos","events_url":"https://api.github.com/users/justinjk007/events{/privacy}","received_events_url":"https://api.github.com/users/justinjk007/received_events","type":"User","site_admin":false},"created_at":"2018-09-17T15:31:41Z","updated_at":"2018-09-17T15:31:41Z","author_association":"NONE","body":"Ok, sorry for the late reply, I will try that today."}}
{"actor":{"display_login":"justinjk007"}}