{"action":"created","issue":{"url":"https://api.github.com/repos/prometheus/alertmanager/issues/1550","repository_url":"https://api.github.com/repos/prometheus/alertmanager","labels_url":"https://api.github.com/repos/prometheus/alertmanager/issues/1550/labels{/name}","comments_url":"https://api.github.com/repos/prometheus/alertmanager/issues/1550/comments","events_url":"https://api.github.com/repos/prometheus/alertmanager/issues/1550/events","html_url":"https://github.com/prometheus/alertmanager/issues/1550","id":360285822,"node_id":"MDU6SXNzdWUzNjAyODU4MjI=","number":1550,"title":"Duplicate notifications after upgrading to Alertmanager 0.15","user":{"login":"tanji","id":971260,"node_id":"MDQ6VXNlcjk3MTI2MA==","avatar_url":"https://avatars2.githubusercontent.com/u/971260?v=4","gravatar_id":"","url":"https://api.github.com/users/tanji","html_url":"https://github.com/tanji","followers_url":"https://api.github.com/users/tanji/followers","following_url":"https://api.github.com/users/tanji/following{/other_user}","gists_url":"https://api.github.com/users/tanji/gists{/gist_id}","starred_url":"https://api.github.com/users/tanji/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tanji/subscriptions","organizations_url":"https://api.github.com/users/tanji/orgs","repos_url":"https://api.github.com/users/tanji/repos","events_url":"https://api.github.com/users/tanji/events{/privacy}","received_events_url":"https://api.github.com/users/tanji/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-09-14T12:32:57Z","updated_at":"2018-09-18T09:28:20Z","closed_at":null,"author_association":"NONE","body":"\r\nAfter upgrading from Alertmanager 0.13 to 0.15.2 in a cluster of two members we've started receiving double notifications in slack. It used to work flawlessly with 0.13. Weirdly we're receiving the 2 notifications exactly at the same time, they don't seem to be apart by more than a couple of secs.\r\n\r\n* System information:\r\n\r\nLinux pmm-server 3.16.0-4-amd64 #1 SMP Debian 3.16.39-1 (2016-12-30) x86_64 GNU/Linux\r\n\r\nBoth instances using ntp.\r\n\r\n* Alertmanager version:\r\n\r\nalertmanager, version 0.15.2 (branch: HEAD, revision: d19fae3bae451940b8470abb680cfdd59bfa7cfa)\r\n  build user:       root@3101e5b68a55\r\n  build date:       20180814-10:53:39\r\n  go version:       go1.10.3\r\n\r\nCluster status reports up:\r\n\r\nStatus\r\nUptime:\r\n2018-09-09T19:03:01.726517546Z\r\nCluster Status\r\nName:\r\n01CPZVEFADF9GE2G9F2CTZZZQ6\r\nStatus:\r\nready\r\nPeers:\r\nName: 01CPZV0HDRQY5M5TW6FDS31MKS\r\nAddress: <secret>:9094\r\nName: 01CPZVEFADF9GE2G9F2CTZZZQ6\r\nAddress: <secret>:9094\r\n\r\n* Prometheus version:\r\n\r\nIrrelevant\r\n\r\n* Alertmanager configuration file:\r\n```\r\nglobal:\r\n  resolve_timeout: 5m\r\n  http_config: {}\r\n  smtp_hello: localhost\r\n  smtp_require_tls: true\r\n  slack_api_url: <secret>\r\n  pagerduty_url: https://events.pagerduty.com/v2/enqueue\r\n  hipchat_api_url: https://api.hipchat.com/\r\n  opsgenie_api_url: https://api.opsgenie.com/\r\n  wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/\r\n  victorops_api_url: https://alert.victorops.com/integrations/generic/20131114/alert/\r\nroute:\r\n  receiver: ops-slack\r\n  group_by:\r\n  - alertname\r\n  - group\r\n  routes:\r\n  - receiver: ops-pager\r\n    match:\r\n      alertname: MySQLDown\r\n  - receiver: ops-pager\r\n    match:\r\n      offline: critical\r\n    continue: true\r\n    routes:\r\n    - receiver: ops-slack\r\n      match:\r\n        offline: critical\r\n  - receiver: ops-pager\r\n    match:\r\n      pager: \"yes\"\r\n  group_wait: 30s\r\n  group_interval: 5m\r\n  repeat_interval: 30m\r\nreceivers:\r\n- name: ops-slack\r\n  slack_configs:\r\n  - send_resolved: true\r\n    http_config: {}\r\n    api_url: <secret>\r\n    channel: alerts\r\n    username: prometheus\r\n    color: '{{ if eq .Status \"firing\" }}danger{{ else }}good{{ end }}'\r\n    title: '[{{ .Status | toUpper }}{{ if eq .Status \"firing\" }}:{{ .Alerts.Firing\r\n      | len }}{{ end }}] {{ .CommonAnnotations.summary }}'\r\n    title_link: '{{ template \"slack.default.titlelink\" . }}'\r\n    pretext: '{{ template \"slack.default.pretext\" . }}'\r\n    text: |-\r\n      {{ range .Alerts }}\r\n        *Alert:* {{ .Annotations.summary }} - *{{ .Labels.severity | toUpper }}* on {{ .Labels.instance }}\r\n        *Description:* {{ .Annotations.description }}\r\n        *Details:*\r\n        {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`\r\n        {{ end }}\r\n      {{ end }}\r\n    footer: '{{ template \"slack.default.footer\" . }}'\r\n    fallback: '{{ template \"slack.default.fallback\" . }}'\r\n    icon_emoji: '{{ template \"slack.default.iconemoji\" . }}'\r\n    icon_url: http://cdn.rancher.com/wp-content/uploads/2015/05/27094511/prometheus-logo-square.png\r\n```\r\n\r\n* Logs:\r\nno errors to speak of\r\n```\r\nlevel=info ts=2018-09-06T11:10:16.242620478Z caller=main.go:174 msg=\"Starting Alertmanager\" version=\"(version=0.15.2, branch=HEAD, revision=d19fae3bae451940b8470abb680cfdd59bfa7cfa)\"\r\nlevel=info ts=2018-09-06T11:10:16.242654842Z caller=main.go:175 build_context=\"(go=go1.10.3, user=root@3101e5b68a55, date=20180814-10:53:39)\"\r\nlevel=info ts=2018-09-06T11:10:16.313588161Z caller=main.go:322 msg=\"Loading configuration file\" file=/etc/alertmanager/config.yml\r\nlevel=info ts=2018-09-06T11:10:16.313610447Z caller=cluster.go:570 component=cluster msg=\"Waiting for gossip to settle...\" interval=2s\r\nlevel=info ts=2018-09-06T11:10:16.315607053Z caller=main.go:398 msg=Listening address=:9093\r\nlevel=info ts=2018-09-06T11:10:18.313944578Z caller=cluster.go:595 component=cluster msg=\"gossip not settled\" polls=0 before=0 now=2 elapsed=2.000297466s\r\nlevel=info ts=2018-09-06T11:10:22.314297199Z caller=cluster.go:595 component=cluster msg=\"gossip not settled\" polls=2 before=2 now=1 elapsed=6.000647448s\r\nlevel=info ts=2018-09-06T11:10:30.315059802Z caller=cluster.go:587 component=cluster msg=\"gossip settled; proceeding\" elapsed=14.001414171s\r\nlevel=info ts=2018-09-09T18:55:23.930653016Z caller=main.go:426 msg=\"Received SIGTERM, exiting gracefully...\"\r\nlevel=info ts=2018-09-09T18:55:25.067197275Z caller=main.go:174 msg=\"Starting Alertmanager\" version=\"(version=0.15.2, branch=HEAD, revision=d19fae3bae451940b8470abb680cfdd59bfa7cfa)\"\r\nlevel=info ts=2018-09-09T18:55:25.067233709Z caller=main.go:175 build_context=\"(go=go1.10.3, user=root@3101e5b68a55, date=20180814-10:53:39)\"\r\nlevel=info ts=2018-09-09T18:55:25.128486689Z caller=main.go:322 msg=\"Loading configuration file\" file=/etc/alertmanager/config.yml\r\nlevel=info ts=2018-09-09T18:55:25.128488742Z caller=cluster.go:570 component=cluster msg=\"Waiting for gossip to settle...\" interval=2s\r\nlevel=info ts=2018-09-09T18:55:25.131985874Z caller=main.go:398 msg=Listening address=:9093\r\nlevel=info ts=2018-09-09T18:55:27.128662897Z caller=cluster.go:595 component=cluster msg=\"gossip not settled\" polls=0 before=0 now=3 elapsed=2.000096829s\r\nlevel=info ts=2018-09-09T18:55:31.128969079Z caller=cluster.go:595 component=cluster msg=\"gossip not settled\" polls=2 before=3 now=2 elapsed=6.000402722s\r\nlevel=info ts=2018-09-09T18:55:33.129130021Z caller=cluster.go:595 component=cluster msg=\"gossip not settled\" polls=3 before=2 now=1 elapsed=8.000564176s\r\nlevel=info ts=2018-09-09T18:55:37.129427658Z caller=cluster.go:595 component=cluster msg=\"gossip not settled\" polls=5 before=1 now=2 elapsed=12.000855483s\r\nlevel=info ts=2018-09-09T18:55:45.130073007Z caller=cluster.go:587 component=cluster msg=\"gossip settled; proceeding\" elapsed=20.001506309s\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/prometheus/alertmanager/issues/comments/422323111","html_url":"https://github.com/prometheus/alertmanager/issues/1550#issuecomment-422323111","issue_url":"https://api.github.com/repos/prometheus/alertmanager/issues/1550","id":422323111,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMjMyMzExMQ==","user":{"login":"tanji","id":971260,"node_id":"MDQ6VXNlcjk3MTI2MA==","avatar_url":"https://avatars2.githubusercontent.com/u/971260?v=4","gravatar_id":"","url":"https://api.github.com/users/tanji","html_url":"https://github.com/tanji","followers_url":"https://api.github.com/users/tanji/followers","following_url":"https://api.github.com/users/tanji/following{/other_user}","gists_url":"https://api.github.com/users/tanji/gists{/gist_id}","starred_url":"https://api.github.com/users/tanji/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tanji/subscriptions","organizations_url":"https://api.github.com/users/tanji/orgs","repos_url":"https://api.github.com/users/tanji/repos","events_url":"https://api.github.com/users/tanji/events{/privacy}","received_events_url":"https://api.github.com/users/tanji/received_events","type":"User","site_admin":false},"created_at":"2018-09-18T09:28:20Z","updated_at":"2018-09-18T09:28:20Z","author_association":"NONE","body":"That just happens when starting alertmanager, no messages are appearing after this, those alertmanagers are deployed in the cloud and are pretty close to each other. Of note, we use a docker image to deploy. here are the configs and startup parameters\r\n\r\n```\r\n  alertmanager:\r\n    image: prom/alertmanager:latest\r\n    ports:\r\n      - 9093:9093\r\n      - 9094:9094\r\n    volumes:\r\n      - alertmanager_data:/alertmanager\r\n      - /etc/alertmanager:/etc/alertmanager\r\n    restart: always\r\n    command:\r\n      - --config.file=/etc/alertmanager/config.yml\r\n      - --storage.path=/alertmanager\r\n      - --web.external-url=http://pmm-server-ec2:9093\r\n      - --cluster.peer=pmm-server:9094\r\n      - --cluster.advertise-address=(external IP of the EC2 VM)`\r\n```\r\n\r\nSecond AM\r\n```  alertmanager:\r\n    image: prom/alertmanager:latest\r\n    ports:\r\n      - 127.0.0.1:9093:9093\r\n      - 9094:9094\r\n    volumes:\r\n      - alertmanager_data:/alertmanager\r\n      - /etc/alertmanager:/etc/alertmanager\r\n    restart: always\r\n    command:\r\n      - --config.file=/etc/alertmanager/config.yml\r\n      - --storage.path=/alertmanager\r\n      - --web.external-url=https://alertmanager\r\n      - --cluster.peer=pmm-server-ec2:9094\r\n      - --cluster.advertise-address=(external IP of the server):9094```\r\n\r\nNothing really rocket science, this setup (with the old mesh protocol) worked without duplicates until I upgraded.\r\n\r\nRe. 3 nodes could it be that it's also trying to connect to itself?"}}
{"actor":{"display_login":"tanji"}}