{"action":"opened","issue":{"url":"https://api.github.com/repos/prometheus/alertmanager/issues/1661","repository_url":"https://api.github.com/repos/prometheus/alertmanager","labels_url":"https://api.github.com/repos/prometheus/alertmanager/issues/1661/labels{/name}","comments_url":"https://api.github.com/repos/prometheus/alertmanager/issues/1661/comments","events_url":"https://api.github.com/repos/prometheus/alertmanager/issues/1661/events","html_url":"https://github.com/prometheus/alertmanager/issues/1661","id":390299943,"node_id":"MDU6SXNzdWUzOTAyOTk5NDM=","number":1661,"title":"Initially unresolved peers are lost forever","user":{"login":"redbaron","id":16624,"node_id":"MDQ6VXNlcjE2NjI0","avatar_url":"https://avatars3.githubusercontent.com/u/16624?v=4","gravatar_id":"","url":"https://api.github.com/users/redbaron","html_url":"https://github.com/redbaron","followers_url":"https://api.github.com/users/redbaron/followers","following_url":"https://api.github.com/users/redbaron/following{/other_user}","gists_url":"https://api.github.com/users/redbaron/gists{/gist_id}","starred_url":"https://api.github.com/users/redbaron/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/redbaron/subscriptions","organizations_url":"https://api.github.com/users/redbaron/orgs","repos_url":"https://api.github.com/users/redbaron/repos","events_url":"https://api.github.com/users/redbaron/events{/privacy}","received_events_url":"https://api.github.com/users/redbaron/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-12-12T16:16:34Z","updated_at":"2018-12-12T16:16:34Z","closed_at":null,"author_association":"NONE","body":"**What did you do?**\r\n\r\nRun alertmanager as a statefulset with `--cluster-peer` command line given once per member. If initial DNS resolution of peers fails, they  are forgotten and never retried. They also neither  show up in `clusterStatus.peers`, nor established connections are seen in netstat\r\n\r\nEach pod of statefulset of 3  shows their own IP address in `clusterStatus` and report themselves as ready. Should I restart any one of them (`kubectl exec pkill alert`), it comes up, manages to resolve peer IP addresses and joins other 2, forming cluster of 3 (those which didn't get restarted are probably find about each other via gossip)\r\n\r\n**What did you expect to see?**\r\n\r\nearly DNS failures shouldn't cause isolated \"islands\" to form.\r\n\r\nI suspect problem is in https://github.com/prometheus/alertmanager/blob/7f34cb471671abb3df6e3ef5e918cfa7fc3c4958/cluster/cluster.go#L693-L698 where DNS failure is interpreted as if exact IP was given.\r\n\r\n\r\n\r\n**What did you see instead? Under which circumstances?**\r\n\r\nEach member of statefulset forms single-node cluster.\r\n\r\n**Environment**\r\n\r\n\r\n* Alertmanager version:\r\n\r\n0.15.3\r\n\r\n\r\n* Alertmanager configuration file:\r\n```\r\nglobal:\r\n  resolve_timeout: 5m\r\nroute:\r\n  group_by: ['job']\r\n  group_wait: 30s\r\n  group_interval: 5m\r\n  repeat_interval: 12h\r\n  receiver: 'null'\r\n  routes:\r\n  - match:\r\n      alertname: DeadMansSwitch\r\n    receiver: 'null'\r\nreceivers:\r\n- name: 'null'\r\n\r\n```\r\n* Logs:\r\n```\r\nlevel=info ts=2018-12-12T15:59:11.569012901Z caller=main.go:174 msg=\"Starting Alertmanager\" version=\"(version=0.15.3, branch=HEAD, revision=d4a7697cc90f8bce62efe7c44b63b542578ec0a1)\"\r\nlevel=info ts=2018-12-12T15:59:11.569095844Z caller=main.go:175 build_context=\"(go=go1.11.2, user=root@4ecc17c53d26, date=20181109-15:40:48)\"\r\nlevel=debug ts=2018-12-12T15:59:11.576487801Z caller=cluster.go:143 component=cluster msg=\"resolved peers to following addresses\" peers=alertmanager-test-0.alertmanager-operated.allns-x-amclustergossipsilences-pjmsed-0.svc:6783,alertmanag\r\ner-test-1.alertmanager-operated.allns-x-amclustergossipsilences-pjmsed-0.svc:6783,alertmanager-test-2.alertmanager-operated.allns-x-amclustergossipsilences-pjmsed-0.svc:6783\r\nlevel=debug ts=2018-12-12T15:59:11.579127508Z caller=delegate.go:209 component=cluster received=NotifyJoin node=01CYHJ9DPRWZWC3P8VZW6J2E1H addr=192.168.0.129:6783\r\nlevel=debug ts=2018-12-12T15:59:11.582852186Z caller=cluster.go:287 component=cluster memberlist=\"2018/12/12 15:59:11 [WARN] memberlist: Failed to resolve alertmanager-test-0.alertmanager-operated.allns-x-amclustergossipsilences-pjmsed-0.\r\nsvc:6783: lookup alertmanager-test-0.alertmanager-operated.allns-x-amclustergossipsilences-pjmsed-0.svc on 10.96.0.10:53: no such host\\n\"\r\nlevel=debug ts=2018-12-12T15:59:11.586089894Z caller=cluster.go:287 component=cluster memberlist=\"2018/12/12 15:59:11 [WARN] memberlist: Failed to resolve alertmanager-test-1.alertmanager-operated.allns-x-amclustergossipsilences-pjmsed-0.\r\nsvc:6783: lookup alertmanager-test-1.alertmanager-operated.allns-x-amclustergossipsilences-pjmsed-0.svc on 10.96.0.10:53: no such host\\n\"\r\nlevel=debug ts=2018-12-12T15:59:11.589100603Z caller=cluster.go:287 component=cluster memberlist=\"2018/12/12 15:59:11 [WARN] memberlist: Failed to resolve alertmanager-test-2.alertmanager-operated.allns-x-amclustergossipsilences-pjmsed-0.\r\nsvc:6783: lookup alertmanager-test-2.alertmanager-operated.allns-x-amclustergossipsilences-pjmsed-0.svc on 10.96.0.10:53: no such host\\n\"\r\nlevel=warn ts=2018-12-12T15:59:11.589148819Z caller=cluster.go:219 component=cluster msg=\"failed to join cluster\" err=\"3 errors occurred:\\n\\n* Failed to resolve alertmanager-test-0.alertmanager-operated.allns-x-amclustergossipsilences-pjm\r\nsed-0.svc:6783: lookup alertmanager-test-0.alertmanager-operated.allns-x-amclustergossipsilences-pjmsed-0.svc on 10.96.0.10:53: no such host\\n* Failed to resolve alertmanager-test-1.alertmanager-operated.allns-x-amclustergossipsilences-pj\r\nmsed-0.svc:6783: lookup alertmanager-test-1.alertmanager-operated.allns-x-amclustergossipsilences-pjmsed-0.svc on 10.96.0.10:53: no such host\\n* Failed to resolve alertmanager-test-2.alertmanager-operated.allns-x-amclustergossipsilences-p\r\njmsed-0.svc:6783: lookup alertmanager-test-2.alertmanager-operated.allns-x-amclustergossipsilences-pjmsed-0.svc on 10.96.0.10:53: no such host\"\r\nlevel=info ts=2018-12-12T15:59:11.589212472Z caller=cluster.go:221 component=cluster msg=\"will retry joining cluster every 10s\"\r\nlevel=warn ts=2018-12-12T15:59:11.589237137Z caller=main.go:265 msg=\"unable to join gossip mesh\" err=\"3 errors occurred:\\n\\n* Failed to resolve alertmanager-test-0.alertmanager-operated.allns-x-amclustergossipsilences-pjmsed-0.svc:6783: l\r\nookup alertmanager-test-0.alertmanager-operated.allns-x-amclustergossipsilences-pjmsed-0.svc on 10.96.0.10:53: no such host\\n* Failed to resolve alertmanager-test-1.alertmanager-operated.allns-x-amclustergossipsilences-pjmsed-0.svc:6783: \r\nlookup alertmanager-test-1.alertmanager-operated.allns-x-amclustergossipsilences-pjmsed-0.svc on 10.96.0.10:53: no such host\\n* Failed to resolve alertmanager-test-2.alertmanager-operated.allns-x-amclustergossipsilences-pjmsed-0.svc:6783:\r\n lookup alertmanager-test-2.alertmanager-operated.allns-x-amclustergossipsilences-pjmsed-0.svc on 10.96.0.10:53: no such host\"\r\nlevel=info ts=2018-12-12T15:59:11.589316523Z caller=main.go:322 msg=\"Loading configuration file\" file=/etc/alertmanager/config/alertmanager.yaml\r\nlevel=info ts=2018-12-12T15:59:11.589762081Z caller=cluster.go:570 component=cluster msg=\"Waiting for gossip to settle...\" interval=2s\r\nlevel=info ts=2018-12-12T15:59:11.593358442Z caller=main.go:398 msg=Listening address=:9093\r\nlevel=info ts=2018-12-12T15:59:13.590121522Z caller=cluster.go:595 component=cluster msg=\"gossip not settled\" polls=0 before=0 now=1 elapsed=2.000133907s\r\nlevel=debug ts=2018-12-12T15:59:15.590328722Z caller=cluster.go:592 component=cluster msg=\"gossip looks settled\" elapsed=4.000379327s\r\nlevel=debug ts=2018-12-12T15:59:17.590574734Z caller=cluster.go:592 component=cluster msg=\"gossip looks settled\" elapsed=6.000623571s\r\nlevel=debug ts=2018-12-12T15:59:19.590842429Z caller=cluster.go:592 component=cluster msg=\"gossip looks settled\" elapsed=8.000892937s\r\nlevel=info ts=2018-12-12T15:59:21.59108732Z caller=cluster.go:587 component=cluster msg=\"gossip settled; proceeding\" elapsed=10.0011107s\r\n\r\n```\r\n\r\n\r\nStatus:\r\n\r\n```\r\n{\"status\":\"success\",\"data\":{\"configYAML\":\"global:\\n  resolve_timeout: 5m\\n  http_config: {}\\n  smtp_hello: localhost\\n  smtp_require_tls: true\\n  pagerduty_url: https://events.pagerduty.com/v2/enqueue\\n  hipchat_api_url: https://api.hipch\r\nat.com/\\n  opsgenie_api_url: https://api.opsgenie.com/\\n  wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/\\n  victorops_api_url: https://alert.victorops.com/integrations/generic/20131114/alert/\\nroute:\\n  receiver: \\\"null\\\"\\n  group_b\r\ny:\\n  - job\\n  routes:\\n  - receiver: \\\"null\\\"\\n    match:\\n      alertname: DeadMansSwitch\\n  group_wait: 30s\\n  group_interval: 5m\\n  repeat_interval: 12h\\nreceivers:\\n- name: \\\"null\\\"\\ntemplates: []\\n\",\"configJSON\":{\"global\":{\"resolve_\r\ntimeout\":300000000000,\"http_config\":{\"BasicAuth\":null,\"BearerToken\":\"\",\"BearerTokenFile\":\"\",\"ProxyURL\":{},\"TLSConfig\":{\"CAFile\":\"\",\"CertFile\":\"\",\"KeyFile\":\"\",\"ServerName\":\"\",\"InsecureSkipVerify\":false}},\"smtp_hello\":\"localhost\",\"smtp_requ\r\nire_tls\":true,\"pagerduty_url\":\"https://events.pagerduty.com/v2/enqueue\",\"hipchat_api_url\":\"https://api.hipchat.com/\",\"opsgenie_api_url\":\"https://api.opsgenie.com/\",\"wechat_api_url\":\"https://qyapi.weixin.qq.com/cgi-bin/\",\"victorops_api_url\r\n\":\"https://alert.victorops.com/integrations/generic/20131114/alert/\"},\"route\":{\"receiver\":\"null\",\"group_by\":[\"job\"],\"routes\":[{\"receiver\":\"null\",\"match\":{\"alertname\":\"DeadMansSwitch\"}}],\"group_wait\":30000000000,\"group_interval\":3000000000\r\n00,\"repeat_interval\":43200000000000},\"receivers\":[{\"name\":\"null\"}],\"templates\":null},\"versionInfo\":{\"branch\":\"HEAD\",\"buildDate\":\"20181109-15:40:48\",\"buildUser\":\"root@4ecc17c53d26\",\"goVersion\":\"go1.11.2\",\"revision\":\"d4a7697cc90f8bce62efe7c\r\n44b63b542578ec0a1\",\"version\":\"0.15.3\"},\"uptime\":\"2018-12-12T15:59:11.589298893Z\",\"clusterStatus\":{\"name\":\"01CYHJ9DPRWZWC3P8VZW6J2E1H\",\"status\":\"ready\",\"peers\":[{\"name\":\"01CYHJ9DPRWZWC3P8VZW6J2E1H\",\"address\":\"192.168.0.129:6783\"}]}}}\r\n```"}}
{"actor":{"display_login":"redbaron"}}