{"action":"created","issue":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/1316","labels_url":"https://api.github.com/repos/prometheus/prometheus/issues/1316/labels{/name}","comments_url":"https://api.github.com/repos/prometheus/prometheus/issues/1316/comments","events_url":"https://api.github.com/repos/prometheus/prometheus/issues/1316/events","html_url":"https://github.com/prometheus/prometheus/issues/1316","id":126691763,"number":1316,"title":"Kubernetes discovery misses some changes","user":{"login":"matthiasr","id":131590,"avatar_url":"https://avatars.githubusercontent.com/u/131590?v=3","gravatar_id":"","url":"https://api.github.com/users/matthiasr","html_url":"https://github.com/matthiasr","followers_url":"https://api.github.com/users/matthiasr/followers","following_url":"https://api.github.com/users/matthiasr/following{/other_user}","gists_url":"https://api.github.com/users/matthiasr/gists{/gist_id}","starred_url":"https://api.github.com/users/matthiasr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matthiasr/subscriptions","organizations_url":"https://api.github.com/users/matthiasr/orgs","repos_url":"https://api.github.com/users/matthiasr/repos","events_url":"https://api.github.com/users/matthiasr/events{/privacy}","received_events_url":"https://api.github.com/users/matthiasr/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/prometheus/prometheus/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-01-14T16:32:21Z","updated_at":"2016-01-14T16:54:01Z","closed_at":null,"body":"When bringing up a local cluster for integration tests, sometimes Prometheus (using the example Kubernetes configuration) does not pick up a service that requests to be scraped using the `prometheus.io/scrape` annotation.\r\n\r\nIn this situation, sending a SIGHUP to the Prometheus process seems to \"fix\" this:\r\n\r\n```\r\nmr@ip-10-33-34-50 ~/k8s (git)-[master] % kubectl-local --namespace=kube-system exec -ti deploymentrc-2964570028-e8nbh -- wget -q -O - \"http://localhost:9090/api/v1/query?query=haproxy_backend_up&time=$(date -u +%s)\"\r\n{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[]}}\r\nmr@ip-10-33-34-50 ~/k8s (git)-[master] % kubectl-local --namespace=kube-system exec -ti deploymentrc-2964570028-e8nbh -- pkill -HUP prometheus\r\nmr@ip-10-33-34-50 ~/k8s (git)-[master] % kubectl-local --namespace=kube-system exec -ti deploymentrc-2964570028-e8nbh -- wget -q -O - \"http://localhost:9090/api/v1/query?query=haproxy_backend_up&time=$(date -u +%s)\"\r\n{\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"haproxy_backend_up\",\"backend\":\"default-web-80\",\"instance\":\"172.17.128.7:9101\",\"job\":\"kubernetes-services\",\"kubernetes_name\":\"ingress-controller\",\"kubernetes_namespace\":\"ingress-internal\",\"kubernetes_role\":\"service\"},\"value\":[1452788773,\"1\"]},{\"metric\":{\"__name__\":\"haproxy_backend_up\",\"backend\":\"deny\",\"instance\":\"172.17.128.7:9101\",\"job\":\"kubernetes-services\",\"kubernetes_name\":\"ingress-controller\",\"kubernetes_namespace\":\"ingress-internal\",\"kubernetes_role\":\"service\"},\"value\":[1452788773,\"1\"]},{\"metric\":{\"__name__\":\"haproxy_backend_up\",\"backend\":\"kube-system-kibana-logging-5601\",\"instance\":\"172.17.128.7:9101\",\"job\":\"kubernetes-services\",\"kubernetes_name\":\"ingress-controller\",\"kubernetes_namespace\":\"ingress-internal\",\"kubernetes_role\":\"service\"},\"value\":[1452788773,\"1\"]},{\"metric\":{\"__name__\":\"haproxy_backend_up\",\"backend\":\"kube-system-prometheus-9090\",\"instance\":\"172.17.128.7:9101\",\"job\":\"kubernetes-services\",\"kubernetes_name\":\"ingress-controller\",\"kubernetes_namespace\":\"ingress-internal\",\"kubernetes_role\":\"service\"},\"value\":[1452788773,\"1\"]},{\"metric\":{\"__name__\":\"haproxy_backend_up\",\"backend\":\"stats\",\"instance\":\"172.17.128.7:9101\",\"job\":\"kubernetes-services\",\"kubernetes_name\":\"ingress-controller\",\"kubernetes_namespace\":\"ingress-internal\",\"kubernetes_role\":\"service\"},\"value\":[1452788773,\"1\"]}]}}\r\nmr@ip-10-33-34-50 ~/k8s (git)-[master] % kubectl-local --namespace=kube-system logs deploymentrc-2964570028-e8nbh\r\nprometheus, version 0.16.1 (branch: HEAD, revision: 968ee35)\r\n  build user:       @84057be02ab6\r\n  build date:       20160111-15:47:17\r\n  go version:       1.5.1\r\ntime=\"2016-01-14T16:08:50Z\" level=info msg=\"Loading configuration file /etc/prometheus/prometheus.yml\" source=\"main.go:196\"\r\ntime=\"2016-01-14T16:08:50Z\" level=info msg=\"Loading series map and head chunks...\" source=\"storage.go:262\"\r\ntime=\"2016-01-14T16:08:50Z\" level=info msg=\"0 series loaded.\" source=\"storage.go:267\"\r\ntime=\"2016-01-14T16:08:50Z\" level=info msg=\"Starting target manager...\" source=\"targetmanager.go:114\"\r\ntime=\"2016-01-14T16:08:50Z\" level=info msg=\"Listening on :9090\" source=\"web.go:223\"\r\ntime=\"2016-01-14T16:09:00Z\" level=error msg=\"Unable to list Kubernetes nodes: Unable to query any masters\" source=\"discovery.go:113\"\r\ntime=\"2016-01-14T16:09:10Z\" level=error msg=\"Unable to list Kubernetes nodes: Unable to query any masters\" source=\"discovery.go:113\"\r\ntime=\"2016-01-14T16:09:20Z\" level=error msg=\"Failed to watch service endpoints: Unable to query any masters\" source=\"discovery.go:551\"\r\ntime=\"2016-01-14T16:09:20Z\" level=error msg=\"Failed to watch service endpoints: Unable to query any masters\" source=\"discovery.go:551\"\r\ntime=\"2016-01-14T16:09:30Z\" level=error msg=\"Failed to watch nodes: Unable to query any masters\" source=\"discovery.go:365\"\r\ntime=\"2016-01-14T16:09:30Z\" level=error msg=\"Failed to watch nodes: Unable to query any masters\" source=\"discovery.go:365\"\r\ntime=\"2016-01-14T16:09:40Z\" level=error msg=\"Failed to watch services: Unable to query any masters\" source=\"discovery.go:407\"\r\ntime=\"2016-01-14T16:09:40Z\" level=error msg=\"Failed to watch services: Unable to query any masters\" source=\"discovery.go:407\"\r\ntime=\"2016-01-14T16:09:50Z\" level=error msg=\"Failed to watch service endpoints: Unable to query any masters\" source=\"discovery.go:551\"\r\ntime=\"2016-01-14T16:09:50Z\" level=error msg=\"Failed to watch service endpoints: Unable to query any masters\" source=\"discovery.go:551\"\r\ntime=\"2016-01-14T16:10:00Z\" level=error msg=\"Failed to watch nodes: Unable to query any masters\" source=\"discovery.go:365\"\r\ntime=\"2016-01-14T16:10:00Z\" level=error msg=\"Failed to watch nodes: Unable to query any masters\" source=\"discovery.go:365\"\r\ntime=\"2016-01-14T16:10:10Z\" level=error msg=\"Failed to watch services: Unable to query any masters\" source=\"discovery.go:407\"\r\ntime=\"2016-01-14T16:10:10Z\" level=error msg=\"Failed to watch services: Unable to query any masters\" source=\"discovery.go:407\"\r\ntime=\"2016-01-14T16:10:20Z\" level=error msg=\"Failed to watch service endpoints: Unable to query any masters\" source=\"discovery.go:551\"\r\ntime=\"2016-01-14T16:10:20Z\" level=error msg=\"Failed to watch service endpoints: Unable to query any masters\" source=\"discovery.go:551\"\r\ntime=\"2016-01-14T16:10:30Z\" level=error msg=\"Failed to watch nodes: Unable to query any masters\" source=\"discovery.go:365\"\r\ntime=\"2016-01-14T16:10:30Z\" level=error msg=\"Failed to watch nodes: Unable to query any masters\" source=\"discovery.go:365\"\r\ntime=\"2016-01-14T16:10:40Z\" level=error msg=\"Failed to watch services: Unable to query any masters\" source=\"discovery.go:407\"\r\ntime=\"2016-01-14T16:10:40Z\" level=error msg=\"Failed to watch services: Unable to query any masters\" source=\"discovery.go:407\"\r\ntime=\"2016-01-14T16:13:50Z\" level=info msg=\"Checkpointing in-memory metrics and chunks...\" source=\"persistence.go:539\"\r\ntime=\"2016-01-14T16:13:50Z\" level=info msg=\"Done checkpointing in-memory metrics and chunks in 178.226463ms.\" source=\"persistence.go:563\"\r\ntime=\"2016-01-14T16:18:50Z\" level=info msg=\"Checkpointing in-memory metrics and chunks...\" source=\"persistence.go:539\"\r\ntime=\"2016-01-14T16:18:50Z\" level=info msg=\"Done checkpointing in-memory metrics and chunks in 167.210103ms.\" source=\"persistence.go:563\"\r\ntime=\"2016-01-14T16:23:50Z\" level=info msg=\"Checkpointing in-memory metrics and chunks...\" source=\"persistence.go:539\"\r\ntime=\"2016-01-14T16:23:50Z\" level=info msg=\"Done checkpointing in-memory metrics and chunks in 144.101098ms.\" source=\"persistence.go:563\"\r\ntime=\"2016-01-14T16:26:10Z\" level=info msg=\"Loading configuration file /etc/prometheus/prometheus.yml\" source=\"main.go:196\"\r\ntime=\"2016-01-14T16:26:10Z\" level=info msg=\"Stopping target manager...\" source=\"targetmanager.go:203\"\r\ntime=\"2016-01-14T16:26:10Z\" level=info msg=\"Target manager stopped.\" source=\"targetmanager.go:216\"\r\ntime=\"2016-01-14T16:26:10Z\" level=info msg=\"Starting target manager...\" source=\"targetmanager.go:114\"\r\n```\r\n\r\nOf note may be that during the bring-up of this cluster, several components are dumped in pretty much at once, in parallel, and in no particular order. Prometheus and the to-be-scraped HAProxy exporter may come up in either order or at the same time. Once in about 5 times it gets into the situation above, which may or may not be a race condition …\r\n\r\nThere's also a bunch of errors about being unable to query any masters, but they also happen when everything works out just fine.\r\n\r\n@jimmidyson any idea what's up with that?"},"comment":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/comments/171700809","html_url":"https://github.com/prometheus/prometheus/issues/1316#issuecomment-171700809","issue_url":"https://api.github.com/repos/prometheus/prometheus/issues/1316","id":171700809,"user":{"login":"matthiasr","id":131590,"avatar_url":"https://avatars.githubusercontent.com/u/131590?v=3","gravatar_id":"","url":"https://api.github.com/users/matthiasr","html_url":"https://github.com/matthiasr","followers_url":"https://api.github.com/users/matthiasr/followers","following_url":"https://api.github.com/users/matthiasr/following{/other_user}","gists_url":"https://api.github.com/users/matthiasr/gists{/gist_id}","starred_url":"https://api.github.com/users/matthiasr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matthiasr/subscriptions","organizations_url":"https://api.github.com/users/matthiasr/orgs","repos_url":"https://api.github.com/users/matthiasr/repos","events_url":"https://api.github.com/users/matthiasr/events{/privacy}","received_events_url":"https://api.github.com/users/matthiasr/received_events","type":"User","site_admin":false},"created_at":"2016-01-14T16:54:01Z","updated_at":"2016-01-14T16:54:01Z","body":"PS: I'm now serializing pushing components into the cluster, I'll report back if that makes things better."}}