{"action":"created","issue":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/3413","repository_url":"https://api.github.com/repos/prometheus/prometheus","labels_url":"https://api.github.com/repos/prometheus/prometheus/issues/3413/labels{/name}","comments_url":"https://api.github.com/repos/prometheus/prometheus/issues/3413/comments","events_url":"https://api.github.com/repos/prometheus/prometheus/issues/3413/events","html_url":"https://github.com/prometheus/prometheus/issues/3413","id":271218444,"node_id":"MDU6SXNzdWUyNzEyMTg0NDQ=","number":3413,"title":"Prometheus 2.0: spurious errors in consul service discovery","user":{"login":"thesamet","id":307373,"node_id":"MDQ6VXNlcjMwNzM3Mw==","avatar_url":"https://avatars0.githubusercontent.com/u/307373?v=4","gravatar_id":"","url":"https://api.github.com/users/thesamet","html_url":"https://github.com/thesamet","followers_url":"https://api.github.com/users/thesamet/followers","following_url":"https://api.github.com/users/thesamet/following{/other_user}","gists_url":"https://api.github.com/users/thesamet/gists{/gist_id}","starred_url":"https://api.github.com/users/thesamet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thesamet/subscriptions","organizations_url":"https://api.github.com/users/thesamet/orgs","repos_url":"https://api.github.com/users/thesamet/repos","events_url":"https://api.github.com/users/thesamet/events{/privacy}","received_events_url":"https://api.github.com/users/thesamet/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2017-11-04T19:40:26Z","updated_at":"2018-09-26T16:03:08Z","closed_at":"2017-11-07T21:56:44Z","author_association":"NONE","body":"**What did you do?**\r\nStart prometheus\r\n\r\n**What did you expect to see?**\r\nNo errors in the logs\r\n\r\n**What did you see instead? Under which circumstances?**\r\nFrom time to time prometheus would print a log message indicating that it failed fetching service information from consul. There is little impact, since it does seem to be able to connect to it at start time.\r\n\r\n**Environment**\r\nPrometheus 2.0.0-rc2 on Linux. Service discovery using consul.\r\n\r\n* System information:\r\n\r\n        Linux 4.4.0-57-generic x86_64\r\n\t\r\n* Prometheus version:\r\n```prometheus, version 2.0.0-rc.2 (branch: HEAD, revision: ce63a5a8557bb33e2030a7756c58fd773736b592)\r\n  build user:       root@a6d2e4a7b8da\r\n  build date:       20171025-18:42:54\r\n  go version:       go1.9.1\r\n```\r\n\r\n* Alertmanager version:\r\nNot installed.\r\n\r\n* Prometheus configuration file:\r\n```\r\nglobal:\r\n  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.\r\n  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.\r\n  # scrape_timeout is set to the global default (10s).\r\n\r\n  # Attach these labels to any time series or alerts when communicating with\r\n  # external systems (federation, remote storage, Alertmanager).\r\n  external_labels:\r\n      monitor: 'codelab-monitor'\r\n\r\n\r\nscrape_configs:\r\n  - job_name: 'prometheus'\r\n    static_configs:\r\n      - targets: ['localhost:9090']\r\n\r\n  - job_name: consul\r\n    consul_sd_configs:\r\n    - server: consul:8500\r\n      services:\r\n      - myservice\r\n    relabel_configs:\r\n    - source_labels: [\"__meta_consul_address\", \"__meta_consul_service_port\"]\r\n      separator: \":\"\r\n      target_label: __address__\r\n      regex: \"(.*):9(.*)\"\r\n      replacement: \"$1:19$2\"\r\n    - source_labels: [\"__meta_consul_service\"]\r\n      target_label: job\r\n    - source_labels: [\"__meta_consul_tags\"]\r\n      regex: \".*(prod|stage).*\"\r\n      target_label: env\r\n```\r\n\r\n* Logs:\r\n```\r\nlevel=error ts=2017-11-04T19:20:21.399644298Z caller=consul.go:283 component=\"target manager\" discovery=consul msg=\"Error refreshing service\" service=trends err=\"Get http://consul:8500/v1/catalog/service/trends?index=9598981&wait=30000ms: net/http: request canceled (Client.Timeout exceeded while awaiting headers)\"\r\nlevel=error ts=2017-11-04T19:20:21.399854827Z caller=consul.go:186 component=\"target manager\" discovery=consul msg=\"Error refreshing service list\" err=\"Get http://consul:8500/v1/catalog/services?index=9598981&wait=30000ms: net/http: request canceled (Client.Timeout exceeded while awaiting headers)\"\r\n```"},"comment":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/comments/424772773","html_url":"https://github.com/prometheus/prometheus/issues/3413#issuecomment-424772773","issue_url":"https://api.github.com/repos/prometheus/prometheus/issues/3413","id":424772773,"node_id":"MDEyOklzc3VlQ29tbWVudDQyNDc3Mjc3Mw==","user":{"login":"hbagdi","id":4694585,"node_id":"MDQ6VXNlcjQ2OTQ1ODU=","avatar_url":"https://avatars1.githubusercontent.com/u/4694585?v=4","gravatar_id":"","url":"https://api.github.com/users/hbagdi","html_url":"https://github.com/hbagdi","followers_url":"https://api.github.com/users/hbagdi/followers","following_url":"https://api.github.com/users/hbagdi/following{/other_user}","gists_url":"https://api.github.com/users/hbagdi/gists{/gist_id}","starred_url":"https://api.github.com/users/hbagdi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hbagdi/subscriptions","organizations_url":"https://api.github.com/users/hbagdi/orgs","repos_url":"https://api.github.com/users/hbagdi/repos","events_url":"https://api.github.com/users/hbagdi/events{/privacy}","received_events_url":"https://api.github.com/users/hbagdi/received_events","type":"User","site_admin":false},"created_at":"2018-09-26T16:03:08Z","updated_at":"2018-09-26T16:03:08Z","author_association":"NONE","body":"Facing the same issue as posted by @vishksaj  with 2.2.1 with no apparent pattern"}}
{"actor":{"display_login":"hbagdi"}}