{"action":"created","issue":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/4253","repository_url":"https://api.github.com/repos/prometheus/prometheus","labels_url":"https://api.github.com/repos/prometheus/prometheus/issues/4253/labels{/name}","comments_url":"https://api.github.com/repos/prometheus/prometheus/issues/4253/comments","events_url":"https://api.github.com/repos/prometheus/prometheus/issues/4253/events","html_url":"https://github.com/prometheus/prometheus/issues/4253","id":331479378,"node_id":"MDU6SXNzdWUzMzE0NzkzNzg=","number":4253,"title":"prometheus 2.3.0 become OOM when consul is unavailable","user":{"login":"Wing924","id":8895721,"node_id":"MDQ6VXNlcjg4OTU3MjE=","avatar_url":"https://avatars0.githubusercontent.com/u/8895721?v=4","gravatar_id":"","url":"https://api.github.com/users/Wing924","html_url":"https://github.com/Wing924","followers_url":"https://api.github.com/users/Wing924/followers","following_url":"https://api.github.com/users/Wing924/following{/other_user}","gists_url":"https://api.github.com/users/Wing924/gists{/gist_id}","starred_url":"https://api.github.com/users/Wing924/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Wing924/subscriptions","organizations_url":"https://api.github.com/users/Wing924/orgs","repos_url":"https://api.github.com/users/Wing924/repos","events_url":"https://api.github.com/users/Wing924/events{/privacy}","received_events_url":"https://api.github.com/users/Wing924/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2018-06-12T08:02:47Z","updated_at":"2018-06-13T09:55:33Z","closed_at":null,"author_association":"NONE","body":"## Bug Report\r\n**What did you do?**\r\nI use consul SD in prometheus.\r\nI upgraded consul servers but failed to failover and cause consul cluster died for a few minutes.\r\n\r\n**What did you expect to see?**\r\nPrometheus running as usual.\r\n\r\n**What did you see instead? Under which circumstances?**\r\nPrometheus become out of memory.\r\n\r\n**Environment**\r\n\r\n* System information:\r\n\r\n\tLinux 3.10.0-693.2.2.el7.x86_64 x86_64\r\n\r\n* Prometheus version:\r\n\r\n\tprometheus, version 2.3.0 (branch: HEAD, revision: 290d71791a507a5057b9a099c9d48703d86dc941)\r\n      build user:       root@d539e167976a\r\n      build date:       20180607-08:46:54\r\n      go version:       go1.10.2\r\n\r\n* Prometheus configuration file:\r\n```\r\n    global:\r\n      scrape_interval:     15s\r\n      evaluation_interval: 15s\r\n\r\n    scrape_configs:\r\n    - job_name: 'netdata'\r\n      metrics_path: '/api/v1/allmetrics?format=prometheus'\r\n      relabel_configs:\r\n      - source_labels: ['__address__']\r\n        modulus: 4\r\n        target_label: __tmp_hash\r\n        action: hashmod\r\n      - source_labels: ['__tmp_hash']\r\n        regex: '^0$'\r\n        action: keep\r\n      - source_labels: ['env']\r\n        regex: '^(dev|stg)$'\r\n        action: keep\r\n      - source_labels: ['__address__']\r\n        regex: '^([^:]+):\\d+'\r\n        target_label: 'fqdn'\r\n        replacement: '$1'\r\n      - source_labels: ['fqdn']\r\n        regex: '^([^\\.]+)\\..+$'\r\n        target_label: 'hostname'\r\n        replacement: '$1'\r\n      - source_labels: ['hostname']\r\n        regex: '^([^.]+\\d)(\\d\\d)(z|zd|c)?'\r\n        target_label: 'host_group'\r\n        replacement: '$1'\r\n      file_sd_configs:\r\n      - files:\r\n        - /etc/prometheus/conf.d/netdata.yml\r\n\r\n    - job_name: 'prometheus_exporter'\r\n      relabel_configs:\r\n      - source_labels: ['__meta_consul_address']\r\n        modulus: 4\r\n        target_label: __tmp_hash\r\n        action: hashmod\r\n      - source_labels: ['__tmp_hash']\r\n        regex: '^0$'\r\n        action: keep\r\n      - source_labels: [__meta_consul_tags]\r\n        regex: '.*,type-([^,]+),.*'\r\n        replacement: '$1'\r\n        target_label: job\r\n      - source_labels: [__meta_consul_node]\r\n        target_label: fqdn\r\n      - source_labels: [fqdn]\r\n        regex: '^([^\\.]+)\\..+$'\r\n        replacement: '$1'\r\n        target_label: 'hostname'\r\n      - source_labels: [__meta_consul_metadata_env]\r\n        target_label: 'env'\r\n      - source_labels: [__meta_consul_metadata_host_group]\r\n        target_label: 'host_group'\r\n      consul_sd_configs:\r\n      - server: localhost:8500\r\n        services:\r\n        - prometheus_exporter\r\n```\r\n\r\n* Logs:\r\n```\r\nJun 12 16:09:41  prometheus[12720]: level=info ts=2018-06-12T07:09:41.437914536Z caller=main.go:222 msg=\"Starting Prometheus\" version=\"(version=2.3.0, branch=HEAD, revision=290d71791a507a5057b9a099c9d48703d86dc941)\"\r\nJun 12 16:09:41  prometheus[12720]: level=info ts=2018-06-12T07:09:41.437996743Z caller=main.go:223 build_context=\"(go=go1.10.2, user=root@d539e167976a, date=20180607-08:46:54)\"\r\nJun 12 16:09:41  prometheus[12720]: level=info ts=2018-06-12T07:09:41.438016309Z caller=main.go:224 host_details=\"(Linux 3.10.0-862.el7.x86_64 #1 SMP Fri Apr 20 16:44:24 UTC 2018 x86_64  (none))\"\r\nJun 12 16:09:41  prometheus[12720]: level=info ts=2018-06-12T07:09:41.438031205Z caller=main.go:225 fd_limits=\"(soft=1006500, hard=1006500)\"\r\nJun 12 16:09:41  prometheus[12720]: level=info ts=2018-06-12T07:09:41.439396398Z caller=main.go:514 msg=\"Starting TSDB ...\"\r\nJun 12 16:09:41  prometheus[12720]: level=info ts=2018-06-12T07:09:41.439664658Z caller=web.go:426 component=web msg=\"Start listening for connections\" address=0.0.0.0:9090\r\nJun 12 16:09:41  prometheus[12720]: level=info ts=2018-06-12T07:09:41.443322109Z caller=repair.go:39 component=tsdb msg=\"found healthy block\" mint=1528768800000 maxt=1528776000000 ulid=01CFS5QNSYDD4V91EHVGXHHPC2\r\nJun 12 16:09:41  prometheus[12720]: level=info ts=2018-06-12T07:09:41.444473845Z caller=repair.go:39 component=tsdb msg=\"found healthy block\" mint=1528776000000 maxt=1528783200000 ulid=01CFSCKD20V0X9Z619CTJG0HC1\r\nJun 12 16:09:55  prometheus[12720]: level=info ts=2018-06-12T07:09:55.243619665Z caller=main.go:524 msg=\"TSDB started\"\r\nJun 12 16:09:55  prometheus[12720]: level=info ts=2018-06-12T07:09:55.243684491Z caller=main.go:603 msg=\"Loading configuration file\" filename=/etc/prometheus/prometheus.yml\r\nJun 12 16:09:55  prometheus[12720]: level=info ts=2018-06-12T07:09:55.244611863Z caller=main.go:500 msg=\"Server is ready to receive web requests.\"\r\nJun 12 16:09:55  prometheus[12720]: level=error ts=2018-06-12T07:09:55.246268501Z caller=consul.go:258 component=\"discovery manager scrape\" discovery=consul msg=\"Error retrieving datacenter name\" err=\"Unexpected response code: 403 (Permission denied)\"\r\nJun 12 16:10:10  prometheus[12720]: level=error ts=2018-06-12T07:10:10.246998917Z caller=consul.go:258 component=\"discovery manager scrape\" discovery=consul msg=\"Error retrieving datacenter name\" err=\"Unexpected response code: 403 (Permission denied)\"\r\nJun 12 16:10:25  prometheus[12720]: level=error ts=2018-06-12T07:10:25.24927276Z caller=consul.go:258 component=\"discovery manager scrape\" discovery=consul msg=\"Error retrieving datacenter name\" err=\"Unexpected response code: 403 (Permission denied)\"\r\nJun 12 16:10:40  prometheus[12720]: level=error ts=2018-06-12T07:10:40.250021419Z caller=consul.go:258 component=\"discovery manager scrape\" discovery=consul msg=\"Error retrieving datacenter name\" err=\"Unexpected response code: 403 (Permission denied)\"\r\nJun 12 16:10:55  prometheus[12720]: level=error ts=2018-06-12T07:10:55.251577846Z caller=consul.go:258 component=\"discovery manager scrape\" discovery=consul msg=\"Error retrieving datacenter name\" err=\"Unexpected response code: 403 (Permission denied)\"\r\nJun 12 16:11:10  prometheus[12720]: level=error ts=2018-06-12T07:11:10.252305118Z caller=consul.go:258 component=\"discovery manager scrape\" discovery=consul msg=\"Error retrieving datacenter name\" err=\"Unexpected response code: 403 (Permission denied)\"\r\nJun 12 16:11:25  prometheus[12720]: level=error ts=2018-06-12T07:11:25.253499684Z caller=consul.go:258 component=\"discovery manager scrape\" discovery=consul msg=\"Error retrieving datacenter name\" err=\"Unexpected response code: 403 (Permission denied)\"\r\nJun 12 16:11:40  prometheus[12720]: level=error ts=2018-06-12T07:11:40.254119744Z caller=consul.go:258 component=\"discovery manager scrape\" discovery=consul msg=\"Error retrieving datacenter name\" err=\"Unexpected response code: 403 (Permission denied)\"\r\nJun 12 16:11:55  prometheus[12720]: level=error ts=2018-06-12T07:11:55.255685656Z caller=consul.go:258 component=\"discovery manager scrape\" discovery=consul msg=\"Error retrieving datacenter name\" err=\"Unexpected response code: 403 (Permission denied)\"\r\nJun 12 16:12:10  prometheus[12720]: level=error ts=2018-06-12T07:12:10.257246775Z caller=consul.go:258 component=\"discovery manager scrape\" discovery=consul msg=\"Error retrieving datacenter name\" err=\"Unexpected response code: 403 (Permission denied)\"\r\nJun 12 16:12:25  prometheus[12720]: level=error ts=2018-06-12T07:12:25.258391992Z caller=consul.go:258 component=\"discovery manager scrape\" discovery=consul msg=\"Error retrieving datacenter name\" err=\"Unexpected response code: 403 (Permission denied)\"\r\nJun 12 16:12:40  prometheus[12720]: level=error ts=2018-06-12T07:12:40.259062034Z caller=consul.go:258 component=\"discovery manager scrape\" discovery=consul msg=\"Error retrieving datacenter name\" err=\"Unexpected response code: 403 (Permission denied)\"\r\nJun 12 16:12:55  prometheus[12720]: level=error ts=2018-06-12T07:12:55.261606034Z caller=consul.go:258 component=\"discovery manager scrape\" discovery=consul msg=\"Error retrieving datacenter name\" err=\"Unexpected response code: 403 (Permission denied)\"\r\nJun 12 16:13:10  prometheus[12720]: level=error ts=2018-06-12T07:13:10.262981208Z caller=consul.go:258 component=\"discovery manager scrape\" discovery=consul msg=\"Error retrieving datacenter name\" err=\"Unexpected response code: 403 (Permission denied)\"\r\nJun 12 16:13:25  prometheus[12720]: level=error ts=2018-06-12T07:13:25.264128827Z caller=consul.go:258 component=\"discovery manager scrape\" discovery=consul msg=\"Error retrieving datacenter name\" err=\"Unexpected response code: 403 (Permission denied)\"\r\nJun 12 16:13:40  prometheus[12720]: level=error ts=2018-06-12T07:13:40.264831005Z caller=consul.go:258 component=\"discovery manager scrape\" discovery=consul msg=\"Error retrieving datacenter name\" err=\"Unexpected response code: 403 (Permission denied)\"\r\nJun 12 16:13:55  prometheus[12720]: level=error ts=2018-06-12T07:13:55.2664757Z caller=consul.go:258 component=\"discovery manager scrape\" discovery=consul msg=\"Error retrieving datacenter name\" err=\"Unexpected response code: 403 (Permission denied)\"\r\nJun 12 16:14:10  prometheus[12720]: level=error ts=2018-06-12T07:14:10.268095616Z caller=consul.go:258 component=\"discovery manager scrape\" discovery=consul msg=\"Error retrieving datacenter name\" err=\"Unexpected response code: 403 (Permission denied)\"\r\nJun 12 16:14:25  prometheus[12720]: level=error ts=2018-06-12T07:14:25.269409927Z caller=consul.go:258 component=\"discovery manager scrape\" discovery=consul msg=\"Error retrieving datacenter name\" err=\"Unexpected response code: 403 (Permission denied)\"\r\nJun 12 16:14:40  prometheus[12720]: level=error ts=2018-06-12T07:14:40.270111576Z caller=consul.go:258 component=\"discovery manager scrape\" discovery=consul msg=\"Error retrieving datacenter name\" err=\"Unexpected response code: 403 (Permission denied)\"\r\nJun 12 16:14:55  prometheus[12720]: level=error ts=2018-06-12T07:14:55.27160297Z caller=consul.go:258 component=\"discovery manager scrape\" discovery=consul msg=\"Error retrieving datacenter name\" err=\"Unexpected response code: 403 (Permission denied)\"\r\nJun 12 16:15:10  prometheus[12720]: level=error ts=2018-06-12T07:15:10.272358567Z caller=consul.go:258 component=\"discovery manager scrape\" discovery=consul msg=\"Error retrieving datacenter name\" err=\"Unexpected response code: 403 (Permission denied)\"\r\nJun 12 16:15:25  prometheus[12720]: level=error ts=2018-06-12T07:15:25.273978302Z caller=consul.go:258 component=\"discovery manager scrape\" discovery=consul msg=\"Error retrieving datacenter name\" err=\"Unexpected response code: 403 (Permission denied)\"\r\nJun 12 16:15:40  prometheus[12720]: level=error ts=2018-06-12T07:15:40.274724767Z caller=consul.go:258 component=\"discovery manager scrape\" discovery=consul msg=\"Error retrieving datacenter name\" err=\"Unexpected response code: 403 (Permission denied)\"\r\nJun 12 16:15:55  prometheus[12720]: level=error ts=2018-06-12T07:15:55.277669968Z caller=consul.go:258 component=\"discovery manager scrape\" discovery=consul msg=\"Error retrieving datacenter name\" err=\"Unexpected response code: 403 (Permission denied)\"\r\nJun 12 16:16:10  prometheus[12720]: level=error ts=2018-06-12T07:16:10.278402869Z caller=consul.go:258 component=\"discovery manager scrape\" discovery=consul msg=\"Error retrieving datacenter name\" err=\"Unexpected response code: 403 (Permission denied)\"\r\nJun 12 16:16:25  prometheus[12720]: level=error ts=2018-06-12T07:16:25.280993084Z caller=consul.go:258 component=\"discovery manager scrape\" discovery=consul msg=\"Error retrieving datacenter name\" err=\"Unexpected response code: 403 (Permission denied)\"\r\nJun 12 16:16:40  prometheus[12720]: level=error ts=2018-06-12T07:16:40.281668811Z caller=consul.go:258 component=\"discovery manager scrape\" discovery=consul msg=\"Error retrieving datacenter name\" err=\"Unexpected response code: 403 (Permission denied)\"\r\nJun 12 16:23:57  prometheus[12720]: fatal error: runtime: out of memory\r\nJun 12 16:23:57  prometheus[12720]: runtime stack:\r\nJun 12 16:23:57  prometheus[12720]: runtime.throw(0x1c859d7, 0x16)\r\nJun 12 16:23:57  prometheus[12720]: /usr/local/go/src/runtime/panic.go:616 +0x81\r\nJun 12 16:23:57  prometheus[12720]: runtime.sysMap(0xc6aef40000, 0x200000000, 0x0, 0x2b205f8)\r\nJun 12 16:23:57  prometheus[12720]: /usr/local/go/src/runtime/mem_linux.go:216 +0x20a\r\nJun 12 16:23:57  prometheus[12720]: runtime.(*mheap).sysAlloc(0x2b06ec0, 0x200000000, 0x470c90)\r\nJun 12 16:23:57  prometheus[12720]: /usr/local/go/src/runtime/malloc.go:470 +0xd4\r\nJun 12 16:23:57  prometheus[12720]: runtime.(*mheap).grow(0x2b06ec0, 0x100000, 0x0)\r\nJun 12 16:23:57  prometheus[12720]: /usr/local/go/src/runtime/mheap.go:907 +0x60\r\nJun 12 16:23:57  prometheus[12720]: runtime.(*mheap).allocSpanLocked(0x2b06ec0, 0x100000, 0x2b20608, 0xc41d6a48ad)\r\nJun 12 16:23:57  prometheus[12720]: /usr/local/go/src/runtime/mheap.go:820 +0x301\r\nJun 12 16:23:57  prometheus[12720]: runtime.(*mheap).alloc_m(0x2b06ec0, 0x100000, 0x101, 0x70)\r\nJun 12 16:23:57  prometheus[12720]: /usr/local/go/src/runtime/mheap.go:686 +0x118\r\nJun 12 16:23:57  prometheus[12720]: runtime.(*mheap).alloc.func1()\r\nJun 12 16:23:57  prometheus[12720]: /usr/local/go/src/runtime/mheap.go:753 +0x4d\r\nJun 12 16:23:57  prometheus[12720]: runtime.(*mheap).alloc(0x2b06ec0, 0x100000, 0xc420010101, 0x4140bc)\r\nJun 12 16:23:58  prometheus[12720]: /usr/local/go/src/runtime/mheap.go:752 +0x8a\r\nJun 12 16:23:58  prometheus[12720]: runtime.largeAlloc(0x200000000, 0x101, 0xc420054ec0)\r\nJun 12 16:23:58  prometheus[12720]: /usr/local/go/src/runtime/malloc.go:826 +0x94\r\nJun 12 16:23:58  prometheus[12720]: runtime.mallocgc.func1()\r\nJun 12 16:23:58  prometheus[12720]: /usr/local/go/src/runtime/malloc.go:721 +0x46\r\nJun 12 16:23:58  prometheus[12720]: runtime.systemstack(0x0)\r\nJun 12 16:23:58  prometheus[12720]: /usr/local/go/src/runtime/asm_amd64.s:409 +0x79\r\nJun 12 16:23:58  prometheus[12720]: runtime.mstart()\r\nJun 12 16:23:58  prometheus[12720]: /usr/local/go/src/runtime/proc.go:1175\r\nJun 12 16:23:58  prometheus[12720]: goroutine 112619 [running]:\r\nJun 12 16:23:58  prometheus[12720]: runtime.systemstack_switch()\r\nJun 12 16:23:58  prometheus[12720]: /usr/local/go/src/runtime/asm_amd64.s:363 fp=0xc4ad17b0e0 sp=0xc4ad17b0d8 pc=0x457300\r\nJun 12 16:23:58  prometheus[12720]: runtime.mallocgc(0x200000000, 0x1a15ee0, 0x152a101, 0xc4ad17b1c0)\r\nJun 12 16:23:58  prometheus[12720]: /usr/local/go/src/runtime/malloc.go:720 +0x8a2 fp=0xc4ad17b180 sp=0xc4ad17b0e0 pc=0x410692\r\nJun 12 16:23:58  prometheus[12720]: runtime.makeslice(0x1a15ee0, 0x20000000, 0x20000000, 0x1565384, 0xc49296e6c0, 0x163f2deaa58)\r\nJun 12 16:23:58  prometheus[12720]: /usr/local/go/src/runtime/slice.go:61 +0x77 fp=0xc4ad17b1b0 sp=0xc4ad17b180 pc=0x441497\r\nJun 12 16:23:58  prometheus[12720]: github.com/prometheus/prometheus/storage.(*sampleRing).add(0xc492968870, 0x163f2deaa58, 0x7ff0000000000002)\r\nJun 12 16:23:58  prometheus[12720]: /go/src/github.com/prometheus/prometheus/storage/buffer.go:186 +0x10b fp=0xc4ad17b220 sp=0xc4ad17b1b0 pc=0x152696b\r\nJun 12 16:23:58  prometheus[12720]: github.com/prometheus/prometheus/storage.(*BufferedSeriesIterator).Next(0xc492945d10, 0x163f2db0001)\r\nJun 12 16:23:58  prometheus[12720]: /go/src/github.com/prometheus/prometheus/storage/buffer.go:96 +0x51 fp=0xc4ad17b248 sp=0xc4ad17b220 pc=0x1526641\r\nJun 12 16:23:58  prometheus[12720]: github.com/prometheus/prometheus/storage.(*BufferedSeriesIterator).Seek(0xc492945d10, 0x163f2def39e, 0x493e0)\r\nJun 12 16:23:58  prometheus[12720]: /go/src/github.com/prometheus/prometheus/storage/buffer.go:80 +0x53 fp=0xc4ad17b270 sp=0xc4ad17b248 pc=0x1526523\r\nJun 12 16:23:58  prometheus[12720]: github.com/prometheus/prometheus/web.(*Handler).federation(0xc420229500, 0x1de1240, 0xc48200ab20, 0xc4459f1100)\r\nJun 12 16:23:58  prometheus[12720]: /go/src/github.com/prometheus/prometheus/web/federate.go:97 +0xb6b fp=0xc4ad17b758 sp=0xc4ad17b270 pc=0x16ffe8b\r\nJun 12 16:23:58  prometheus[12720]: github.com/prometheus/prometheus/web.(*Handler).(github.com/prometheus/prometheus/web.federation)-fm(0x1de1240, 0xc48200ab20, 0xc4459f1100)\r\nJun 12 16:23:58  prometheus[12720]: /go/src/github.com/prometheus/prometheus/web/web.go:272 +0x48 fp=0xc4ad17b788 sp=0xc4ad17b758 pc=0x170af18\r\nJun 12 16:23:58  prometheus[12720]: net/http.HandlerFunc.ServeHTTP(0xc42034b560, 0x1de1240, 0xc48200ab20, 0xc4459f1100)\r\nJun 12 16:23:58  prometheus[12720]: /usr/local/go/src/net/http/server.go:1947 +0x44 fp=0xc4ad17b7b0 sp=0xc4ad17b788 pc=0x68c674\r\nJun 12 16:23:58  prometheus[12720]: github.com/prometheus/prometheus/util/httputil.CompressionHandler.ServeHTTP(0x1dd6c00, 0xc42034b560, 0x7f5c1a56f0a0, 0xc4459db720, 0xc4459f1100)\r\nJun 12 16:23:58  prometheus[12720]: /go/src/github.com/prometheus/prometheus/util/httputil/compression.go:90 +0x7c fp=0xc4ad17b7e8 sp=0xc4ad17b7b0 pc=0x169e39c\r\nJun 12 16:23:58  prometheus[12720]: github.com/prometheus/prometheus/util/httputil.(CompressionHandler).ServeHTTP-fm(0x7f5c1a56f0a0, 0xc4459db720, 0xc4459f1100)\r\nJun 12 16:23:58  prometheus[12720]: /go/src/github.com/prometheus/prometheus/web/web.go:273 +0x57 fp=0xc4ad17b820 sp=0xc4ad17b7e8 pc=0x16e2fc7\r\nJun 12 16:23:58  prometheus[12720]: github.com/prometheus/prometheus/web.(*Handler).testReady.func1(0x7f5c1a56f0a0, 0xc4459db720, 0xc4459f1100)\r\nJun 12 16:23:58  prometheus[12720]: /go/src/github.com/prometheus/prometheus/web/web.go:401 +0x55 fp=0xc4ad17b880 sp=0xc4ad17b820 pc=0x17097a5\r\nJun 12 16:23:58  prometheus[12720]: net/http.HandlerFunc.ServeHTTP(0xc42052d600, 0x7f5c1a56f0a0, 0xc4459db720, 0xc4459f1100)\r\nJun 12 16:23:58  prometheus[12720]: /usr/local/go/src/net/http/server.go:1947 +0x44 fp=0xc4ad17b8a8 sp=0xc4ad17b880 pc=0x68c674\r\nJun 12 16:23:58  prometheus[12720]: github.com/prometheus/prometheus/vendor/github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerResponseSize.func1(0x1de3480, 0xc48200ab00, 0xc4459f1100)\r\nJun 12 16:23:58  prometheus[12720]: /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/client_golang/prometheus/promhttp/instrument_server.go:196 +0xed fp=0xc4ad17b930 sp=0xc4ad17b8a8 pc=0x1689c3d\r\nJun 12 16:23:58  prometheus[12720]: net/http.HandlerFunc.ServeHTTP(0xc4204c34a0, 0x1de3480, 0xc48200ab00, 0xc4459f1100)\r\nJun 12 16:23:58  prometheus[12720]: /usr/local/go/src/net/http/server.go:1947 +0x44 fp=0xc4ad17b958 sp=0xc4ad17b930 pc=0x68c674\r\nJun 12 16:23:58  prometheus[12720]: github.com/prometheus/prometheus/vendor/github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2(0x1de3480, 0xc48200ab00, 0xc4459f1100)\r\nJun 12 16:23:58  prometheus[12720]: /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/client_golang/prometheus/promhttp/instrument_server.go:76 +0xb5 fp=0xc4ad17b9e0 sp=0xc4ad17b958 pc=0x16897e5\r\nJun 12 16:23:58  prometheus[12720]: github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route.(*Router).handle.func1(0x1de3480, 0xc48200ab00, 0xc4459f1000, 0x0, 0x0, 0x0)\r\nJun 12 16:23:58  prometheus[12720]: /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route/route.go:60 +0x222 fp=0xc4ad17ba98 sp=0xc4ad17b9e0 pc=0x169ce02\r\nJun 12 16:23:58  prometheus[12720]: github.com/prometheus/prometheus/vendor/github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc4203824c0, 0x1de3480, 0xc48200ab00, 0xc4459f1000)\r\nJun 12 16:23:58  prometheus[12720]: /go/src/github.com/prometheus/prometheus/vendor/github.com/julienschmidt/httprouter/router.go:299 +0x6d1 fp=0xc4ad17bb78 sp=0xc4ad17ba98 pc=0x1698de1\r\nJun 12 16:23:58  prometheus[12720]: github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route.(*Router).ServeHTTP(0xc42052c640, 0x1de3480, 0xc48200ab00, 0xc4459f1000)\r\nJun 12 16:23:58  prometheus[12720]: /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route/route.go:98 +0x4c fp=0xc4ad17bba8 sp=0xc4ad17bb78 pc=0x169caac\r\nJun 12 16:23:58  prometheus[12720]: net/http.(*ServeMux).ServeHTTP(0xc42097e330, 0x1de3480, 0xc48200ab00, 0xc4459f1000)\r\nJun 12 16:23:58  prometheus[12720]: /usr/local/go/src/net/http/server.go:2337 +0x130 fp=0xc4ad17bbe8 sp=0xc4ad17bba8 pc=0x68e3e0\r\nJun 12 16:23:58  prometheus[12720]: github.com/prometheus/prometheus/web.secureHeadersMiddleware.func1(0x1de3480, 0xc48200ab00, 0xc4459f1000)\r\nJun 12 16:23:58  prometheus[12720]: /go/src/github.com/prometheus/prometheus/web/web.go:81 +0x195 fp=0xc4ad17bc30 sp=0xc4ad17bbe8 pc=0x17090e5\r\nJun 12 16:23:58  prometheus[12720]: net/http.HandlerFunc.ServeHTTP(0xc4204d5f20, 0x1de3480, 0xc48200ab00, 0xc4459f1000)\r\nJun 12 16:23:58  prometheus[12720]: /usr/local/go/src/net/http/server.go:1947 +0x44 fp=0xc4ad17bc58 sp=0xc4ad17bc30 pc=0x68c674\r\nJun 12 16:23:58  prometheus[12720]: github.com/prometheus/prometheus/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.Middleware.func2(0x1dead80, 0xc4203f3dc0, 0xc4459f0f00)\r\nJun 12 16:23:58  prometheus[12720]: /go/src/github.com/prometheus/prometheus/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:74 +0x3ab fp=0xc4ad17bd30 sp=0xc4ad17bc58 pc=0x168217b\r\nJun 12 16:23:58  prometheus[12720]: net/http.HandlerFunc.ServeHTTP(0xc4204c24e0, 0x1dead80, 0xc4203f3dc0, 0xc4459f0f00)\r\nJun 12 16:23:58  prometheus[12720]: /usr/local/go/src/net/http/server.go:1947 +0x44 fp=0xc4ad17bd58 sp=0xc4ad17bd30 pc=0x68c674\r\nJun 12 16:23:58  prometheus[12720]: net/http.serverHandler.ServeHTTP(0xc420532a90, 0x1dead80, 0xc4203f3dc0, 0xc4459f0f00)\r\nJun 12 16:23:58  prometheus[12720]: /usr/local/go/src/net/http/server.go:2694 +0xbc fp=0xc4ad17bd88 sp=0xc4ad17bd58 pc=0x68f48c\r\nJun 12 16:23:58  prometheus[12720]: net/http.(*conn).serve(0xc4459e9220, 0x1decd80, 0xc4947f99c0)\r\nJun 12 16:23:58  prometheus[12720]: /usr/local/go/src/net/http/server.go:1830 +0x651 fp=0xc4ad17bfc8 sp=0xc4ad17bd88 pc=0x68b691\r\nJun 12 16:23:58  prometheus[12720]: runtime.goexit()\r\nJun 12 16:23:58  prometheus[12720]: /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc4ad17bfd0 sp=0xc4ad17bfc8 pc=0x459d11\r\nJun 12 16:23:58  prometheus[12720]: created by net/http.(*Server).Serve\r\nJun 12 16:23:58  prometheus[12720]: /usr/local/go/src/net/http/server.go:2795 +0x27b\r\nJun 12 16:23:58  prometheus[12720]: goroutine 1 [chan receive, 14 minutes]:\r\nJun 12 16:23:58  prometheus[12720]: github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run(0xc4209a5c90, 0xc4206cc070, 0x8)\r\nJun 12 16:23:58  prometheus[12720]: /go/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:43 +0xec\r\nJun 12 16:23:58  prometheus[12720]: main.main()\r\n...\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/comments/396882457","html_url":"https://github.com/prometheus/prometheus/issues/4253#issuecomment-396882457","issue_url":"https://api.github.com/repos/prometheus/prometheus/issues/4253","id":396882457,"node_id":"MDEyOklzc3VlQ29tbWVudDM5Njg4MjQ1Nw==","user":{"login":"krasi-georgiev","id":8903888,"node_id":"MDQ6VXNlcjg5MDM4ODg=","avatar_url":"https://avatars0.githubusercontent.com/u/8903888?v=4","gravatar_id":"","url":"https://api.github.com/users/krasi-georgiev","html_url":"https://github.com/krasi-georgiev","followers_url":"https://api.github.com/users/krasi-georgiev/followers","following_url":"https://api.github.com/users/krasi-georgiev/following{/other_user}","gists_url":"https://api.github.com/users/krasi-georgiev/gists{/gist_id}","starred_url":"https://api.github.com/users/krasi-georgiev/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/krasi-georgiev/subscriptions","organizations_url":"https://api.github.com/users/krasi-georgiev/orgs","repos_url":"https://api.github.com/users/krasi-georgiev/repos","events_url":"https://api.github.com/users/krasi-georgiev/events{/privacy}","received_events_url":"https://api.github.com/users/krasi-georgiev/received_events","type":"User","site_admin":false},"created_at":"2018-06-13T09:55:33Z","updated_at":"2018-06-13T09:55:33Z","author_association":"MEMBER","body":"@Wing924 this doesn't show  high RAM usage. Can you take the profile when at the time it replicates the problem or before it is OOM killed."}}
{"actor":{"display_login":"krasi-georgiev"}}