{"action":"created","issue":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/1786","repository_url":"https://api.github.com/repos/prometheus/prometheus","labels_url":"https://api.github.com/repos/prometheus/prometheus/issues/1786/labels{/name}","comments_url":"https://api.github.com/repos/prometheus/prometheus/issues/1786/comments","events_url":"https://api.github.com/repos/prometheus/prometheus/issues/1786/events","html_url":"https://github.com/prometheus/prometheus/issues/1786","id":163637052,"number":1786,"title":"Fatal Error : concurrent map read and map write","user":{"login":"PrFalken","id":334048,"avatar_url":"https://avatars.githubusercontent.com/u/334048?v=3","gravatar_id":"","url":"https://api.github.com/users/PrFalken","html_url":"https://github.com/PrFalken","followers_url":"https://api.github.com/users/PrFalken/followers","following_url":"https://api.github.com/users/PrFalken/following{/other_user}","gists_url":"https://api.github.com/users/PrFalken/gists{/gist_id}","starred_url":"https://api.github.com/users/PrFalken/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PrFalken/subscriptions","organizations_url":"https://api.github.com/users/PrFalken/orgs","repos_url":"https://api.github.com/users/PrFalken/repos","events_url":"https://api.github.com/users/PrFalken/events{/privacy}","received_events_url":"https://api.github.com/users/PrFalken/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/prometheus/prometheus/labels/kind/bug","name":"kind/bug","color":"ff0000"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2016-07-04T09:25:25Z","updated_at":"2016-07-04T12:18:42Z","closed_at":null,"body":"Runing prometheus for a while, 1.2M metrics.\r\nThe server randomly crashes with a huge trace. Occurs about every 10 hours or so.\r\nNote : This is a federation server but the crash occured the same way when there was a single server scraping everything\r\n\r\nprocess runs in an RKT container on Linux CoreOS (Linux 4.5.0-coreos-r1 x86_64)\r\n\r\n* Prometheus version: \r\nprometheus, version 0.19.2 (branch: master, revision: 23ca13c)\r\n  build user:       root@134dc6bbc274\r\n  build date:       20160529-18:58:00\r\n  go version:       go1.6.2\r\n\r\n* Prometheus configuration file:\r\n```\r\nglobal:\r\n  evaluation_interval: 20s\r\n  scrape_interval:     20s\r\n  scrape_timeout:      20s\r\n\r\n\r\nscrape_configs:\r\n  - job_name: 'prometheus'\r\n    target_groups:\r\n      - targets: ['127.0.0.1:9090']\r\n\r\n  - job_name: federation\r\n    metrics_path: /federate\r\n    scrape_interval: 20s\r\n    honor_labels: true\r\n    nerve_sd_configs:\r\n      - servers:\r\n        - zk1\r\n        - zk2\r\n        - zk3\r\n        paths:\r\n          - /services/prometheus/prometheus\r\n    params:\r\n      'match[]':\r\n         - '{env=\"prod\"}'\r\n\r\n    relabel_configs:\r\n      - source_labels: [__meta_nerve_endpoint_name]\r\n        target_label: name\r\n\r\n      - source_labels: [__meta_nerve_endpoint_name]\r\n        regex: ([a-z\\-]+).*\r\n        replacement: ${1}\r\n        target_label: service\r\n\r\n      - source_labels: [__meta_nerve_endpoint_name]\r\n        regex: (r\\d+-srv\\d+)\r\n        replacement: host\r\n        target_label: service\r\n\r\n      - source_labels: [job]\r\n        regex: discovery_([a-z0-9\\-]+)\r\n        replacement: ${1}\r\n        target_label: env\r\n\r\nrule_files:\r\n  - /etc/prometheus/rules/rules\r\n```\r\n\r\n* Logs:\r\n```\r\nfatal error: concurrent map read and map write\r\ngoroutine 332096 [running]:\r\nruntime.throw(0x1001ce0, 0x21)\r\n/usr/local/go/src/runtime/panic.go:547 +0x90 fp=0xc98e9f7100 sp=0xc98e9f70e8\r\nruntime.mapaccess1_faststr(0xe68260, 0xc82bcaf620, 0xc9097613c2, 0x3, 0xc8f7efe1a0)\r\n/usr/local/go/src/runtime/hashmap_fast.go:202 +0x5b fp=0xc98e9f7160 sp=0xc98e9f7100\r\ngithub.com/prometheus/prometheus/vendor/github.com/prometheus/common/model.SignatureForLabels(0xc82bcaf620, 0xc9a3a97180, 0x3, 0x4, 0xc86e5766af112801)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/model/signature.go:111 +0x1bb fp=0xc98e9f7220 sp=0xc98e9f7160\r\ngithub.com/prometheus/prometheus/promql.(*evaluator).aggregation(0xc9415590c0, 0x28, 0xc9a3a97180, 0x3, 0x4, 0x0, 0xcab131c000, 0xbd00, 0xc000, 0x0, ...)\r\n/go/src/github.com/prometheus/prometheus/promql/engine.go:1082 +0xc7c fp=0xc98e9f7450 sp=0xc98e9f7220\r\ngithub.com/prometheus/prometheus/promql.(*evaluator).eval(0xc9415590c0, 0x7f8775541548, 0xc9a3a971c0, 0x0, 0x0)\r\n/go/src/github.com/prometheus/prometheus/promql/engine.go:613 +0x744 fp=0xc98e9f7640 sp=0xc98e9f7450\r\ngithub.com/prometheus/prometheus/promql.(*evaluator).Eval(0xc9415590c0, 0x7f8775541548, 0xc9a3a971c0, 0x0, 0x0, 0x0, 0x0)\r\n/go/src/github.com/prometheus/prometheus/promql/engine.go:599 +0x9a fp=0xc98e9f7680 sp=0xc98e9f7640\r\ngithub.com/prometheus/prometheus/promql.(*Engine).execEvalStmt(0xc8202c8c80, 0x7f876d67a320, 0xc909761530, 0xc9a3a97200, 0xc9097614a0, 0x0, 0x0, 0x0, 0x0)\r\n/go/src/github.com/prometheus/prometheus/promql/engine.go:404 +0x5e6 fp=0xc98e9f79f0 sp=0xc98e9f7680\r\ngithub.com/prometheus/prometheus/promql.(*Engine).exec(0xc8202c8c80, 0xc9a3a97200, 0x0, 0x0, 0x0, 0x0)\r\n/go/src/github.com/prometheus/prometheus/promql/engine.go:357 +0x4ca fp=0xc98e9f7b08 sp=0xc98e9f79f0\r\ngithub.com/prometheus/prometheus/promql.(*query).Exec(0xc9a3a97200, 0xc9097613b0)\r\n/go/src/github.com/prometheus/prometheus/promql/engine.go:195 +0x2e fp=0xc98e9f7b68 sp=0xc98e9f7b08\r\ngithub.com/prometheus/prometheus/rules.RecordingRule.eval(0xc900f476c0, 0x8, 0x7f8775541548, 0xca2a8c3980, 0x0, 0x155b4feb074, 0xc8202c8c80, 0x0, 0x0, 0x0, ...)\r\n/go/src/github.com/prometheus/prometheus/rules/recording.go:55 +0x130 fp=0xc98e9f7d28 sp=0xc98e9f7b68\r\ngithub.com/prometheus/prometheus/rules.(*RecordingRule).eval(0xc94fcd9170, 0x155b4feb074, 0xc8202c8c80, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n<autogenerated>:7 +0xd8 fp=0xc98e9f7d90 sp=0xc98e9f7d28\r\ngithub.com/prometheus/prometheus/rules.(*Group).eval.func1(0xc9295fb220, 0xf1bf30, 0x9, 0x155b4feb074, 0xc99d667590, 0x7f8775541a18, 0xc94fcd9170)\r\n/go/src/github.com/prometheus/prometheus/rules/manager.go:260 +0x1cc fp=0xc98e9f7f58 sp=0xc98e9f7d90\r\nruntime.goexit()\r\n/usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc98e9f7f60 sp=0xc98e9f7f58\r\ncreated by github.com/prometheus/prometheus/rules.(*Group).eval\r\n/go/src/github.com/prometheus/prometheus/rules/manager.go:298 +0x170\r\ngoroutine 1 [select, 628 minutes]:\r\nmain.Main(0x0)\r\n/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:186 +0x1ed5\r\nmain.main()\r\n/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:41 +0x18\r\ngoroutine 22 [syscall, 3 minutes]:\r\nos/signal.signal_recv(0x7f8775629078)\r\n/usr/local/go/src/runtime/sigqueue.go:116 +0x132\r\nos/signal.loop()\r\n/usr/local/go/src/os/signal/signal_unix.go:22 +0x18\r\ncreated by os/signal.init.1\r\n/usr/local/go/src/os/signal/signal_unix.go:28 +0x37\r\ngoroutine 217 [select, 628 minutes, locked to thread]:\r\nruntime.gopark(0x1101ee8, 0xc82048af28, 0xefe178, 0x6, 0x18, 0x2)\r\n/usr/local/go/src/runtime/proc.go:262 +0x163\r\nruntime.selectgoImpl(0xc82048af28, 0x0, 0x18)\r\n/usr/local/go/src/runtime/select.go:392 +0xa67\r\nruntime.selectgo(0xc82048af28)\r\n/usr/local/go/src/runtime/select.go:215 +0x12\r\nruntime.ensureSigM.func1()\r\n/usr/local/go/src/runtime/signal1_unix.go:279 +0x358\r\nruntime.goexit()\r\n/usr/local/go/src/runtime/asm_amd64.s:1998 +0x1\r\ngoroutine 329917 [semacquire]:\r\nsync.runtime_Semacquire(0xc9295fb22c)\r\n/usr/local/go/src/runtime/sema.go:47 +0x26\r\nsync.(*WaitGroup).Wait(0xc9295fb220)\r\n/usr/local/go/src/sync/waitgroup.go:127 +0xb4\r\ngithub.com/prometheus/prometheus/rules.(*Group).eval(0xc99d667590)\r\n/go/src/github.com/prometheus/prometheus/rules/manager.go:300 +0x19d\r\ngithub.com/prometheus/prometheus/rules.(*Group).run.func1()\r\n/go/src/github.com/prometheus/prometheus/rules/manager.go:154 +0xbc\r\ngithub.com/prometheus/prometheus/rules.(*Group).run(0xc99d667590)\r\n/go/src/github.com/prometheus/prometheus/rules/manager.go:172 +0x25e\r\ngithub.com/prometheus/prometheus/rules.(*Manager).ApplyConfig.func1.1(0xc82032a480, 0xc99d667590)\r\n/go/src/github.com/prometheus/prometheus/rules/manager.go:468 +0x48\r\ncreated by github.com/prometheus/prometheus/rules.(*Manager).ApplyConfig.func1\r\n/go/src/github.com/prometheus/prometheus/rules/manager.go:469 +0xa0\r\ngoroutine 218 [select, 3 minutes]:\r\nmain.Main.func2(0xc82029be60, 0xc82029be00, 0xc82000a600, 0xc820086a40, 0x4, 0x4)\r\n/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:132 +0x135\r\ncreated by main.Main\r\n/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:138 +0x1292\r\ngoroutine 219 [select]:\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc8201408c0)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x29e\r\ncreated by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x270\r\ngoroutine 220 [select, 2 minutes]:\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc8200ce840)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x54d\r\ncreated by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x787\r\ngoroutine 221 [select]:\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc8200ce840)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14c\r\ncreated by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7a9\r\ngoroutine 222 [select, 2 minutes]:\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc8200ce840)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x7e0\r\ncreated by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9ad\r\ngoroutine 223 [select, 6 minutes]:\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc8200ce840)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x254\r\ncreated by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9cf\r\ngoroutine 224 [select, 641 minutes]:\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc8200ce840)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x1a3\r\ncreated by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:148 +0x9f1\r\ngoroutine 225 [select]:\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc820140a80)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x29e\r\ncreated by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x270\r\ngoroutine 226 [select, 2 minutes]:\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc8200cef20)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x54d\r\ncreated by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x787\r\ngoroutine 227 [select]:\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc8200cef20)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14c\r\ncreated by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7a9\r\ngoroutine 228 [select, 2 minutes]:\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc8200cef20)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x7e0\r\ncreated by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9ad\r\ngoroutine 229 [select, 11 minutes]:\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc8200cef20)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x254\r\ncreated by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9cf\r\ngoroutine 230 [select, 641 minutes]:\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc8200cef20)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x1a3\r\ncreated by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:148 +0x9f1\r\ngoroutine 231 [select]:\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc820140700)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x29e\r\ncreated by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x270\r\ngoroutine 232 [select]:\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc8200cec60)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x54d\r\ncreated by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x787\r\ngoroutine 233 [select]:\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc8200cec60)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14c\r\ncreated by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7a9\r\ngoroutine 234 [select]:\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).pauseCompaction(0xc8200cec60, 0xc9c86f7a40)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:608 +0xf2\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc8200cec60)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:756 +0x528\r\ncreated by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9ad\r\ngoroutine 235 [runnable]:\r\ngithub.com/prometheus/prometheus/vendor/github.com/golang/snappy.emitLiteral(0xcaf4bc40f8, 0x3a9d34f, 0x3a9d34f, 0xcb167340cf, 0x790da, 0x35aa2ab, 0x6a5f9)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/golang/snappy/encode.go:16\r\ngithub.com/prometheus/prometheus/vendor/github.com/golang/snappy.Encode(0xcaf480c000, 0x3e55447, 0x3e55447, 0xcb1637c000, 0xefc402, 0x396237a, 0x0, 0x0, 0x0)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/golang/snappy/encode.go:130 +0x489\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/table.(*Writer).writeBlock(0xc965186600, 0xc965186658, 0x2, 0x0, 0x0, 0x0, 0x0)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/table/writer.go:170 +0x1d2\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/table.(*Writer).finishBlock(0xc965186600, 0x0, 0x0)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/table/writer.go:221 +0x75\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/table.(*Writer).Append(0xc965186600, 0xcb5f5e4dc5, 0x10, 0x1536523b, 0xcb5f5e4dd5, 0xefc3e4, 0x1536522b, 0x0, 0x0)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/table/writer.go:254 +0x471\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*tWriter).append(0xcb876ef320, 0xcb5f5e4dc5, 0x10, 0x1536523b, 0xcb5f5e4dd5, 0xefc3e4, 0x1536522b, 0x0, 0x0)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/table.go:487 +0x2eb\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*tOps).createFrom(0xc8217e0510, 0x7f8775647620, 0xc9881b3880, 0x0, 0x0, 0x0, 0x0)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/table.go:325 +0x1c5\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*session).flushMemdb(0xc82051a160, 0xcb1a5294a0, 0xc8cdaba000, 0xffffffffffffffff, 0x0, 0x0, 0x0)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/session_compaction.go:27 +0x19b\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).memCompaction.func1(0xca1018aac0, 0x0, 0x0)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:274 +0x8e\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*compactionTransactFunc).run(0xca1018aab0, 0xca1018aac0, 0x0, 0x0)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:220 +0x36\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionTransact(0xc8200cec60, 0xf18a00, 0xb, 0x7f87755eacd0, 0xca1018aab0)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:164 +0x27b\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionTransactFunc(0xc8200cec60, 0xf18a00, 0xb, 0xcb1b7981e0, 0xc8f8584560)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:231 +0xbe\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).memCompaction(0xc8200cec60)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:286 +0x53c\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc8200cec60)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:719 +0x180\r\ncreated by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9cf\r\ngoroutine 236 [select]:\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc8200cec60)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x1a3\r\ncreated by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:148 +0x9f1\r\ngoroutine 82 [select]:\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc82198a000)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x29e\r\ncreated by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x270\r\ngoroutine 83 [select]:\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc82199e000)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x54d\r\ncreated by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x787\r\ngoroutine 84 [select]:\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc82199e000)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14c\r\ncreated by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7a9\r\ngoroutine 85 [select]:\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc82199e000)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x7e0\r\ncreated by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9ad\r\ngoroutine 86 [select]:\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc82199e000)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x254\r\ncreated by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9cf\r\ngoroutine 87 [select]:\r\ngithub.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc82199e000)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x1a3\r\ncreated by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:148 +0x9f1\r\ngoroutine 88 [runnable]:\r\ngithub.com/prometheus/prometheus/storage/local/codable.(*FingerprintSet).UnmarshalBinary(0xc915f4fd60, 0xc9bf32a000, 0x3f529bc, 0x3f54000, 0x0, 0x0)\r\n/go/src/github.com/prometheus/prometheus/storage/local/codable/codable.go:248 +0x34b\r\ngithub.com/prometheus/prometheus/storage/local/index.(*LevelDB).Get(0xc82133c040, 0x7f876dbac478, 0xc925c691a0, 0x7f876dbac450, 0xc915f4fd60, 0x7f876dbac478, 0x0, 0x0)\r\n/go/src/github.com/prometheus/prometheus/storage/local/index/leveldb.go:98 +0x202\r\ngithub.com/prometheus/prometheus/storage/local/index.(*LabelPairFingerprintIndex).LookupSet(0xc8217ea040, 0xc8840cea98, 0x7, 0xc8840ceaa0, 0x4, 0x0, 0x0, 0x0, 0x0)\r\n/go/src/github.com/prometheus/prometheus/storage/local/index/index.go:238 +0x15a\r\ngithub.com/prometheus/prometheus/storage/local.(*persistence).processIndexingQueue(0xc820224b00)\r\n/go/src/github.com/prometheus/prometheus/storage/local/persistence.go:1291 +0x5b7\r\ngithub.com/prometheus/prometheus/storage/local.(*persistence).run(0xc820224b00)\r\n/go/src/github.com/prometheus/prometheus/storage/local/persistence.go:283 +0x21\r\ncreated by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start\r\n/go/src/github.com/prometheus/prometheus/storage/local/storage.go:330 +0x14c\r\ngoroutine 255 [runnable]:\r\ngithub.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).waitForNextFP(0xc820269900, 0xacbd6, 0x3ff0000000000000, 0xacb01)\r\n/go/src/github.com/prometheus/prometheus/storage/local/storage.go:929 +0x19b\r\ngithub.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).cycleThroughArchivedFingerprints.func1(0xc95bff6660, 0xc820269900)\r\n/go/src/github.com/prometheus/prometheus/storage/local/storage.go:1013 +0x41e\r\ncreated by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).cycleThroughArchivedFingerprints\r\n/go/src/github.com/prometheus/prometheus/storage/local/storage.go:1022 +0x5d\r\ngoroutine 323 [chan send, 628 minutes]:\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:205 +0x1f8\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789\r\ngoroutine 329920 [IO wait]:\r\nnet.runtime_pollWait(0x7f876d6fca38, 0x72, 0xcba4998000)\r\n/usr/local/go/src/runtime/netpoll.go:160 +0x60\r\nnet.(*pollDesc).Wait(0xc8d62b3870, 0x72, 0x0, 0x0)\r\n/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\r\nnet.(*pollDesc).WaitRead(0xc8d62b3870, 0x0, 0x0)\r\n/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36\r\nnet.(*netFD).Read(0xc8d62b3810, 0xcba4998000, 0x4, 0x180000, 0x0, 0x7f8775629050, 0xc82000e150)\r\n/usr/local/go/src/net/fd_unix.go:250 +0x23a\r\nnet.(*conn).Read(0xc941425100, 0xcba4998000, 0x4, 0x180000, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/net.go:172 +0xe4\r\nio.ReadAtLeast(0x7f87753f3820, 0xc941425100, 0xcba4998000, 0x4, 0x180000, 0x4, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/io/io.go:297 +0xe6\r\nio.ReadFull(0x7f87753f3820, 0xc941425100, 0xcba4998000, 0x4, 0x180000, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/io/io.go:315 +0x62\r\ngithub.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk.(*Conn).recvLoop(0xc911dc4620, 0x7f87753f3770, 0xc941425100, 0x0, 0x0)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk/conn.go:544 +0x20f\r\ngithub.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk.(*Conn).loop.func2(0xc8ad730a10, 0xc911dc4620, 0xc8d6df11a0, 0xc8ad730b20)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk/conn.go:279 +0x46\r\ncreated by github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk.(*Conn).loop\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk/conn.go:285 +0x667\r\ngoroutine 254 [runnable]:\r\ngithub.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).waitForNextFP(0xc820269900, 0x169d26, 0x3fe6d57555555556, 0x1)\r\n/go/src/github.com/prometheus/prometheus/storage/local/storage.go:929 +0x19b\r\ngithub.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).cycleThroughMemoryFingerprints.func1(0xc95bff6600, 0xc820269900)\r\n/go/src/github.com/prometheus/prometheus/storage/local/storage.go:969 +0x250\r\ncreated by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).cycleThroughMemoryFingerprints\r\n/go/src/github.com/prometheus/prometheus/storage/local/storage.go:979 +0x5d\r\ngoroutine 124 [select, 628 minutes]:\r\ngithub.com/prometheus/prometheus/retrieval.(*targetSet).runScraping(0xc923f8f090, 0x7f8775632620, 0xc9b49cb340)\r\n/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:238 +0x2bb\r\ngithub.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0xc8202ea9a0, 0xc923f8f090)\r\n/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:118 +0x43\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload\r\n/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:120 +0x20f\r\ngoroutine 359 [select]:\r\ngithub.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).handleEvictList(0xc820269900)\r\n/go/src/github.com/prometheus/prometheus/storage/local/storage.go:825 +0x53a\r\ncreated by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start\r\n/go/src/github.com/prometheus/prometheus/storage/local/storage.go:353 +0x4b8\r\ngoroutine 360 [select, 628 minutes]:\r\ngithub.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).handleQuarantine(0xc820269900)\r\n/go/src/github.com/prometheus/prometheus/storage/local/storage.go:1435 +0x514\r\ncreated by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start\r\n/go/src/github.com/prometheus/prometheus/storage/local/storage.go:354 +0x4dd\r\ngoroutine 361 [select, 628 minutes]:\r\ngithub.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).logThrottling(0xc820269900)\r\n/go/src/github.com/prometheus/prometheus/storage/local/storage.go:698 +0x8e2\r\ncreated by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start\r\n/go/src/github.com/prometheus/prometheus/storage/local/storage.go:355 +0x502\r\ngoroutine 362 [select]:\r\ngithub.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).loop(0xc820269900)\r\n/go/src/github.com/prometheus/prometheus/storage/local/storage.go:1042 +0x5d6\r\ncreated by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start\r\n/go/src/github.com/prometheus/prometheus/storage/local/storage.go:356 +0x527\r\ngoroutine 367 [select, 4 minutes]:\r\ngithub.com/prometheus/prometheus/notifier.(*Notifier).Run(0xc82000a540)\r\n/go/src/github.com/prometheus/prometheus/notifier/notifier.go:169 +0x59a\r\ncreated by main.Main\r\n/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:166 +0x167c\r\ngoroutine 433 [semacquire, 628 minutes]:\r\nsync.runtime_Semacquire(0xc8202eaa04)\r\n/usr/local/go/src/runtime/sema.go:47 +0x26\r\nsync.(*WaitGroup).Wait(0xc8202ea9f8)\r\n/usr/local/go/src/sync/waitgroup.go:127 +0xb4\r\ngithub.com/prometheus/prometheus/retrieval.(*TargetManager).Run(0xc8202ea9a0)\r\n/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:83 +0x209\r\ncreated by main.Main\r\n/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:172 +0x171f\r\ngoroutine 434 [IO wait]:\r\nnet.runtime_pollWait(0x7f876d6fd3f8, 0x72, 0x0)\r\n/usr/local/go/src/runtime/netpoll.go:160 +0x60\r\nnet.(*pollDesc).Wait(0xc941c287d0, 0x72, 0x0, 0x0)\r\n/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\r\nnet.(*pollDesc).WaitRead(0xc941c287d0, 0x0, 0x0)\r\n/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36\r\nnet.(*netFD).accept(0xc941c28770, 0x0, 0x7f876d6fd4f0, 0xca160ff2e0)\r\n/usr/local/go/src/net/fd_unix.go:426 +0x27c\r\nnet.(*TCPListener).AcceptTCP(0xc97cce2c78, 0x45a730, 0x0, 0x0)\r\n/usr/local/go/src/net/tcpsock_posix.go:254 +0x4d\r\nnet/http.tcpKeepAliveListener.Accept(0xc97cce2c78, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/http/server.go:2427 +0x41\r\nnet/http.(*Server).Serve(0xc99dacf000, 0x7f876d6fd4b8, 0xc97cce2c78, 0x0, 0x0)\r\n/usr/local/go/src/net/http/server.go:2117 +0x129\r\nnet/http.(*Server).ListenAndServe(0xc99dacf000, 0x0, 0x0)\r\n/usr/local/go/src/net/http/server.go:2098 +0x136\r\nnet/http.ListenAndServe(0xef4e40, 0x5, 0x7f876d6fc420, 0xc820279a80, 0x0, 0x0)\r\n/usr/local/go/src/net/http/server.go:2195 +0x98\r\ngithub.com/prometheus/prometheus/web.(*Handler).Run(0xc82000a600)\r\n/go/src/github.com/prometheus/prometheus/web/web.go:242 +0x144\r\ncreated by main.Main\r\n/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:179 +0x179a\r\ngoroutine 329854 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc8f1954b40, 0x4a817c800, 0x4a817c800, 0x0)\r\n/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:433 +0x6ea\r\ncreated by github.com/prometheus/prometheus/retrieval.(*scrapePool).reload.func1\r\n/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:196 +0x7c\r\ngoroutine 450 [select, 43 minutes]:\r\ngithub.com/prometheus/prometheus/retrieval.(*targetSet).runScraping(0xc8f4a4a140, 0x7f8775632620, 0xc99ee1e280)\r\n/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:238 +0x2bb\r\ngithub.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0xc8202ea9a0, 0xc8f4a4a140)\r\n/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:118 +0x43\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload\r\n/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:120 +0x20f\r\ngoroutine 455 [chan send, 628 minutes]:\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:205 +0x1f8\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789\r\ngoroutine 456 [chan send, 628 minutes]:\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:205 +0x1f8\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789\r\ngoroutine 457 [chan send, 628 minutes]:\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:205 +0x1f8\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789\r\ngoroutine 420 [chan send, 628 minutes]:\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:205 +0x1f8\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789\r\ngoroutine 421 [chan send, 628 minutes]:\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:205 +0x1f8\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789\r\ngoroutine 329629 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc8f1954c80, 0x4a817c800, 0x4a817c800, 0x0)\r\n/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:433 +0x6ea\r\ncreated by github.com/prometheus/prometheus/retrieval.(*scrapePool).reload.func1\r\n/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:196 +0x7c\r\ngoroutine 329630 [runnable]:\r\ngithub.com/prometheus/prometheus/vendor/github.com/prometheus/common/model.LabelSet.Equal(0xc9690db9b0, 0xc944fddb90, 0xe80b01)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/model/labelset.go:49 +0x1a4\r\ngithub.com/prometheus/prometheus/vendor/github.com/prometheus/common/model.Metric.Equal(0xc9690db9b0, 0xc944fddb90, 0xc9a3fd4380)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/model/metric.go:34 +0x2b\r\ngithub.com/prometheus/prometheus/storage/local.(*fpMapper).mapFP(0xc908b41180, 0x614909a940e372fd, 0xc9690db9b0, 0x0)\r\n/go/src/github.com/prometheus/prometheus/storage/local/mapper.go:100 +0x93\r\ngithub.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Append(0xc820269900, 0xc9c7b864e0, 0x0, 0x0)\r\n/go/src/github.com/prometheus/prometheus/storage/local/storage.go:622 +0x1b3\r\ngithub.com/prometheus/prometheus/storage.Fanout.Append(0xc8202b32d0, 0x1, 0x1, 0xc9c7b864e0, 0x0, 0x0)\r\n/go/src/github.com/prometheus/prometheus/storage/storage.go:60 +0xc8\r\ngithub.com/prometheus/prometheus/storage.(*Fanout).Append(0xc8202799c0, 0xc9c7b864e0, 0x0, 0x0)\r\n<autogenerated>:3 +0xb7\r\ngithub.com/prometheus/prometheus/retrieval.honorLabelsAppender.Append(0x7f8775632690, 0xc8202799c0, 0xc910b45290, 0xc9c7b864e0, 0x0, 0x0)\r\n/go/src/github.com/prometheus/prometheus/retrieval/target.go:264 +0x1d0\r\ngithub.com/prometheus/prometheus/retrieval.(*honorLabelsAppender).Append(0xc9b2e03fe0, 0xc9c7b864e0, 0x0, 0x0)\r\n<autogenerated>:30 +0xb7\r\ngithub.com/prometheus/prometheus/retrieval.(*scrapeLoop).append(0xc8f1954cd0, 0xc99a020000, 0x15775, 0x17800)\r\n/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:453 +0xc7\r\ngithub.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc8f1954cd0, 0x4a817c800, 0x4a817c800, 0x0)\r\n/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:422 +0x539\r\ncreated by github.com/prometheus/prometheus/retrieval.(*scrapePool).reload.func1\r\n/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:196 +0x7c\r\ngoroutine 330040 [select]:\r\ngithub.com/prometheus/prometheus/vendor/golang.org/x/net/context/ctxhttp.Do(0x7f876d67a320, 0xca83e33050, 0xc910b451a0, 0xc97b21d260, 0xcc, 0x0, 0x0)\r\n/go/src/github.com/prometheus/prometheus/vendor/golang.org/x/net/context/ctxhttp/ctxhttp.go:56 +0x4bc\r\ngithub.com/prometheus/prometheus/retrieval.(*targetScraper).scrape(0xc8ae7d6000, 0x7f876d67a320, 0xca83e33050, 0xecf0c1344, 0x7f8732153743, 0x1500ba0, 0x0, 0x0, 0x0, 0x0, ...)\r\n/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:320 +0x185\r\ngithub.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc8f1954d70, 0x4a817c800, 0x4a817c800, 0x0)\r\n/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:420 +0x4c6\r\ncreated by github.com/prometheus/prometheus/retrieval.(*scrapePool).reload.func1\r\n/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:196 +0x7c\r\ngoroutine 332021 [IO wait]:\r\nnet.runtime_pollWait(0x7f876d6fc978, 0x77, 0x5c2e34)\r\n/usr/local/go/src/runtime/netpoll.go:160 +0x60\r\nnet.(*pollDesc).Wait(0xc8ec9b1e90, 0x77, 0x0, 0x0)\r\n/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\r\nnet.(*pollDesc).WaitWrite(0xc8ec9b1e90, 0x0, 0x0)\r\n/usr/local/go/src/net/fd_poll_runtime.go:82 +0x36\r\nnet.(*netFD).connect(0xc8ec9b1e30, 0x0, 0x0, 0x7f87753f37d0, 0xca1fb36a60, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n/usr/local/go/src/net/fd_unix.go:127 +0x28e\r\nnet.(*netFD).dial(0xc8ec9b1e30, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xc92e764d20, 0x0, 0xc900000000, 0x0, 0x0, 0x0, ...)\r\n/usr/local/go/src/net/sock_posix.go:137 +0x364\r\nnet.socket(0xefebb8, 0x3, 0x2, 0x1, 0x0, 0xc92e764d00, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xc92e764d20, ...)\r\n/usr/local/go/src/net/sock_posix.go:89 +0x429\r\nnet.internetSocket(0xefebb8, 0x3, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xc92e764d20, 0x0, 0x0, 0x0, 0x1, ...)\r\n/usr/local/go/src/net/ipsock_posix.go:161 +0x153\r\nnet.dialTCP(0xefebb8, 0x3, 0x0, 0xc92e764d20, 0x0, 0xc900000000, 0x0, 0x0, 0x0, 0x0, ...)\r\n/usr/local/go/src/net/tcpsock_posix.go:171 +0x12b\r\nnet.dialSingle(0xc95884e120, 0x7f876d6fc448, 0xc92e764d20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/dial.go:371 +0x40c\r\nnet.dialSerial.func1(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/dial.go:343 +0x75\r\nnet.dial(0xefebb8, 0x3, 0x7f876d6fc448, 0xc92e764d20, 0xca1fb36a40, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n/usr/local/go/src/net/fd_unix.go:40 +0x60\r\nnet.dialSerial(0xc95884e120, 0xc935b9a670, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/dial.go:345 +0x7d0\r\nnet.(*Dialer).Dial(0xc86bdc9868, 0xefebb8, 0x3, 0xc99a93f187, 0x12, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/dial.go:239 +0x512\r\nnet.Dial(0xefebb8, 0x3, 0xc99a93f187, 0x12, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/dial.go:193 +0x96\r\nnet/http.(*Transport).dial(0xc8f003a540, 0xefebb8, 0x3, 0xc99a93f187, 0x12, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/http/transport.go:665 +0x1d3\r\nnet/http.(*Transport).dialConn(0xc8f003a540, 0x0, 0xc99a93f180, 0x4, 0xc99a93f187, 0x12, 0xc910b451a0, 0x0, 0x0)\r\n/usr/local/go/src/net/http/transport.go:762 +0x1e3d\r\nnet/http.(*Transport).getConn.func4(0xc8f003a540, 0x0, 0xc99a93f180, 0x4, 0xc99a93f187, 0x12, 0xc9387b2ba0)\r\n/usr/local/go/src/net/http/transport.go:706 +0x66\r\ncreated by net/http.(*Transport).getConn\r\n/usr/local/go/src/net/http/transport.go:708 +0x262\r\ngoroutine 329913 [select, 3 minutes]:\r\ngithub.com/prometheus/prometheus/retrieval.(*targetSet).runProviders.func1(0xc963da72f0, 0xca189398c0, 0xc8f4a4a140, 0xc963da7310, 0xc963da72d0, 0x7, 0x7f87755413d8, 0xc9793b6720)\r\n/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:310 +0x465\r\ncreated by github.com/prometheus/prometheus/retrieval.(*targetSet).runProviders\r\n/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:326 +0x2bc\r\ngoroutine 331235 [IO wait, 1 minutes]:\r\nnet.runtime_pollWait(0x7f876a68bc50, 0x77, 0x5c2e34)\r\n/usr/local/go/src/runtime/netpoll.go:160 +0x60\r\nnet.(*pollDesc).Wait(0xc9490b0760, 0x77, 0x0, 0x0)\r\n/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\r\nnet.(*pollDesc).WaitWrite(0xc9490b0760, 0x0, 0x0)\r\n/usr/local/go/src/net/fd_poll_runtime.go:82 +0x36\r\nnet.(*netFD).connect(0xc9490b0700, 0x0, 0x0, 0x7f87753f37d0, 0xc967b74a60, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n/usr/local/go/src/net/fd_unix.go:127 +0x28e\r\nnet.(*netFD).dial(0xc9490b0700, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xca49a09860, 0x0, 0xc900000000, 0x0, 0x0, 0x0, ...)\r\n/usr/local/go/src/net/sock_posix.go:137 +0x364\r\nnet.socket(0xefebb8, 0x3, 0x2, 0x1, 0x0, 0xca49a09800, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xca49a09860, ...)\r\n/usr/local/go/src/net/sock_posix.go:89 +0x429\r\nnet.internetSocket(0xefebb8, 0x3, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xca49a09860, 0x0, 0x0, 0x0, 0x1, ...)\r\n/usr/local/go/src/net/ipsock_posix.go:161 +0x153\r\nnet.dialTCP(0xefebb8, 0x3, 0x0, 0xca49a09860, 0x0, 0xc900000000, 0x0, 0x0, 0x0, 0x0, ...)\r\n/usr/local/go/src/net/tcpsock_posix.go:171 +0x12b\r\nnet.dialSingle(0xc9a2ede090, 0x7f876d6fc448, 0xca49a09860, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/dial.go:371 +0x40c\r\nnet.dialSerial.func1(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/dial.go:343 +0x75\r\nnet.dial(0xefebb8, 0x3, 0x7f876d6fc448, 0xca49a09860, 0xc967b749e0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n/usr/local/go/src/net/fd_unix.go:40 +0x60\r\nnet.dialSerial(0xc9a2ede090, 0xc93040cb00, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/dial.go:345 +0x7d0\r\nnet.(*Dialer).Dial(0xc96dced868, 0xefebb8, 0x3, 0xc8d8152be7, 0x12, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/dial.go:239 +0x512\r\nnet.Dial(0xefebb8, 0x3, 0xc8d8152be7, 0x12, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/dial.go:193 +0x96\r\nnet/http.(*Transport).dial(0xc8f003a540, 0xefebb8, 0x3, 0xc8d8152be7, 0x12, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/http/transport.go:665 +0x1d3\r\nnet/http.(*Transport).dialConn(0xc8f003a540, 0x0, 0xc8d8152be0, 0x4, 0xc8d8152be7, 0x12, 0xc910b451a0, 0x0, 0x0)\r\n/usr/local/go/src/net/http/transport.go:762 +0x1e3d\r\nnet/http.(*Transport).getConn.func4(0xc8f003a540, 0x0, 0xc8d8152be0, 0x4, 0xc8d8152be7, 0x12, 0xc956e44b40)\r\n/usr/local/go/src/net/http/transport.go:706 +0x66\r\ncreated by net/http.(*Transport).getConn\r\n/usr/local/go/src/net/http/transport.go:708 +0x262\r\ngoroutine 331718 [IO wait, 1 minutes]:\r\nnet.runtime_pollWait(0x7f876d6fc4f8, 0x77, 0x5c2e34)\r\n/usr/local/go/src/runtime/netpoll.go:160 +0x60\r\nnet.(*pollDesc).Wait(0xc91813bfe0, 0x77, 0x0, 0x0)\r\n/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\r\nnet.(*pollDesc).WaitWrite(0xc91813bfe0, 0x0, 0x0)\r\n/usr/local/go/src/net/fd_poll_runtime.go:82 +0x36\r\nnet.(*netFD).connect(0xc91813bf80, 0x0, 0x0, 0x7f87753f37d0, 0xc950de53e0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n/usr/local/go/src/net/fd_unix.go:127 +0x28e\r\nnet.(*netFD).dial(0xc91813bf80, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xc8cef5d470, 0x0, 0xc800000000, 0x0, 0x0, 0x0, ...)\r\n/usr/local/go/src/net/sock_posix.go:137 +0x364\r\nnet.socket(0xefebb8, 0x3, 0x2, 0x1, 0x0, 0xc8cef5d400, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xc8cef5d470, ...)\r\n/usr/local/go/src/net/sock_posix.go:89 +0x429\r\nnet.internetSocket(0xefebb8, 0x3, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xc8cef5d470, 0x0, 0x0, 0x0, 0x1, ...)\r\n/usr/local/go/src/net/ipsock_posix.go:161 +0x153\r\nnet.dialTCP(0xefebb8, 0x3, 0x0, 0xc8cef5d470, 0x0, 0xc800000000, 0x0, 0x0, 0x0, 0x0, ...)\r\n/usr/local/go/src/net/tcpsock_posix.go:171 +0x12b\r\nnet.dialSingle(0xc918135830, 0x7f876d6fc448, 0xc8cef5d470, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/dial.go:371 +0x40c\r\nnet.dialSerial.func1(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/dial.go:343 +0x75\r\nnet.dial(0xefebb8, 0x3, 0x7f876d6fc448, 0xc8cef5d470, 0xc950de53c0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n/usr/local/go/src/net/fd_unix.go:40 +0x60\r\nnet.dialSerial(0xc918135830, 0xc8c7a57540, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/dial.go:345 +0x7d0\r\nnet.(*Dialer).Dial(0xc91fd3f868, 0xefebb8, 0x3, 0xcb1b55b5e7, 0x12, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/dial.go:239 +0x512\r\nnet.Dial(0xefebb8, 0x3, 0xcb1b55b5e7, 0x12, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/dial.go:193 +0x96\r\nnet/http.(*Transport).dial(0xc8f003a540, 0xefebb8, 0x3, 0xcb1b55b5e7, 0x12, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/http/transport.go:665 +0x1d3\r\nnet/http.(*Transport).dialConn(0xc8f003a540, 0x0, 0xcb1b55b5e0, 0x4, 0xcb1b55b5e7, 0x12, 0xc910b451a0, 0x0, 0x0)\r\n/usr/local/go/src/net/http/transport.go:762 +0x1e3d\r\nnet/http.(*Transport).getConn.func4(0xc8f003a540, 0x0, 0xcb1b55b5e0, 0x4, 0xcb1b55b5e7, 0x12, 0xc9602e5e60)\r\n/usr/local/go/src/net/http/transport.go:706 +0x66\r\ncreated by net/http.(*Transport).getConn\r\n/usr/local/go/src/net/http/transport.go:708 +0x262\r\ngoroutine 332215 [IO wait]:\r\nnet.runtime_pollWait(0x7f876d6fc678, 0x77, 0x5c2e34)\r\n/usr/local/go/src/runtime/netpoll.go:160 +0x60\r\nnet.(*pollDesc).Wait(0xc92f246370, 0x77, 0x0, 0x0)\r\n/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\r\nnet.(*pollDesc).WaitWrite(0xc92f246370, 0x0, 0x0)\r\n/usr/local/go/src/net/fd_poll_runtime.go:82 +0x36\r\nnet.(*netFD).connect(0xc92f246310, 0x0, 0x0, 0x7f87753f37d0, 0xcacd9dc300, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n/usr/local/go/src/net/fd_unix.go:127 +0x28e\r\nnet.(*netFD).dial(0xc92f246310, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xca83e33170, 0x0, 0xc800000000, 0x0, 0x0, 0x0, ...)\r\n/usr/local/go/src/net/sock_posix.go:137 +0x364\r\nnet.socket(0xefebb8, 0x3, 0x2, 0x1, 0x0, 0xca83e33100, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xca83e33170, ...)\r\n/usr/local/go/src/net/sock_posix.go:89 +0x429\r\nnet.internetSocket(0xefebb8, 0x3, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xca83e33170, 0x0, 0x0, 0x0, 0x1, ...)\r\n/usr/local/go/src/net/ipsock_posix.go:161 +0x153\r\nnet.dialTCP(0xefebb8, 0x3, 0x0, 0xca83e33170, 0x0, 0xc800000000, 0x0, 0x0, 0x0, 0x0, ...)\r\n/usr/local/go/src/net/tcpsock_posix.go:171 +0x12b\r\nnet.dialSingle(0xc944bf2fc0, 0x7f876d6fc448, 0xca83e33170, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/dial.go:371 +0x40c\r\nnet.dialSerial.func1(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/dial.go:343 +0x75\r\nnet.dial(0xefebb8, 0x3, 0x7f876d6fc448, 0xca83e33170, 0xcacd9dc2c0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n/usr/local/go/src/net/fd_unix.go:40 +0x60\r\nnet.dialSerial(0xc944bf2fc0, 0xc8f34e0720, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/dial.go:345 +0x7d0\r\nnet.(*Dialer).Dial(0xc83929b868, 0xefebb8, 0x3, 0xc9cc6a2af7, 0x12, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/dial.go:239 +0x512\r\nnet.Dial(0xefebb8, 0x3, 0xc9cc6a2af7, 0x12, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/dial.go:193 +0x96\r\nnet/http.(*Transport).dial(0xc8f003a540, 0xefebb8, 0x3, 0xc9cc6a2af7, 0x12, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/http/transport.go:665 +0x1d3\r\nnet/http.(*Transport).dialConn(0xc8f003a540, 0x0, 0xc9cc6a2af0, 0x4, 0xc9cc6a2af7, 0x12, 0xc910b451a0, 0x0, 0x0)\r\n/usr/local/go/src/net/http/transport.go:762 +0x1e3d\r\nnet/http.(*Transport).getConn.func4(0xc8f003a540, 0x0, 0xc9cc6a2af0, 0x4, 0xc9cc6a2af7, 0x12, 0xca5810e1e0)\r\n/usr/local/go/src/net/http/transport.go:706 +0x66\r\ncreated by net/http.(*Transport).getConn\r\n/usr/local/go/src/net/http/transport.go:708 +0x262\r\ngoroutine 330038 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc8f1954dc0, 0x4a817c800, 0x4a817c800, 0x0)\r\n/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:433 +0x6ea\r\ncreated by github.com/prometheus/prometheus/retrieval.(*scrapePool).reload.func1\r\n/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:196 +0x7c\r\ngoroutine 332258 [chan receive]:\r\nnet/http.(*Transport).getConn.func2.1(0xc9387b2ba0, 0xc8f003a540, 0x1101a98)\r\n/usr/local/go/src/net/http/transport.go:695 +0x48\r\ncreated by net/http.(*Transport).getConn.func2\r\n/usr/local/go/src/net/http/transport.go:699 +0x70\r\ngoroutine 296915 [chan send]:\r\ngithub.com/prometheus/prometheus/storage/local.(*seriesMap).fpIter.func1(0xc820966f60, 0xc86e726ba0)\r\n/go/src/github.com/prometheus/prometheus/storage/local/series.go:129 +0xe1\r\ncreated by github.com/prometheus/prometheus/storage/local.(*seriesMap).fpIter\r\n/go/src/github.com/prometheus/prometheus/storage/local/series.go:134 +0x5d\r\ngoroutine 298583 [chan send, 3 minutes]:\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:207 +0x318\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789\r\ngoroutine 332040 [IO wait]:\r\nnet.runtime_pollWait(0x7f876d6fd0f8, 0x72, 0xca95c9c000)\r\n/usr/local/go/src/runtime/netpoll.go:160 +0x60\r\nnet.(*pollDesc).Wait(0xc959e18290, 0x72, 0x0, 0x0)\r\n/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\r\nnet.(*pollDesc).WaitRead(0xc959e18290, 0x0, 0x0)\r\n/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36\r\nnet.(*netFD).Read(0xc959e18230, 0xca95c9c000, 0x1000, 0x1000, 0x0, 0x7f8775629050, 0xc82000e150)\r\n/usr/local/go/src/net/fd_unix.go:250 +0x23a\r\nnet.(*conn).Read(0xc886b48668, 0xca95c9c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/net.go:172 +0xe4\r\nnet/http.(*connReader).Read(0xca3e430500, 0xca95c9c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/http/server.go:526 +0x196\r\nbufio.(*Reader).fill(0xc9307bdbc0)\r\n/usr/local/go/src/bufio/bufio.go:97 +0x1e9\r\nbufio.(*Reader).ReadSlice(0xc9307bdbc0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/bufio/bufio.go:328 +0x21a\r\nbufio.(*Reader).ReadLine(0xc9307bdbc0, 0x0, 0x0, 0x0, 0xeb8c00, 0x0, 0x0)\r\n/usr/local/go/src/bufio/bufio.go:357 +0x53\r\nnet/textproto.(*Reader).readLineSlice(0xc8bf1fa390, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/textproto/reader.go:55 +0x81\r\nnet/textproto.(*Reader).ReadLine(0xc8bf1fa390, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/textproto/reader.go:36 +0x40\r\nnet/http.readRequest(0xc9307bdbc0, 0x700, 0xcb9da581c0, 0x0, 0x0)\r\n/usr/local/go/src/net/http/request.go:721 +0xb6\r\nnet/http.(*conn).readRequest(0xc9182a0380, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/http/server.go:705 +0x359\r\nnet/http.(*conn).serve(0xc9182a0380)\r\n/usr/local/go/src/net/http/server.go:1425 +0x947\r\ncreated by net/http.(*Server).Serve\r\n/usr/local/go/src/net/http/server.go:2137 +0x44e\r\ngoroutine 331566 [IO wait, 1 minutes]:\r\nnet.runtime_pollWait(0x7f876d6fc8b8, 0x77, 0x5c2e34)\r\n/usr/local/go/src/runtime/netpoll.go:160 +0x60\r\nnet.(*pollDesc).Wait(0xc9cb03b560, 0x77, 0x0, 0x0)\r\n/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\r\nnet.(*pollDesc).WaitWrite(0xc9cb03b560, 0x0, 0x0)\r\n/usr/local/go/src/net/fd_poll_runtime.go:82 +0x36\r\nnet.(*netFD).connect(0xc9cb03b500, 0x0, 0x0, 0x7f87753f37d0, 0xc94efd8d00, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n/usr/local/go/src/net/fd_unix.go:127 +0x28e\r\nnet.(*netFD).dial(0xc9cb03b500, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xc923dcf620, 0x0, 0xc900000000, 0x0, 0x0, 0x0, ...)\r\n/usr/local/go/src/net/sock_posix.go:137 +0x364\r\nnet.socket(0xefebb8, 0x3, 0x2, 0x1, 0x0, 0xc923dcf600, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xc923dcf620, ...)\r\n/usr/local/go/src/net/sock_posix.go:89 +0x429\r\nnet.internetSocket(0xefebb8, 0x3, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xc923dcf620, 0x0, 0x0, 0x0, 0x1, ...)\r\n/usr/local/go/src/net/ipsock_posix.go:161 +0x153\r\nnet.dialTCP(0xefebb8, 0x3, 0x0, 0xc923dcf620, 0x0, 0xc900000000, 0x0, 0x0, 0x0, 0x0, ...)\r\n/usr/local/go/src/net/tcpsock_posix.go:171 +0x12b\r\nnet.dialSingle(0xc82130a2d0, 0x7f876d6fc448, 0xc923dcf620, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/dial.go:371 +0x40c\r\nnet.dialSerial.func1(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/dial.go:343 +0x75\r\nnet.dial(0xefebb8, 0x3, 0x7f876d6fc448, 0xc923dcf620, 0xc94efd8ce0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n/usr/local/go/src/net/fd_unix.go:40 +0x60\r\nnet.dialSerial(0xc82130a2d0, 0xc95c4b4920, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/dial.go:345 +0x7d0\r\nnet.(*Dialer).Dial(0xcb1c8f7868, 0xefebb8, 0x3, 0xc8e32dda97, 0x12, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/dial.go:239 +0x512\r\nnet.Dial(0xefebb8, 0x3, 0xc8e32dda97, 0x12, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/dial.go:193 +0x96\r\nnet/http.(*Transport).dial(0xc8f003a540, 0xefebb8, 0x3, 0xc8e32dda97, 0x12, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/http/transport.go:665 +0x1d3\r\nnet/http.(*Transport).dialConn(0xc8f003a540, 0x0, 0xc8e32dda90, 0x4, 0xc8e32dda97, 0x12, 0xc910b451a0, 0x0, 0x0)\r\n/usr/local/go/src/net/http/transport.go:762 +0x1e3d\r\nnet/http.(*Transport).getConn.func4(0xc8f003a540, 0x0, 0xc8e32dda90, 0x4, 0xc8e32dda97, 0x12, 0xc822bc2d80)\r\n/usr/local/go/src/net/http/transport.go:706 +0x66\r\ncreated by net/http.(*Transport).getConn\r\n/usr/local/go/src/net/http/transport.go:708 +0x262\r\ngoroutine 330050 [select, 3 minutes]:\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:203 +0x360\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789\r\ngoroutine 298407 [chan send, 3 minutes]:\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:205 +0x1f8\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789\r\ngoroutine 329919 [select]:\r\ngithub.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk.(*Conn).sendLoop(0xc911dc4620, 0x7f87753f3770, 0xc941425100, 0xc8d6df11a0, 0x0, 0x0)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk/conn.go:480 +0xd8c\r\ngithub.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk.(*Conn).loop.func1(0xc911dc4620, 0xc8d6df11a0, 0xc8ad730b20)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk/conn.go:272 +0x48\r\ncreated by github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk.(*Conn).loop\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk/conn.go:275 +0x60d\r\ngoroutine 329911 [select, 3 minutes]:\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:203 +0x360\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789\r\ngoroutine 297813 [chan send, 3 minutes]:\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:205 +0x1f8\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789\r\ngoroutine 331587 [chan receive, 1 minutes]:\r\nnet/http.(*Transport).getConn.func2.1(0xcb860ebce0, 0xc8f003a540, 0x1101a98)\r\n/usr/local/go/src/net/http/transport.go:695 +0x48\r\ncreated by net/http.(*Transport).getConn.func2\r\n/usr/local/go/src/net/http/transport.go:699 +0x70\r\ngoroutine 330065 [select, 3 minutes]:\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:203 +0x360\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789\r\ngoroutine 329632 [chan receive, 3 minutes]:\r\ngithub.com/prometheus/prometheus/retrieval/discovery.(*NerveDiscovery).processUpdates(0xc9793b6720)\r\ncreated by github.com/prometheus/prometheus/retrieval/discovery.NewNerveDiscovery\r\ngoroutine 331395 [IO wait, 1 minutes]:\r\nnet.runtime_pollWait(0x7f876d6fcbb8, 0x77, 0x5c2e34)\r\n/usr/local/go/src/runtime/netpoll.go:160 +0x60\r\nnet.(*pollDesc).Wait(0xc94643fd40, 0x77, 0x0, 0x0)\r\n/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\r\nnet.(*pollDesc).WaitWrite(0xc94643fd40, 0x0, 0x0)\r\n/usr/local/go/src/net/fd_poll_runtime.go:82 +0x36\r\nnet.(*netFD).connect(0xc94643fce0, 0x0, 0x0, 0x7f87753f37d0, 0xc9182e4fc0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n/usr/local/go/src/net/fd_unix.go:127 +0x28e\r\nnet.(*netFD).dial(0xc94643fce0, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xc9a07054d0, 0x0, 0xc800000000, 0x0, 0x0, 0x0, ...)\r\n/usr/local/go/src/net/sock_posix.go:137 +0x364\r\nnet.socket(0xefebb8, 0x3, 0x2, 0x1, 0x0, 0xc9a0705400, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xc9a07054d0, ...)\r\n/usr/local/go/src/net/sock_posix.go:89 +0x429\r\nnet.internetSocket(0xefebb8, 0x3, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xc9a07054d0, 0x0, 0x0, 0x0, 0x1, ...)\r\n/usr/local/go/src/net/ipsock_posix.go:161 +0x153\r\nnet.dialTCP(0xefebb8, 0x3, 0x0, 0xc9a07054d0, 0x0, 0xc800000000, 0x0, 0x0, 0x0, 0x0, ...)\r\n/usr/local/go/src/net/tcpsock_posix.go:171 +0x12b\r\nnet.dialSingle(0xc9f3936bd0, 0x7f876d6fc448, 0xc9a07054d0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/dial.go:371 +0x40c\r\nnet.dialSerial.func1(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/dial.go:343 +0x75\r\nnet.dial(0xefebb8, 0x3, 0x7f876d6fc448, 0xc9a07054d0, 0xc9182e4fa0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n/usr/local/go/src/net/fd_unix.go:40 +0x60\r\nnet.dialSerial(0xc9f3936bd0, 0xc8e55da6e0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/dial.go:345 +0x7d0\r\nnet.(*Dialer).Dial(0xc96dce7868, 0xefebb8, 0x3, 0xc9cae86697, 0x12, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/dial.go:239 +0x512\r\nnet.Dial(0xefebb8, 0x3, 0xc9cae86697, 0x12, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/dial.go:193 +0x96\r\nnet/http.(*Transport).dial(0xc8f003a540, 0xefebb8, 0x3, 0xc9cae86697, 0x12, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/http/transport.go:665 +0x1d3\r\nnet/http.(*Transport).dialConn(0xc8f003a540, 0x0, 0xc9cae86690, 0x4, 0xc9cae86697, 0x12, 0xc910b451a0, 0x0, 0x0)\r\n/usr/local/go/src/net/http/transport.go:762 +0x1e3d\r\nnet/http.(*Transport).getConn.func4(0xc8f003a540, 0x0, 0xc9cae86690, 0x4, 0xc9cae86697, 0x12, 0xcb860ebce0)\r\n/usr/local/go/src/net/http/transport.go:706 +0x66\r\ncreated by net/http.(*Transport).getConn\r\n/usr/local/go/src/net/http/transport.go:708 +0x262\r\ngoroutine 329914 [chan receive, 3 minutes]:\r\ngithub.com/prometheus/prometheus/retrieval/discovery.(*NerveDiscovery).Run(0xc9793b6720, 0x7f8775632620, 0xca2a8c3740, 0xca189398c0)\r\ncreated by github.com/prometheus/prometheus/retrieval.(*targetSet).runProviders\r\n/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:328 +0x308\r\ngoroutine 305585 [chan send, 3 minutes]:\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:207 +0x318\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789\r\ngoroutine 305605 [chan send, 3 minutes]:\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:207 +0x318\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789\r\ngoroutine 329631 [semacquire, 3 minutes]:\r\nsync.runtime_Semacquire(0xc8ad730b2c)\r\n/usr/local/go/src/runtime/sema.go:47 +0x26\r\nsync.(*WaitGroup).Wait(0xc8ad730b20)\r\n/usr/local/go/src/sync/waitgroup.go:127 +0xb4\r\ngithub.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk.(*Conn).loop(0xc911dc4620)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk/conn.go:287 +0x675\r\ngithub.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk.ConnectWithDialer.func1(0xc911dc4620)\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk/conn.go:185 +0x21\r\ncreated by github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk.ConnectWithDialer\r\n/go/src/github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk/conn.go:189 +0x5b1\r\ngoroutine 331663 [chan receive]:\r\nnet/http.(*Transport).getConn.func2.1(0xc9602e5e60, 0xc8f003a540, 0x1101a98)\r\n/usr/local/go/src/net/http/transport.go:695 +0x48\r\ncreated by net/http.(*Transport).getConn.func2\r\n/usr/local/go/src/net/http/transport.go:699 +0x70\r\ngoroutine 331664 [IO wait]:\r\nnet.runtime_pollWait(0x7f876d6fc738, 0x72, 0xc8faf54000)\r\n/usr/local/go/src/runtime/netpoll.go:160 +0x60\r\nnet.(*pollDesc).Wait(0xc8ec7a8c30, 0x72, 0x0, 0x0)\r\n/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\r\nnet.(*pollDesc).WaitRead(0xc8ec7a8c30, 0x0, 0x0)\r\n/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36\r\nnet.(*netFD).Read(0xc8ec7a8bd0, 0xc8faf54000, 0x1000, 0x1000, 0x0, 0x7f8775629050, 0xc82000e150)\r\n/usr/local/go/src/net/fd_unix.go:250 +0x23a\r\nnet.(*conn).Read(0xc929cabff0, 0xc8faf54000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/net.go:172 +0xe4\r\nnet/http.(*connReader).Read(0xca3a9ee500, 0xc8faf54000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/http/server.go:526 +0x196\r\nbufio.(*Reader).fill(0xc936a17d40)\r\n/usr/local/go/src/bufio/bufio.go:97 +0x1e9\r\nbufio.(*Reader).ReadSlice(0xc936a17d40, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/bufio/bufio.go:328 +0x21a\r\nbufio.(*Reader).ReadLine(0xc936a17d40, 0x0, 0x0, 0x0, 0xeb8c00, 0x0, 0x0)\r\n/usr/local/go/src/bufio/bufio.go:357 +0x53\r\nnet/textproto.(*Reader).readLineSlice(0xc8bbdd61e0, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/textproto/reader.go:55 +0x81\r\nnet/textproto.(*Reader).ReadLine(0xc8bbdd61e0, 0x0, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/textproto/reader.go:36 +0x40\r\nnet/http.readRequest(0xc936a17d40, 0x900, 0xc9a519e7e0, 0x0, 0x0)\r\n/usr/local/go/src/net/http/request.go:721 +0xb6\r\nnet/http.(*conn).readRequest(0xc976b0d900, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/http/server.go:705 +0x359\r\nnet/http.(*conn).serve(0xc976b0d900)\r\n/usr/local/go/src/net/http/server.go:1425 +0x947\r\ncreated by net/http.(*Server).Serve\r\n/usr/local/go/src/net/http/server.go:2137 +0x44e\r\ngoroutine 329910 [select, 3 minutes]:\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:203 +0x360\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789\r\ngoroutine 329907 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc8f1954d20, 0x4a817c800, 0x4a817c800, 0x0)\r\n/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:433 +0x6ea\r\ncreated by github.com/prometheus/prometheus/retrieval.(*scrapePool).reload.func1\r\n/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:196 +0x7c\r\ngoroutine 330066 [select, 3 minutes]:\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:203 +0x360\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789\r\ngoroutine 305620 [chan send, 3 minutes]:\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:205 +0x1f8\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789\r\ngoroutine 331254 [chan receive, 1 minutes]:\r\nnet/http.(*Transport).getConn.func2.1(0xc956e44b40, 0xc8f003a540, 0x1101a98)\r\n/usr/local/go/src/net/http/transport.go:695 +0x48\r\ncreated by net/http.(*Transport).getConn.func2\r\n/usr/local/go/src/net/http/transport.go:699 +0x70\r\ngoroutine 332214 [select]:\r\nnet/http.(*Transport).getConn(0xc8f003a540, 0xc97b21d260, 0x0, 0xc9cc6a2af0, 0x4, 0xc9cc6a2af7, 0x12, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/http/transport.go:711 +0x4ef\r\nnet/http.(*Transport).RoundTrip(0xc8f003a540, 0xc97b21d260, 0xc8f003a540, 0x0, 0x0)\r\n/usr/local/go/src/net/http/transport.go:311 +0x7e9\r\nnet/http.send(0xc97b21d260, 0x7f8775631648, 0xc8f003a540, 0x0, 0x0, 0x0, 0xc9cc6a2b40, 0x0, 0x0)\r\n/usr/local/go/src/net/http/client.go:260 +0x6b7\r\nnet/http.(*Client).send(0xc910b451a0, 0xc97b21d260, 0x0, 0x0, 0x0, 0xc86a9a7000, 0x0, 0x0)\r\n/usr/local/go/src/net/http/client.go:155 +0x185\r\nnet/http.(*Client).doFollowingRedirects(0xc910b451a0, 0xc97b21d260, 0x1101ab8, 0x0, 0x0, 0x0)\r\n/usr/local/go/src/net/http/client.go:475 +0x8a4\r\nnet/http.(*Client).Do(0xc910b451a0, 0xc97b21d260, 0x1102020, 0x0, 0x0)\r\n/usr/local/go/src/net/http/client.go:188 +0xff\r\ngithub.com/prometheus/prometheus/vendor/golang.org/x/net/context/ctxhttp.Do.func1(0xc910b451a0, 0xc97b21d260, 0x1101420, 0xca5810e180)\r\n/go/src/github.com/prometheus/prometheus/vendor/golang.org/x/net/context/ctxhttp/ctxhttp.go:49 +0x2f\r\ncreated by github.com/prometheus/prometheus/vendor/golang.org/x/net/context/ctxhttp.Do\r\n/go/src/github.com/prometheus/prometheus/vendor/golang.org/x/net/context/ctxhttp/ctxhttp.go:52 +0x10d\r\ngoroutine 330049 [select, 3 minutes]:\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:203 +0x360\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789\r\ngoroutine 331875 [chan receive, 1 minutes]:\r\nnet/http.(*Transport).getConn.func2.1(0xc822bc2d80, 0xc8f003a540, 0x1101a98)\r\n/usr/local/go/src/net/http/transport.go:695 +0x48\r\ncreated by net/http.(*Transport).getConn.func2\r\n/usr/local/go/src/net/http/transport.go:699 +0x70\r\ngoroutine 329912 [select, 3 minutes]:\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:94 +0xd85\r\n/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:72 +0x32e\r\n\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/comments/230277702","html_url":"https://github.com/prometheus/prometheus/issues/1786#issuecomment-230277702","issue_url":"https://api.github.com/repos/prometheus/prometheus/issues/1786","id":230277702,"user":{"login":"PrFalken","id":334048,"avatar_url":"https://avatars.githubusercontent.com/u/334048?v=3","gravatar_id":"","url":"https://api.github.com/users/PrFalken","html_url":"https://github.com/PrFalken","followers_url":"https://api.github.com/users/PrFalken/followers","following_url":"https://api.github.com/users/PrFalken/following{/other_user}","gists_url":"https://api.github.com/users/PrFalken/gists{/gist_id}","starred_url":"https://api.github.com/users/PrFalken/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PrFalken/subscriptions","organizations_url":"https://api.github.com/users/PrFalken/orgs","repos_url":"https://api.github.com/users/PrFalken/repos","events_url":"https://api.github.com/users/PrFalken/events{/privacy}","received_events_url":"https://api.github.com/users/PrFalken/received_events","type":"User","site_admin":false},"created_at":"2016-07-04T12:18:42Z","updated_at":"2016-07-04T12:18:42Z","body":"I'm digging in logs. Crash recoveries on 1.2M metrics makes them difficult to find :) \r\nKeep you posted."}}
{"actor":{"display_login":"PrFalken"}}