{"action":"created","issue":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/877","labels_url":"https://api.github.com/repos/prometheus/prometheus/issues/877/labels{/name}","comments_url":"https://api.github.com/repos/prometheus/prometheus/issues/877/comments","events_url":"https://api.github.com/repos/prometheus/prometheus/issues/877/events","html_url":"https://github.com/prometheus/prometheus/issues/877","id":93279349,"number":877,"title":"Isolate corrupted series files instead of panicking.","user":{"login":"TheTincho","id":5602605,"avatar_url":"https://avatars.githubusercontent.com/u/5602605?v=3","gravatar_id":"","url":"https://api.github.com/users/TheTincho","html_url":"https://github.com/TheTincho","followers_url":"https://api.github.com/users/TheTincho/followers","following_url":"https://api.github.com/users/TheTincho/following{/other_user}","gists_url":"https://api.github.com/users/TheTincho/gists{/gist_id}","starred_url":"https://api.github.com/users/TheTincho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/TheTincho/subscriptions","organizations_url":"https://api.github.com/users/TheTincho/orgs","repos_url":"https://api.github.com/users/TheTincho/repos","events_url":"https://api.github.com/users/TheTincho/events{/privacy}","received_events_url":"https://api.github.com/users/TheTincho/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/prometheus/prometheus/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":{"login":"beorn7","id":5609886,"avatar_url":"https://avatars.githubusercontent.com/u/5609886?v=3","gravatar_id":"","url":"https://api.github.com/users/beorn7","html_url":"https://github.com/beorn7","followers_url":"https://api.github.com/users/beorn7/followers","following_url":"https://api.github.com/users/beorn7/following{/other_user}","gists_url":"https://api.github.com/users/beorn7/gists{/gist_id}","starred_url":"https://api.github.com/users/beorn7/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/beorn7/subscriptions","organizations_url":"https://api.github.com/users/beorn7/orgs","repos_url":"https://api.github.com/users/beorn7/repos","events_url":"https://api.github.com/users/beorn7/events{/privacy}","received_events_url":"https://api.github.com/users/beorn7/received_events","type":"User","site_admin":false},"milestone":null,"comments":6,"created_at":"2015-07-06T13:31:30Z","updated_at":"2016-01-08T14:55:36Z","closed_at":null,"body":"While looking though data with the prometheus console, I get many errors in the log like this:\r\n\r\n```\r\n2015/07/06 08:37:15 http: panic serving 127.0.0.1:40562: unknown chunk encoding: 255\r\ngoroutine 661098 [running]:\r\nnet/http.func·011()\r\n\t/usr/local/go/src/net/http/server.go:1130 +0xbb\r\ngithub.com/prometheus/prometheus/storage/local.newChunkForEncoding(0x7fca38e1beff, 0x0, 0x0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/chunk.go:286 +0x265\r\ngithub.com/prometheus/prometheus/storage/local.(*persistence).loadChunks(0xc2082b7d00, 0xb88cfd1aad811e79, 0xc2419e8000, 0x48, 0x80, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/persistence.go:437 +0x51d\r\ngithub.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).loadChunks(0xc2080661c0, 0xb88cfd1aad811e79, 0xc2419e8000, 0x48, 0x80, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:1093 +0xa0\r\ngithub.com/prometheus/prometheus/storage/local.(*memorySeries).preloadChunks(0xc21f1ed1a0, 0xc225664280, 0x49, 0x49, 0xb88cfd1aad811e79, 0xc2080661c0, 0x0, 0x0, 0x0, 0x0, ...)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/series.go:323 +0x4c5\r\ngithub.com/prometheus/prometheus/storage/local.(*memorySeries).preloadChunksForRange(0xc21f1ed1a0, 0x14e3f2e8450, 0x14e63443010, 0xb88cfd1aad811e79, 0xc2080661c0, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/series.go:413 +0x5cc\r\ngithub.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).preloadChunksForRange(0xc2080661c0, 0xb88cfd1aad811e79, 0x14e3f2e8450, 0x14e63443010, 0x45d964b800, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:611 +0x36b\r\ngithub.com/prometheus/prometheus/storage/local.(*memorySeriesPreloader).PreloadRange(0xc2581a9da0, 0xb88cfd1aad811e79, 0x14e3f2e8450, 0x14e63443010, 0x45d964b800, 0x0, 0x0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/preload.go:34 +0x82\r\ngithub.com/prometheus/prometheus/promql.(*Analyzer).Prepare(0xc2218e5b00, 0x7fca1f5fc608, 0xc25a01f4a0, 0x0, 0x0, 0x0, 0x0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/promql/analyzer.go:136 +0x4ef\r\ngithub.com/prometheus/prometheus/promql.(*Engine).execEvalStmt(0xc20800bdc0, 0x7fca1f5fc608, 0xc25a01f4a0, 0xc2218e5a80, 0xc233f4b0e0, 0x0, 0x0, 0x0, 0x0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/promql/engine.go:411 +0x39d\r\ngithub.com/prometheus/prometheus/promql.(*Engine).exec(0xc20800bdc0, 0xc2218e5a80, 0x0, 0x0, 0x0, 0x0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/promql/engine.go:376 +0x5a7\r\ngithub.com/prometheus/prometheus/promql.(*query).Exec(0xc2218e5a80, 0xc229dc3570)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/promql/engine.go:212 +0x31\r\ngithub.com/prometheus/prometheus/web/api/legacy.(*API).QueryRange(0xc20801f4e0, 0x7fca1e6444e0, 0xc23fbb5160, 0xc25b999380)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/web/api/legacy/query.go:144 +0x92a\r\ngithub.com/prometheus/prometheus/web/api/legacy.*API.QueryRange·fm(0x7fca1e6444e0, 0xc23fbb5160, 0xc25b999380)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/web/api/legacy/api.go:39 +0x45\r\nnet/http.HandlerFunc.ServeHTTP(0xc208101250, 0x7fca1e6444e0, 0xc23fbb5160, 0xc25b999380)\r\n\t/usr/local/go/src/net/http/server.go:1265 +0x41\r\ngithub.com/prometheus/prometheus/util/httputil.CompressionHandler.ServeHTTP(0x7fca38e1c070, 0xc208101250, 0x7fca34c53988, 0xc2294965f0, 0xc25b999380)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/util/httputil/compression.go:90 +0x9a\r\ngithub.com/prometheus/prometheus/util/httputil.(*CompressionHandler).ServeHTTP(0xc208101260, 0x7fca34c53988, 0xc2294965f0, 0xc25b999380)\r\n\t<autogenerated>:5 +0xbd\r\nnet/http.Handler.ServeHTTP·fm(0x7fca34c53988, 0xc2294965f0, 0xc25b999380)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/web/web.go:153 +0x57\r\ngithub.com/prometheus/client_golang/prometheus.func·008(0x7fca34c53950, 0xc230d6a6e0, 0xc25b999380)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/client_golang/prometheus/http.go:144 +0x1b0\r\ngithub.com/prometheus/prometheus/util/route.func·001(0x7fca34c53950, 0xc230d6a6e0, 0xc25b999380, 0x0, 0x0, 0x0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/util/route/route.go:49 +0x427\r\ngithub.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc2080d6180, 0x7fca34c53950, 0xc230d6a6e0, 0xc25b999380)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/julienschmidt/httprouter/router.go:299 +0x18e\r\ngithub.com/prometheus/prometheus/util/route.(*Router).ServeHTTP(0xc20801f4a0, 0x7fca34c53950, 0xc230d6a6e0, 0xc25b999380)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/util/route/route.go:90 +0x49\r\nnet/http.serverHandler.ServeHTTP(0xc23a2cc7e0, 0x7fca34c53950, 0xc230d6a6e0, 0xc25b999380)\r\n\t/usr/local/go/src/net/http/server.go:1703 +0x19a\r\nnet/http.(*conn).serve(0xc230d6a0a0)\r\n\t/usr/local/go/src/net/http/server.go:1204 +0xb57\r\ncreated by net/http.(*Server).Serve\r\n\t/usr/local/go/src/net/http/server.go:1751 +0x35e\r\n```\r\n\r\nAnd some times this one:\r\n\r\n```\r\n2015/07/03 11:54:57 http: panic serving 127.0.0.1:23506: runtime error: slice bounds out of range\r\ngoroutine 2779141 [running]:\r\nnet/http.func·011()\r\n\t/usr/local/go/src/net/http/server.go:1130 +0xbb\r\ngithub.com/prometheus/prometheus/storage/local.(*deltaEncodedChunk).unmarshalFromBuf(0xc270845e20, 0xc298e1bf30, 0x4510, 0x4510)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/delta.go:238 +0x10f\r\ngithub.com/prometheus/prometheus/storage/local.(*persistence).loadChunks(0xc2082b4900, 0x14fbde92b6c46404, 0xc270883000, 0x65, 0x80, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/persistence.go:438 +0x5b4\r\ngithub.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).loadChunks(0xc2080621c0, 0x14fbde92b6c46404, 0xc270883000, 0x65, 0x80, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:1093 +0xa0\r\ngithub.com/prometheus/prometheus/storage/local.(*memorySeries).preloadChunks(0xc273a3a300, 0xc270882800, 0x7a, 0x7a, 0x14fbde92b6c46404, 0xc2080621c0, 0x0, 0x0, 0x0, 0x0, ...)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/series.go:323 +0x4c5\r\ngithub.com/prometheus/prometheus/storage/local.(*memorySeries).preloadChunksForRange(0xc273a3a300, 0x14e30799800, 0x14e548f43c0, 0x14fbde92b6c46404, 0xc2080621c0, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/series.go:413 +0x5cc\r\ngithub.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).preloadChunksForRange(0xc2080621c0, 0x14fbde92b6c46404, 0x14e30799800, 0x14e548f43c0, 0x45d964b800, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:611 +0x36b\r\ngithub.com/prometheus/prometheus/storage/local.(*memorySeriesPreloader).PreloadRange(0xc29ad35e00, 0x14fbde92b6c46404, 0x14e30799800, 0x14e548f43c0, 0x45d964b800, 0x0, 0x0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/preload.go:34 +0x82\r\ngithub.com/prometheus/prometheus/promql.(*Analyzer).Prepare(0xc299f6f7c0, 0x7fcce27fe100, 0xc2974be4e0, 0x0, 0x0, 0x0, 0x0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/promql/analyzer.go:136 +0x4ef\r\ngithub.com/prometheus/prometheus/promql.(*Engine).execEvalStmt(0xc208057d80, 0x7fcce27fe100, 0xc2974be4e0, 0xc299f6f740, 0xc2974d5950, 0x0, 0x0, 0x0, 0x0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/promql/engine.go:411 +0x39d\r\ngithub.com/prometheus/prometheus/promql.(*Engine).exec(0xc208057d80, 0xc299f6f740, 0x0, 0x0, 0x0, 0x0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/promql/engine.go:376 +0x5a7\r\ngithub.com/prometheus/prometheus/promql.(*query).Exec(0xc299f6f740, 0xc2974b8a80)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/promql/engine.go:212 +0x31\r\ngithub.com/prometheus/prometheus/web/api/legacy.(*API).QueryRange(0xc20801f4e0, 0x7fcce27fe028, 0xc29acbf480, 0xc29a458680)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/web/api/legacy/query.go:144 +0x92a\r\ngithub.com/prometheus/prometheus/web/api/legacy.*API.QueryRange·fm(0x7fcce27fe028, 0xc29acbf480, 0xc29a458680)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/web/api/legacy/api.go:39 +0x45\r\nnet/http.HandlerFunc.ServeHTTP(0xc2080d9270, 0x7fcce27fe028, 0xc29acbf480, 0xc29a458680)\r\n\t/usr/local/go/src/net/http/server.go:1265 +0x41\r\ngithub.com/prometheus/prometheus/util/httputil.CompressionHandler.ServeHTTP(0x7fcceb638070, 0xc2080d9270, 0x7fcce8b73488, 0xc297b7f400, 0xc29a458680)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/util/httputil/compression.go:90 +0x9a\r\ngithub.com/prometheus/prometheus/util/httputil.(*CompressionHandler).ServeHTTP(0xc2080d9280, 0x7fcce8b73488, 0xc297b7f400, 0xc29a458680)\r\n\t<autogenerated>:5 +0xbd\r\nnet/http.Handler.ServeHTTP·fm(0x7fcce8b73488, 0xc297b7f400, 0xc29a458680)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/web/web.go:153 +0x57\r\ngithub.com/prometheus/client_golang/prometheus.func·008(0x7fcce8b73450, 0xc29acd45a0, 0xc29a458680)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/client_golang/prometheus/http.go:144 +0x1b0\r\ngithub.com/prometheus/prometheus/util/route.func·001(0x7fcce8b73450, 0xc29acd45a0, 0xc29a458680, 0x0, 0x0, 0x0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/util/route/route.go:49 +0x427\r\ngithub.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc2080d4180, 0x7fcce8b73450, 0xc29acd45a0, 0xc29a458680)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/julienschmidt/httprouter/router.go:299 +0x18e\r\ngithub.com/prometheus/prometheus/util/route.(*Router).ServeHTTP(0xc20801f4a0, 0x7fcce8b73450, 0xc29acd45a0, 0xc29a458680)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/util/route/route.go:90 +0x49\r\nnet/http.serverHandler.ServeHTTP(0xc23387d800, 0x7fcce8b73450, 0xc29acd45a0, 0xc29a458680)\r\n\t/usr/local/go/src/net/http/server.go:1703 +0x19a\r\nnet/http.(*conn).serve(0xc29acd4500)\r\n\t/usr/local/go/src/net/http/server.go:1204 +0xb57\r\ncreated by net/http.(*Server).Serve\r\n\t/usr/local/go/src/net/http/server.go:1751 +0x35e\r\n```\r\n\r\nI suspect this might be some DB corruption, as the process has been killed many times (usually through SIGTERM though). I have also seen the whole server come down because of other errors that might be related. For example:\r\n\r\n```\r\npanic: Invalid number of bytes for time delta\r\n\r\ngoroutine 116 [running]:\r\ngithub.com/prometheus/prometheus/storage/local.(*doubleDeltaEncodedChunkIterator).timestampAtIndex(0xc291382190, 0x2, 0x400)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/doubledelta.go:514 +0x1c2\r\ngithub.com/prometheus/prometheus/storage/local.(*doubleDeltaEncodedChunkIterator).lastTimestamp(0xc291382190, 0x7fcce8b63000)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/doubledelta.go:520 +0x34\r\ngithub.com/prometheus/prometheus/storage/local.(*chunkDesc).lastTime(0xc293ebca00, 0x0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/chunk.go:142 +0xc3\r\ngithub.com/prometheus/prometheus/storage/local.func·019()\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/persistence.go:642 +0x7bd\r\ngithub.com/prometheus/prometheus/storage/local.(*persistence).checkpointSeriesMapAndHeads(0xc2082b4900, 0xc208453260, 0xc20801ef00, 0x0, 0x0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/persistence.go:657 +0x5bc\r\ngithub.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).loop(0xc2080621c0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:844 +0x24d\r\ncreated by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:270 +0x51f\r\n\r\ngoroutine 1 [select, 1376 minutes]:\r\nmain.Main(0x0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:149 +0x183b\r\nmain.main()\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:46 +0x1f\r\n\r\ngoroutine 16 [syscall, 1369 minutes]:\r\nos/signal.loop()\r\n\t/usr/local/go/src/os/signal/signal_unix.go:21 +0x1f\r\ncreated by os/signal.init·1\r\n\t/usr/local/go/src/os/signal/signal_unix.go:27 +0x35\r\n\r\ngoroutine 89 [select]:\r\ngithub.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc208062380)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:205 +0x21e\r\ncreated by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:236 +0x24c\r\n\r\ngoroutine 88 [chan receive, 1369 minutes]:\r\nmain.func·005()\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:105 +0xbd\r\ncreated by main.Main\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:108 +0xe1e\r\n\r\ngoroutine 91 [select, 1377 minutes]:\r\ngithub.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc2080b2420)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x55a\r\ncreated by github.com/syndtr/goleveldb/leveldb.openDB\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x7cc\r\n\r\ngoroutine 92 [select]:\r\ngithub.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc2080b2420)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14a\r\ncreated by github.com/syndtr/goleveldb/leveldb.openDB\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7e6\r\n\r\ngoroutine 93 [select, 1377 minutes]:\r\ngithub.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc2080b2420)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x858\r\ncreated by github.com/syndtr/goleveldb/leveldb.openDB\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9cf\r\n\r\ngoroutine 94 [select, 1377 minutes]:\r\ngithub.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc2080b2420)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x283\r\ncreated by github.com/syndtr/goleveldb/leveldb.openDB\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9e9\r\n\r\ngoroutine 95 [select, 1377 minutes]:\r\ngithub.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc2080b2420)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x197\r\ncreated by github.com/syndtr/goleveldb/leveldb.openDB\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:148 +0xa03\r\n\r\ngoroutine 96 [select]:\r\ngithub.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc208062540)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:205 +0x21e\r\ncreated by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:236 +0x24c\r\n\r\ngoroutine 97 [select, 1374 minutes]:\r\ngithub.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc2080b2580)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x55a\r\ncreated by github.com/syndtr/goleveldb/leveldb.openDB\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x7cc\r\n\r\ngoroutine 98 [select]:\r\ngithub.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc2080b2580)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14a\r\ncreated by github.com/syndtr/goleveldb/leveldb.openDB\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7e6\r\n\r\ngoroutine 99 [select, 1374 minutes]:\r\ngithub.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc2080b2580)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x858\r\ncreated by github.com/syndtr/goleveldb/leveldb.openDB\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9cf\r\n\r\ngoroutine 100 [select, 1377 minutes]:\r\ngithub.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc2080b2580)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x283\r\ncreated by github.com/syndtr/goleveldb/leveldb.openDB\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9e9\r\n\r\ngoroutine 101 [select, 1377 minutes]:\r\ngithub.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc2080b2580)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x197\r\ncreated by github.com/syndtr/goleveldb/leveldb.openDB\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:148 +0xa03\r\n\r\ngoroutine 102 [select]:\r\ngithub.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc2080628c0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:205 +0x21e\r\ncreated by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:236 +0x24c\r\n\r\ngoroutine 103 [select, 7 minutes]:\r\ngithub.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc2080b2840)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x55a\r\ncreated by github.com/syndtr/goleveldb/leveldb.openDB\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x7cc\r\n\r\ngoroutine 104 [select]:\r\ngithub.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc2080b2840)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14a\r\ncreated by github.com/syndtr/goleveldb/leveldb.openDB\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7e6\r\n\r\ngoroutine 105 [select, 7 minutes]:\r\ngithub.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc2080b2840)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x858\r\ncreated by github.com/syndtr/goleveldb/leveldb.openDB\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9cf\r\n\r\ngoroutine 106 [select, 7 minutes]:\r\ngithub.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc2080b2840)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x283\r\ncreated by github.com/syndtr/goleveldb/leveldb.openDB\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9e9\r\n\r\ngoroutine 107 [select, 7 minutes]:\r\ngithub.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc2080b2840)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x197\r\ncreated by github.com/syndtr/goleveldb/leveldb.openDB\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:148 +0xa03\r\n\r\ngoroutine 108 [select]:\r\ngithub.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc2080627e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:205 +0x21e\r\ncreated by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:236 +0x24c\r\n\r\ngoroutine 109 [select, 1377 minutes]:\r\ngithub.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc2080b2b00)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x55a\r\ncreated by github.com/syndtr/goleveldb/leveldb.openDB\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x7cc\r\n\r\ngoroutine 110 [select]:\r\ngithub.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc2080b2b00)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14a\r\ncreated by github.com/syndtr/goleveldb/leveldb.openDB\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7e6\r\n\r\ngoroutine 111 [select, 1377 minutes]:\r\ngithub.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc2080b2b00)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x858\r\ncreated by github.com/syndtr/goleveldb/leveldb.openDB\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9cf\r\n\r\ngoroutine 112 [select, 1377 minutes]:\r\ngithub.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc2080b2b00)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x283\r\ncreated by github.com/syndtr/goleveldb/leveldb.openDB\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9e9\r\n\r\ngoroutine 113 [select, 1377 minutes]:\r\ngithub.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc2080b2b00)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x197\r\ncreated by github.com/syndtr/goleveldb/leveldb.openDB\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:148 +0xa03\r\n\r\ngoroutine 114 [select]:\r\ngithub.com/prometheus/prometheus/storage/local.(*persistence).processIndexingQueue(0xc2082b4900)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/persistence.go:1345 +0xba0\r\ngithub.com/prometheus/prometheus/storage/local.(*persistence).run(0xc2082b4900)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/persistence.go:282 +0x28\r\ncreated by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:246 +0x16a\r\n\r\ngoroutine 115 [select]:\r\ngithub.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).handleEvictList(0xc2080621c0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:623 +0x46e\r\ncreated by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:269 +0x502\r\n\r\ngoroutine 121 [select]:\r\ngithub.com/prometheus/prometheus/rules.(*Manager).Run(0xc208011c00)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/rules/manager.go:142 +0x3c6\r\ncreated by main.Main\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:131 +0x11a8\r\n\r\ngoroutine 118 [select]:\r\ngithub.com/prometheus/prometheus/storage/local.func·043()\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:761 +0x300\r\ncreated by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).cycleThroughMemoryFingerprints\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:777 +0xfb\r\n\r\ngoroutine 119 [select]:\r\ngithub.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).waitForNextFP(0xc2080621c0, 0x0, 0x3ff0000000000000, 0x0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:727 +0x199\r\ngithub.com/prometheus/prometheus/storage/local.func·044()\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:800 +0x285\r\ncreated by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).cycleThroughArchivedFingerprints\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:820 +0xfb\r\n\r\ngoroutine 122 [chan receive, 1376 minutes]:\r\ngithub.com/prometheus/prometheus/notification.(*NotificationHandler).Run(0xc20805a280)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/notification/notification.go:187 +0x81\r\ncreated by main.Main\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:134 +0x11eb\r\n\r\ngoroutine 133 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b905a0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 124 [IO wait]:\r\nnet.(*pollDesc).Wait(0xc2338206f0, 0x72, 0x0, 0x0)\r\n\t/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47\r\nnet.(*pollDesc).WaitRead(0xc2338206f0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43\r\nnet.(*netFD).accept(0xc233820690, 0x0, 0x7fcceb637e78, 0xc27e9b8f80)\r\n\t/usr/local/go/src/net/fd_unix.go:419 +0x40b\r\nnet.(*TCPListener).AcceptTCP(0xc231d53318, 0x61e11e, 0x0, 0x0)\r\n\t/usr/local/go/src/net/tcpsock_posix.go:234 +0x4e\r\nnet/http.tcpKeepAliveListener.Accept(0xc231d53318, 0x0, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/http/server.go:1976 +0x4c\r\nnet/http.(*Server).Serve(0xc23387d800, 0x7fcce8bb4438, 0xc231d53318, 0x0, 0x0)\r\n\t/usr/local/go/src/net/http/server.go:1728 +0x92\r\nnet/http.(*Server).ListenAndServe(0xc23387d800, 0x0, 0x0)\r\n\t/usr/local/go/src/net/http/server.go:1718 +0x154\r\nnet/http.ListenAndServe(0xb41980, 0x5, 0x7fcce8bb3348, 0xc20801f4a0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/http/server.go:1808 +0xba\r\ngithub.com/prometheus/prometheus/web.(*Handler).Run(0xc20800a9b0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/web/web.go:188 +0x158\r\ncreated by main.Main\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:142 +0x1294\r\n\r\ngoroutine 204 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc2264d0630, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 132 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b903f0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 131 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b90240, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 134 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b906c0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 135 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b90870, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 136 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b90a20, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 137 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b90bd0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 138 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b90d80, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 139 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b90ea0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 140 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b90fc0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 141 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b910e0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 142 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b91200, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 143 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b91320, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 144 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b91440, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 145 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b91560, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 146 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b91680, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 147 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b917a0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 148 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b918c0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 149 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b919e0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 150 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b91b00, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 151 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b91c20, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 152 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b91dd0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 153 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc226b91ef0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 154 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699c090, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 155 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699c1b0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 156 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699c360, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 157 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699c5a0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 158 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699c6c0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 159 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699c7e0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 160 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699cab0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 161 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699cbd0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 162 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699ccf0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 163 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699cf30, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 164 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699d050, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 165 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699d200, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 166 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699d320, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 167 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699d440, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 168 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699d560, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 169 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699d680, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 170 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699d7a0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 171 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699d950, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 172 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699da70, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 173 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699db90, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 174 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699dcb0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 175 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22699def0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 176 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679c090, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 177 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679c240, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 178 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679c360, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 179 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679c480, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 180 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679c5a0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 181 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679c6c0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 182 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679c7e0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 183 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679c900, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 184 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679cab0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 185 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679cc60, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 186 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679cd80, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 187 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679cea0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 188 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679cfc0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 189 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679d0e0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 190 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679d200, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 191 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679d3b0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 192 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679d560, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 193 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679d680, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 194 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679d7a0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 195 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679d8c0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 196 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679da70, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 197 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679db90, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 198 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679dcb0, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 199 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc22679de60, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 200 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc2264d0120, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 201 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc2264d0240, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 202 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc2264d0360, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 203 [select]:\r\ngithub.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc2264d0480, 0x7fcceb638d80, 0xc20801f3e0)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:290 +0x656\r\ncreated by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c\r\n\r\ngoroutine 209 [syscall, 1376 minutes, locked to thread]:\r\nruntime.goexit()\r\n\t/usr/local/go/src/runtime/asm_amd64.s:2232 +0x1\r\n\r\ngoroutine 2779142 [chan send, 3 minutes]:\r\ngithub.com/prometheus/client_golang/prometheus.computeApproximateRequestSize(0xc29a458680, 0xc2974be2a0, 0x28f)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/client_golang/prometheus/http.go:173 +0x1b9\r\ncreated by github.com/prometheus/client_golang/prometheus.func·008\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/client_golang/prometheus/http.go:143 +0x15a\r\n\r\ngoroutine 2779137 [chan send, 3 minutes]:\r\ngithub.com/prometheus/client_golang/prometheus.computeApproximateRequestSize(0xc28b13f860, 0xc295c1b980, 0x28f)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/client_golang/prometheus/http.go:173 +0x1b9\r\ncreated by github.com/prometheus/client_golang/prometheus.func·008\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/client_golang/prometheus/http.go:143 +0x15a\r\n\r\ngoroutine 2779139 [chan send, 3 minutes]:\r\ngithub.com/prometheus/client_golang/prometheus.computeApproximateRequestSize(0xc258fe7450, 0xc295c1be00, 0x28f)\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/client_golang/prometheus/http.go:173 +0x1b9\r\ncreated by github.com/prometheus/client_golang/prometheus.func·008\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/client_golang/prometheus/http.go:143 +0x15a\r\n\r\ngoroutine 2779151 [IO wait, 1 minutes]:\r\nnet.(*pollDesc).Wait(0xc29ad0a060, 0x72, 0x0, 0x0)\r\n\t/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47\r\nnet.(*pollDesc).WaitRead(0xc29ad0a060, 0x0, 0x0)\r\n\t/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43\r\nnet.(*netFD).Read(0xc29ad0a000, 0xc270996000, 0x1000, 0x1000, 0x0, 0x7fcceb637e78, 0xc265ebcdf8)\r\n\t/usr/local/go/src/net/fd_unix.go:242 +0x40f\r\nnet.(*conn).Read(0xc2708fc000, 0xc270996000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/net.go:121 +0xdc\r\nnet/http.(*liveSwitchReader).Read(0xc27094a048, 0xc270996000, 0x1000, 0x1000, 0xc2708c58b0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/http/server.go:214 +0xab\r\nio.(*LimitedReader).Read(0xc2709d6480, 0xc270996000, 0x1000, 0x1000, 0x7fcceb638da8, 0x0, 0x0)\r\n\t/usr/local/go/src/io/io.go:408 +0xce\r\nbufio.(*Reader).fill(0xc2709da000)\r\n\t/usr/local/go/src/bufio/bufio.go:97 +0x1ce\r\nbufio.(*Reader).ReadSlice(0xc2709da000, 0x5c320a, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/bufio/bufio.go:295 +0x257\r\nbufio.(*Reader).ReadLine(0xc2709da000, 0x0, 0x0, 0x0, 0xc234d5db00, 0x0, 0x0)\r\n\t/usr/local/go/src/bufio/bufio.go:324 +0x62\r\nnet/textproto.(*Reader).readLineSlice(0xc2709c4090, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/textproto/reader.go:55 +0x9e\r\nnet/textproto.(*Reader).ReadLine(0xc2709c4090, 0x0, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/textproto/reader.go:36 +0x4f\r\nnet/http.ReadRequest(0xc2709da000, 0xc26ca951e0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/http/request.go:598 +0xcb\r\nnet/http.(*conn).readRequest(0xc27094a000, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/http/server.go:586 +0x26f\r\nnet/http.(*conn).serve(0xc27094a000)\r\n\t/usr/local/go/src/net/http/server.go:1162 +0x69e\r\ncreated by net/http.(*Server).Serve\r\n\t/usr/local/go/src/net/http/server.go:1751 +0x35e\r\n\r\ngoroutine 106767 [IO wait, 1353 minutes]:\r\nnet.(*pollDesc).Wait(0xc2422715d0, 0x72, 0x0, 0x0)\r\n\t/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47\r\nnet.(*pollDesc).WaitRead(0xc2422715d0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43\r\nnet.(*netFD).Read(0xc242271570, 0xc23c842000, 0x1000, 0x1000, 0x0, 0x7fcceb637e78, 0xc241bab548)\r\n\t/usr/local/go/src/net/fd_unix.go:242 +0x40f\r\nnet.(*conn).Read(0xc243968fd8, 0xc23c842000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/net.go:121 +0xdc\r\nnet/http.(*liveSwitchReader).Read(0xc260f25ee8, 0xc23c842000, 0x1000, 0x1000, 0xc24da718b0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/http/server.go:214 +0xab\r\nio.(*LimitedReader).Read(0xc233bf80a0, 0xc23c842000, 0x1000, 0x1000, 0x7fcceb638da8, 0x0, 0x0)\r\n\t/usr/local/go/src/io/io.go:408 +0xce\r\nbufio.(*Reader).fill(0xc23dc86000)\r\n\t/usr/local/go/src/bufio/bufio.go:97 +0x1ce\r\nbufio.(*Reader).ReadSlice(0xc23dc86000, 0x5c320a, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/bufio/bufio.go:295 +0x257\r\nbufio.(*Reader).ReadLine(0xc23dc86000, 0x0, 0x0, 0x0, 0xc234d6e200, 0x0, 0x0)\r\n\t/usr/local/go/src/bufio/bufio.go:324 +0x62\r\nnet/textproto.(*Reader).readLineSlice(0xc243849f20, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/textproto/reader.go:55 +0x9e\r\nnet/textproto.(*Reader).ReadLine(0xc243849f20, 0x0, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/textproto/reader.go:36 +0x4f\r\nnet/http.ReadRequest(0xc23dc86000, 0xc24c8a4dd0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/http/request.go:598 +0xcb\r\nnet/http.(*conn).readRequest(0xc260f25ea0, 0x0, 0x0, 0x0)\r\n\t/usr/local/go/src/net/http/server.go:586 +0x26f\r\nnet/http.(*conn).serve(0xc260f25ea0)\r\n\t/usr/local/go/src/net/http/server.go:1162 +0x69e\r\ncreated by net/http.(*Server).Serve\r\n\t/usr/local/go/src/net/http/server.go:1751 +0x35e\r\n\r\ngoroutine 2727234 [chan send]:\r\ngithub.com/prometheus/prometheus/storage/local.func·030()\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/series.go:129 +0xec\r\ncreated by github.com/prometheus/prometheus/storage/local.(*seriesMap).fpIter\r\n\t/home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/series.go:134 +0xfb\r\n```\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/comments/170023554","html_url":"https://github.com/prometheus/prometheus/issues/877#issuecomment-170023554","issue_url":"https://api.github.com/repos/prometheus/prometheus/issues/877","id":170023554,"user":{"login":"atombender","id":50314,"avatar_url":"https://avatars.githubusercontent.com/u/50314?v=3","gravatar_id":"","url":"https://api.github.com/users/atombender","html_url":"https://github.com/atombender","followers_url":"https://api.github.com/users/atombender/followers","following_url":"https://api.github.com/users/atombender/following{/other_user}","gists_url":"https://api.github.com/users/atombender/gists{/gist_id}","starred_url":"https://api.github.com/users/atombender/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/atombender/subscriptions","organizations_url":"https://api.github.com/users/atombender/orgs","repos_url":"https://api.github.com/users/atombender/repos","events_url":"https://api.github.com/users/atombender/events{/privacy}","received_events_url":"https://api.github.com/users/atombender/received_events","type":"User","site_admin":false},"created_at":"2016-01-08T14:55:36Z","updated_at":"2016-01-08T14:55:36Z","body":"Considering there's apparently [no way to do consistent snapshots](https://github.com/prometheus/prometheus/issues/651) for backup purposes, it sounds like this problem might conceivably also arise if you try to restore a backup. I'm a bit worried now."}}