{"action":"created","issue":{"url":"https://api.github.com/repos/improbable-eng/thanos/issues/232","repository_url":"https://api.github.com/repos/improbable-eng/thanos","labels_url":"https://api.github.com/repos/improbable-eng/thanos/issues/232/labels{/name}","comments_url":"https://api.github.com/repos/improbable-eng/thanos/issues/232/comments","events_url":"https://api.github.com/repos/improbable-eng/thanos/issues/232/events","html_url":"https://github.com/improbable-eng/thanos/issues/232","id":302126811,"number":232,"title":"Query instance spawns a million goroutines while pulling label values from store instance","user":{"login":"mattbostock","id":4348848,"avatar_url":"https://avatars3.githubusercontent.com/u/4348848?v=4","gravatar_id":"","url":"https://api.github.com/users/mattbostock","html_url":"https://github.com/mattbostock","followers_url":"https://api.github.com/users/mattbostock/followers","following_url":"https://api.github.com/users/mattbostock/following{/other_user}","gists_url":"https://api.github.com/users/mattbostock/gists{/gist_id}","starred_url":"https://api.github.com/users/mattbostock/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mattbostock/subscriptions","organizations_url":"https://api.github.com/users/mattbostock/orgs","repos_url":"https://api.github.com/users/mattbostock/repos","events_url":"https://api.github.com/users/mattbostock/events{/privacy}","received_events_url":"https://api.github.com/users/mattbostock/received_events","type":"User","site_admin":false},"labels":[{"id":737763636,"url":"https://api.github.com/repos/improbable-eng/thanos/labels/bug","name":"bug","color":"ee0701","default":true}],"state":"open","locked":false,"assignee":{"login":"Bplotka","id":6950331,"avatar_url":"https://avatars2.githubusercontent.com/u/6950331?v=4","gravatar_id":"","url":"https://api.github.com/users/Bplotka","html_url":"https://github.com/Bplotka","followers_url":"https://api.github.com/users/Bplotka/followers","following_url":"https://api.github.com/users/Bplotka/following{/other_user}","gists_url":"https://api.github.com/users/Bplotka/gists{/gist_id}","starred_url":"https://api.github.com/users/Bplotka/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Bplotka/subscriptions","organizations_url":"https://api.github.com/users/Bplotka/orgs","repos_url":"https://api.github.com/users/Bplotka/repos","events_url":"https://api.github.com/users/Bplotka/events{/privacy}","received_events_url":"https://api.github.com/users/Bplotka/received_events","type":"User","site_admin":false},"assignees":[{"login":"Bplotka","id":6950331,"avatar_url":"https://avatars2.githubusercontent.com/u/6950331?v=4","gravatar_id":"","url":"https://api.github.com/users/Bplotka","html_url":"https://github.com/Bplotka","followers_url":"https://api.github.com/users/Bplotka/followers","following_url":"https://api.github.com/users/Bplotka/following{/other_user}","gists_url":"https://api.github.com/users/Bplotka/gists{/gist_id}","starred_url":"https://api.github.com/users/Bplotka/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Bplotka/subscriptions","organizations_url":"https://api.github.com/users/Bplotka/orgs","repos_url":"https://api.github.com/users/Bplotka/repos","events_url":"https://api.github.com/users/Bplotka/events{/privacy}","received_events_url":"https://api.github.com/users/Bplotka/received_events","type":"User","site_admin":false}],"milestone":null,"comments":17,"created_at":"2018-03-04T22:01:03Z","updated_at":"2018-03-14T14:21:53Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Using the tip of master at ed6099df6b21c, my query instance is being OOM-killed, apparently while label values are being loaded via gRPC from the store instance.\r\n\r\nHere's the output from 'tsdb ls' on a downloaded copy of the data stored in Ceph:\r\n```\r\nBLOCK ULID                  MIN TIME       MAX TIME       NUM SAMPLES  NUM CHUNKS  NUM SERIES\r\n01C7S44VPRB7MBX7SBF9663KQ8  1520187000000  1520187600000  3600733      2859075     2843911\r\n```\r\n\r\nThere are around 2.8M time-series and 6k datapoints per second on average.\r\n\r\nThe block size is only 10 minutes for reasons not really relevant to this issue (a short block duration will have a negative performance impact when querying across multiple blocks, but right now I'm only working with this one block).\r\n\r\nTest deployment overview:\r\n\r\n1 store instance\r\n2 query instaces\r\n1 compactor instance\r\n\r\nThe query instances have around 16GB memory allocated. I grabbed a profile of the query instance's goroutines from the `/debug/profile` endpoint just before the process was OOM-killed. You can see there are 495k goroutines that appear to be calling `(*storeClient).LabelValues`.\r\n\r\n<details><summary>Query instance goroutine profile</summary>\r\n\r\n```\r\ngoroutine profile: total 990321\r\n495141 @ 0x42d16a 0x43cfb0 0x7da9f2 0x7dac3f 0x7ea68f 0x7ebe42 0x9b8bef 0x9fe015 0x9b8b25 0x9bd419 0x9b8e0f 0x7eb2ad 0x7eb461 0xb2cc42 0xbae55f 0x45ad21\r\n#\t0x7da9f1\tgithub.com/improbable-eng/thanos/vendor/google.golang.org/grpc/transport.(*Stream).waitOnHeader+0x171\t\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/vendor/google.golang.org/grpc/transport/transport.go:258\r\n#\t0x7dac3e\tgithub.com/improbable-eng/thanos/vendor/google.golang.org/grpc/transport.(*Stream).Header+0x2e\t\t\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/vendor/google.golang.org/grpc/transport/transport.go:298\r\n#\t0x7ea68e\tgithub.com/improbable-eng/thanos/vendor/google.golang.org/grpc.recvResponse+0x9e\t\t\t\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/vendor/google.golang.org/grpc/call.go:50\r\n#\t0x7ebe41\tgithub.com/improbable-eng/thanos/vendor/google.golang.org/grpc.invoke+0x9b1\t\t\t\t\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/vendor/google.golang.org/grpc/call.go:302\r\n#\t0x9b8bee\tgithub.com/improbable-eng/thanos/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryClient.func1.1+0x1ce\t\t\t\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:106\r\n#\t0x9fe014\tgithub.com/improbable-eng/thanos/vendor/github.com/grpc-ecosystem/go-grpc-middleware/tracing/opentracing.UnaryClientInterceptor.func1+0x144\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/grpc-ecosystem/go-grpc-middleware/tracing/opentracing/client_interceptors.go:28\r\n#\t0x9b8b24\tgithub.com/improbable-eng/thanos/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryClient.func1.1+0x104\t\t\t\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:109\r\n#\t0x9bd418\tgithub.com/improbable-eng/thanos/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.(*ClientMetrics).UnaryClientInterceptor.func1+0x148\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/client_metrics.go:110\r\n#\t0x9b8e0e\tgithub.com/improbable-eng/thanos/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryClient.func1+0x1de\t\t\t\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:112\r\n#\t0x7eb2ac\tgithub.com/improbable-eng/thanos/vendor/google.golang.org/grpc.(*ClientConn).Invoke+0xdc\t\t\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/vendor/google.golang.org/grpc/call.go:149\r\n#\t0x7eb460\tgithub.com/improbable-eng/thanos/vendor/google.golang.org/grpc.Invoke+0xc0\t\t\t\t\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/vendor/google.golang.org/grpc/call.go:159\r\n#\t0xb2cc41\tgithub.com/improbable-eng/thanos/pkg/store/storepb.(*storeClient).LabelValues+0xd1\t\t\t\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/pkg/store/storepb/rpc.pb.go:360\r\n#\t0xbae55e\tgithub.com/improbable-eng/thanos/pkg/store.(*ProxyStore).LabelValues.func1+0xde\t\t\t\t\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/pkg/store/proxy.go:251\r\n\r\n495140 @ 0x42d16a 0x42d21e 0x43dc34 0x43d859 0x471d92 0xbaadcc 0xb3b6f6 0x9b8354 0x9ba067 0x9b82e7 0x9fe67a 0xa015c1 0x9b82e7 0x9bd96b 0x9b851c 0xb2d5c7 0x7ff85c 0x803168 0x80a0af 0x45ad21\r\n#\t0x43d858\tsync.runtime_Semacquire+0x38\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/runtime/sema.go:56\r\n#\t0x471d91\tsync.(*WaitGroup).Wait+0x71\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/sync/waitgroup.go:129\r\n#\t0xbaadcb\tgithub.com/improbable-eng/thanos/pkg/store.(*ProxyStore).LabelValues+0x1cb\t\t\t\t\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/pkg/store/proxy.go:270\r\n#\t0xb3b6f5\tgithub.com/improbable-eng/thanos/pkg/store/storepb._Store_LabelValues_Handler.func1+0x85\t\t\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/pkg/store/storepb/rpc.pb.go:451\r\n#\t0x9b8353\tgithub.com/improbable-eng/thanos/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1+0x103\t\t\t\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:31\r\n#\t0x9ba066\tgithub.com/improbable-eng/thanos/vendor/github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1+0x96\t\t\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/grpc-ecosystem/go-grpc-middleware/recovery/interceptors.go:25\r\n#\t0x9b82e6\tgithub.com/improbable-eng/thanos/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1+0x96\t\t\t\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34\r\n#\t0x9fe679\tgithub.com/improbable-eng/thanos/vendor/github.com/grpc-ecosystem/go-grpc-middleware/tracing/opentracing.UnaryServerInterceptor.func1+0xd9\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/grpc-ecosystem/go-grpc-middleware/tracing/opentracing/server_interceptors.go:30\r\n#\t0xa015c0\tgithub.com/improbable-eng/thanos/pkg/tracing.UnaryServerInterceptor.func1+0x100\t\t\t\t\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/pkg/tracing/grpc.go:25\r\n#\t0x9b82e6\tgithub.com/improbable-eng/thanos/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1+0x96\t\t\t\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34\r\n#\t0x9bd96a\tgithub.com/improbable-eng/thanos/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1+0xda\t\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/server_metrics.go:112\r\n#\t0x9b851b\tgithub.com/improbable-eng/thanos/vendor/github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1+0x16b\t\t\t\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:37\r\n#\t0xb2d5c6\tgithub.com/improbable-eng/thanos/pkg/store/storepb._Store_LabelValues_Handler+0x166\t\t\t\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/pkg/store/storepb/rpc.pb.go:453\r\n#\t0x7ff85b\tgithub.com/improbable-eng/thanos/vendor/google.golang.org/grpc.(*Server).processUnaryRPC+0x8ab\t\t\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/vendor/google.golang.org/grpc/server.go:921\r\n#\t0x803167\tgithub.com/improbable-eng/thanos/vendor/google.golang.org/grpc.(*Server).handleStream+0x1317\t\t\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/vendor/google.golang.org/grpc/server.go:1143\r\n#\t0x80a0ae\tgithub.com/improbable-eng/thanos/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1+0x9e\t\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/vendor/google.golang.org/grpc/server.go:638\r\n\r\n2 @ 0x42d16a 0x42848a 0x427b07 0x494f9b 0x49501d 0x495e7d 0x5ba03f 0x5cb77a 0x573f58 0x472246 0x4723b8 0x79cfbb 0x79da24 0x7cc1e1 0x45ad21\r\n#\t0x427b06\tinternal/poll.runtime_pollWait+0x56\t\t\t\t\t\t\t\t\t/usr/local/go/src/runtime/netpoll.go:173\r\n#\t0x494f9a\tinternal/poll.(*pollDesc).wait+0x9a\t\t\t\t\t\t\t\t\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:85\r\n#\t0x49501c\tinternal/poll.(*pollDesc).waitRead+0x3c\t\t\t\t\t\t\t\t\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:90\r\n#\t0x495e7c\tinternal/poll.(*FD).Read+0x17c\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/internal/poll/fd_unix.go:157\r\n#\t0x5ba03e\tnet.(*netFD).Read+0x4e\t\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/net/fd_unix.go:202\r\n#\t0x5cb779\tnet.(*conn).Read+0x69\t\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/net/net.go:176\r\n#\t0x573f57\tbufio.(*Reader).Read+0x237\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/bufio/bufio.go:216\r\n#\t0x472245\tio.ReadAtLeast+0x85\t\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/io/io.go:309\r\n#\t0x4723b7\tio.ReadFull+0x57\t\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/io/io.go:327\r\n#\t0x79cfba\tgithub.com/improbable-eng/thanos/vendor/golang.org/x/net/http2.readFrameHeader+0x7a\t\t\t/go/src/github.com/improbable-eng/thanos/vendor/golang.org/x/net/http2/frame.go:237\r\n#\t0x79da23\tgithub.com/improbable-eng/thanos/vendor/golang.org/x/net/http2.(*Framer).ReadFrame+0xa3\t\t\t/go/src/github.com/improbable-eng/thanos/vendor/golang.org/x/net/http2/frame.go:492\r\n#\t0x7cc1e0\tgithub.com/improbable-eng/thanos/vendor/google.golang.org/grpc/transport.(*http2Client).reader+0xe0\t/go/src/github.com/improbable-eng/thanos/vendor/google.golang.org/grpc/transport/http2_client.go:1173\r\n\r\n2 @ 0x42d16a 0x42848a 0x427b07 0x494f9b 0x49501d 0x495e7d 0x5ba03f 0x5cb77a 0x573f58 0x472246 0x4723b8 0x79cfbb 0x79da24 0x7d0277 0x7fe41b 0x809f9c 0x809fd7 0x45ad21\r\n#\t0x427b06\tinternal/poll.runtime_pollWait+0x56\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/runtime/netpoll.go:173\r\n#\t0x494f9a\tinternal/poll.(*pollDesc).wait+0x9a\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:85\r\n#\t0x49501c\tinternal/poll.(*pollDesc).waitRead+0x3c\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:90\r\n#\t0x495e7c\tinternal/poll.(*FD).Read+0x17c\t\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/internal/poll/fd_unix.go:157\r\n#\t0x5ba03e\tnet.(*netFD).Read+0x4e\t\t\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/net/fd_unix.go:202\r\n#\t0x5cb779\tnet.(*conn).Read+0x69\t\t\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/net/net.go:176\r\n#\t0x573f57\tbufio.(*Reader).Read+0x237\t\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/bufio/bufio.go:216\r\n#\t0x472245\tio.ReadAtLeast+0x85\t\t\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/io/io.go:309\r\n#\t0x4723b7\tio.ReadFull+0x57\t\t\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/io/io.go:327\r\n#\t0x79cfba\tgithub.com/improbable-eng/thanos/vendor/golang.org/x/net/http2.readFrameHeader+0x7a\t\t\t\t/go/src/github.com/improbable-eng/thanos/vendor/golang.org/x/net/http2/frame.go:237\r\n#\t0x79da23\tgithub.com/improbable-eng/thanos/vendor/golang.org/x/net/http2.(*Framer).ReadFrame+0xa3\t\t\t\t/go/src/github.com/improbable-eng/thanos/vendor/golang.org/x/net/http2/frame.go:492\r\n#\t0x7d0276\tgithub.com/improbable-eng/thanos/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams+0x36\t/go/src/github.com/improbable-eng/thanos/vendor/google.golang.org/grpc/transport/http2_server.go:395\r\n#\t0x7fe41a\tgithub.com/improbable-eng/thanos/vendor/google.golang.org/grpc.(*Server).serveStreams+0xea\t\t\t/go/src/github.com/improbable-eng/thanos/vendor/google.golang.org/grpc/server.go:634\r\n#\t0x809f9b\tgithub.com/improbable-eng/thanos/vendor/google.golang.org/grpc.(*Server).handleRawConn.func2+0x3b\t\t/go/src/github.com/improbable-eng/thanos/vendor/google.golang.org/grpc/server.go:591\r\n#\t0x809fd6\tgithub.com/improbable-eng/thanos/vendor/google.golang.org/grpc.(*Server).handleRawConn.func3+0x26\t\t/go/src/github.com/improbable-eng/thanos/vendor/google.golang.org/grpc/server.go:599\r\n\r\n2 @ 0x42d16a 0x42848a 0x427b07 0x494f9b 0x49501d 0x495e7d 0x5ba03f 0x5cb77a 0x6b0bfa 0x45ad21\r\n#\t0x427b06\tinternal/poll.runtime_pollWait+0x56\t\t/usr/local/go/src/runtime/netpoll.go:173\r\n#\t0x494f9a\tinternal/poll.(*pollDesc).wait+0x9a\t\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:85\r\n#\t0x49501c\tinternal/poll.(*pollDesc).waitRead+0x3c\t\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:90\r\n#\t0x495e7c\tinternal/poll.(*FD).Read+0x17c\t\t\t/usr/local/go/src/internal/poll/fd_unix.go:157\r\n#\t0x5ba03e\tnet.(*netFD).Read+0x4e\t\t\t\t/usr/local/go/src/net/fd_unix.go:202\r\n#\t0x5cb779\tnet.(*conn).Read+0x69\t\t\t\t/usr/local/go/src/net/net.go:176\r\n#\t0x6b0bf9\tnet/http.(*connReader).backgroundRead+0x59\t/usr/local/go/src/net/http/server.go:668\r\n\r\n2 @ 0x42d16a 0x43cfb0 0x7d4e4b 0x45ad21\r\n#\t0x7d4e4a\tgithub.com/improbable-eng/thanos/vendor/google.golang.org/grpc/transport.(*http2Server).keepalive+0x23a\t/go/src/github.com/improbable-eng/thanos/vendor/google.golang.org/grpc/transport/http2_server.go:956\r\n\r\n2 @ 0x42d16a 0x43cfb0 0x7dbea8 0x7dd4de 0x45ad21\r\n#\t0x7dbea7\tgithub.com/improbable-eng/thanos/vendor/google.golang.org/grpc/transport.loopyWriter+0x367\t\t/go/src/github.com/improbable-eng/thanos/vendor/google.golang.org/grpc/transport/transport.go:727\r\n#\t0x7dd4dd\tgithub.com/improbable-eng/thanos/vendor/google.golang.org/grpc/transport.newHTTP2Client.func3+0x5d\t/go/src/github.com/improbable-eng/thanos/vendor/google.golang.org/grpc/transport/http2_client.go:305\r\n\r\n2 @ 0x42d16a 0x43cfb0 0x7dbea8 0x7dddfe 0x45ad21\r\n#\t0x7dbea7\tgithub.com/improbable-eng/thanos/vendor/google.golang.org/grpc/transport.loopyWriter+0x367\t\t/go/src/github.com/improbable-eng/thanos/vendor/google.golang.org/grpc/transport/transport.go:727\r\n#\t0x7dddfd\tgithub.com/improbable-eng/thanos/vendor/google.golang.org/grpc/transport.newHTTP2Server.func2+0x5d\t/go/src/github.com/improbable-eng/thanos/vendor/google.golang.org/grpc/transport/http2_server.go:261\r\n\r\n2 @ 0x42d16a 0x43cfb0 0x7e958a 0x45ad21\r\n#\t0x7e9589\tgithub.com/improbable-eng/thanos/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher+0x149\t/go/src/github.com/improbable-eng/thanos/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:122\r\n\r\n2 @ 0x42d16a 0x43cfb0 0x7f1945 0x809665 0x45ad21\r\n#\t0x7f1944\tgithub.com/improbable-eng/thanos/vendor/google.golang.org/grpc.(*addrConn).transportMonitor+0x234\t/go/src/github.com/improbable-eng/thanos/vendor/google.golang.org/grpc/clientconn.go:1234\r\n#\t0x809664\tgithub.com/improbable-eng/thanos/vendor/google.golang.org/grpc.(*addrConn).connect.func1+0x1b4\t\t/go/src/github.com/improbable-eng/thanos/vendor/google.golang.org/grpc/clientconn.go:837\r\n\r\n2 @ 0x42d16a 0x43cfb0 0x7f9d72 0x45ad21\r\n#\t0x7f9d71\tgithub.com/improbable-eng/thanos/vendor/google.golang.org/grpc.(*ccResolverWrapper).watcher+0x181\t/go/src/github.com/improbable-eng/thanos/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:110\r\n\r\n2 @ 0x42d16a 0x43cfb0 0xb05118 0x45ad21\r\n#\t0xb05117\tgithub.com/improbable-eng/thanos/vendor/github.com/hashicorp/memberlist.(*Memberlist).triggerFunc+0x1a7\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/hashicorp/memberlist/state.go:128\r\n\r\n1 @ 0x40f972 0x441d46 0x9b79e2 0x45ad21\r\n#\t0x441d45\tos/signal.signal_recv+0xa5\t/usr/local/go/src/runtime/sigqueue.go:139\r\n#\t0x9b79e1\tos/signal.loop+0x21\t\t/usr/local/go/src/os/signal/signal_unix.go:22\r\n\r\n1 @ 0x42d16a 0x42848a 0x427b07 0x494f9b 0x49501d 0x495e7d 0x5ba03f 0x5cb77a 0x6b10a8 0x5738ce 0x5745ac 0x5747c4 0x63af50 0x63ad5b 0x6aba4c 0x6b234f 0x6b604c 0x45ad21\r\n#\t0x427b06\tinternal/poll.runtime_pollWait+0x56\t\t/usr/local/go/src/runtime/netpoll.go:173\r\n#\t0x494f9a\tinternal/poll.(*pollDesc).wait+0x9a\t\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:85\r\n#\t0x49501c\tinternal/poll.(*pollDesc).waitRead+0x3c\t\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:90\r\n#\t0x495e7c\tinternal/poll.(*FD).Read+0x17c\t\t\t/usr/local/go/src/internal/poll/fd_unix.go:157\r\n#\t0x5ba03e\tnet.(*netFD).Read+0x4e\t\t\t\t/usr/local/go/src/net/fd_unix.go:202\r\n#\t0x5cb779\tnet.(*conn).Read+0x69\t\t\t\t/usr/local/go/src/net/net.go:176\r\n#\t0x6b10a7\tnet/http.(*connReader).Read+0xf7\t\t/usr/local/go/src/net/http/server.go:764\r\n#\t0x5738cd\tbufio.(*Reader).fill+0x11d\t\t\t/usr/local/go/src/bufio/bufio.go:100\r\n#\t0x5745ab\tbufio.(*Reader).ReadSlice+0x2b\t\t\t/usr/local/go/src/bufio/bufio.go:341\r\n#\t0x5747c3\tbufio.(*Reader).ReadLine+0x33\t\t\t/usr/local/go/src/bufio/bufio.go:370\r\n#\t0x63af4f\tnet/textproto.(*Reader).readLineSlice+0x6f\t/usr/local/go/src/net/textproto/reader.go:55\r\n#\t0x63ad5a\tnet/textproto.(*Reader).ReadLine+0x2a\t\t/usr/local/go/src/net/textproto/reader.go:36\r\n#\t0x6aba4b\tnet/http.readRequest+0x8b\t\t\t/usr/local/go/src/net/http/request.go:929\r\n#\t0x6b234e\tnet/http.(*conn).readRequest+0x16e\t\t/usr/local/go/src/net/http/server.go:944\r\n#\t0x6b604b\tnet/http.(*conn).serve+0x4db\t\t\t/usr/local/go/src/net/http/server.go:1768\r\n\r\n1 @ 0x42d16a 0x42848a 0x427b07 0x494f9b 0x49501d 0x49628d 0x5ba18b 0x5d819a 0x5d626f 0xb029f5 0x45ad21\r\n#\t0x427b06\tinternal/poll.runtime_pollWait+0x56\t\t\t\t\t\t\t\t\t/usr/local/go/src/runtime/netpoll.go:173\r\n#\t0x494f9a\tinternal/poll.(*pollDesc).wait+0x9a\t\t\t\t\t\t\t\t\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:85\r\n#\t0x49501c\tinternal/poll.(*pollDesc).waitRead+0x3c\t\t\t\t\t\t\t\t\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:90\r\n#\t0x49628c\tinternal/poll.(*FD).ReadFrom+0x17c\t\t\t\t\t\t\t\t\t/usr/local/go/src/internal/poll/fd_unix.go:207\r\n#\t0x5ba18a\tnet.(*netFD).readFrom+0x5a\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/net/fd_unix.go:208\r\n#\t0x5d8199\tnet.(*UDPConn).readFrom+0x69\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/net/udpsock_posix.go:47\r\n#\t0x5d626e\tnet.(*UDPConn).ReadFrom+0x6e\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/net/udpsock.go:118\r\n#\t0xb029f4\tgithub.com/improbable-eng/thanos/vendor/github.com/hashicorp/memberlist.(*NetTransport).udpListen+0xc4\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/hashicorp/memberlist/net_transport.go:247\r\n\r\n1 @ 0x42d16a 0x42848a 0x427b07 0x494f9b 0x49501d 0x497418 0x5ba952 0x5d536e 0x5d3949 0xb0284f 0x45ad21\r\n#\t0x427b06\tinternal/poll.runtime_pollWait+0x56\t\t\t\t\t\t\t\t\t/usr/local/go/src/runtime/netpoll.go:173\r\n#\t0x494f9a\tinternal/poll.(*pollDesc).wait+0x9a\t\t\t\t\t\t\t\t\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:85\r\n#\t0x49501c\tinternal/poll.(*pollDesc).waitRead+0x3c\t\t\t\t\t\t\t\t\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:90\r\n#\t0x497417\tinternal/poll.(*FD).Accept+0x1a7\t\t\t\t\t\t\t\t\t/usr/local/go/src/internal/poll/fd_unix.go:372\r\n#\t0x5ba951\tnet.(*netFD).accept+0x41\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/net/fd_unix.go:238\r\n#\t0x5d536d\tnet.(*TCPListener).accept+0x2d\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/net/tcpsock_posix.go:136\r\n#\t0x5d3948\tnet.(*TCPListener).AcceptTCP+0x48\t\t\t\t\t\t\t\t\t/usr/local/go/src/net/tcpsock.go:246\r\n#\t0xb0284e\tgithub.com/improbable-eng/thanos/vendor/github.com/hashicorp/memberlist.(*NetTransport).tcpListen+0x5e\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/hashicorp/memberlist/net_transport.go:225\r\n\r\n1 @ 0x42d16a 0x42848a 0x427b07 0x494f9b 0x49501d 0x497418 0x5ba952 0x5d536e 0x5d3b59 0x6ba335 0x6b9273 0xc1eaac 0x8cd2d7 0x45ad21\r\n#\t0x427b06\tinternal/poll.runtime_pollWait+0x56\t\t\t\t\t\t\t/usr/local/go/src/runtime/netpoll.go:173\r\n#\t0x494f9a\tinternal/poll.(*pollDesc).wait+0x9a\t\t\t\t\t\t\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:85\r\n#\t0x49501c\tinternal/poll.(*pollDesc).waitRead+0x3c\t\t\t\t\t\t\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:90\r\n#\t0x497417\tinternal/poll.(*FD).Accept+0x1a7\t\t\t\t\t\t\t/usr/local/go/src/internal/poll/fd_unix.go:372\r\n#\t0x5ba951\tnet.(*netFD).accept+0x41\t\t\t\t\t\t\t\t/usr/local/go/src/net/fd_unix.go:238\r\n#\t0x5d536d\tnet.(*TCPListener).accept+0x2d\t\t\t\t\t\t\t\t/usr/local/go/src/net/tcpsock_posix.go:136\r\n#\t0x5d3b58\tnet.(*TCPListener).Accept+0x48\t\t\t\t\t\t\t\t/usr/local/go/src/net/tcpsock.go:259\r\n#\t0x6ba334\tnet/http.(*Server).Serve+0x1a4\t\t\t\t\t\t\t\t/usr/local/go/src/net/http/server.go:2770\r\n#\t0x6b9272\tnet/http.Serve+0x72\t\t\t\t\t\t\t\t\t/usr/local/go/src/net/http/server.go:2389\r\n#\t0xc1eaab\tmain.runQuery.func6+0x4b\t\t\t\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/cmd/thanos/query.go:177\r\n#\t0x8cd2d6\tgithub.com/improbable-eng/thanos/vendor/github.com/oklog/run.(*Group).Run.func1+0x26\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/oklog/run/group.go:38\r\n\r\n1 @ 0x42d16a 0x42848a 0x427b07 0x494f9b 0x49501d 0x497418 0x5ba952 0x5d536e 0x5d3b59 0x7fd235 0xc1eb8c 0x8cd2d7 0x45ad21\r\n#\t0x427b06\tinternal/poll.runtime_pollWait+0x56\t\t\t\t\t\t\t/usr/local/go/src/runtime/netpoll.go:173\r\n#\t0x494f9a\tinternal/poll.(*pollDesc).wait+0x9a\t\t\t\t\t\t\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:85\r\n#\t0x49501c\tinternal/poll.(*pollDesc).waitRead+0x3c\t\t\t\t\t\t\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:90\r\n#\t0x497417\tinternal/poll.(*FD).Accept+0x1a7\t\t\t\t\t\t\t/usr/local/go/src/internal/poll/fd_unix.go:372\r\n#\t0x5ba951\tnet.(*netFD).accept+0x41\t\t\t\t\t\t\t\t/usr/local/go/src/net/fd_unix.go:238\r\n#\t0x5d536d\tnet.(*TCPListener).accept+0x2d\t\t\t\t\t\t\t\t/usr/local/go/src/net/tcpsock_posix.go:136\r\n#\t0x5d3b58\tnet.(*TCPListener).Accept+0x48\t\t\t\t\t\t\t\t/usr/local/go/src/net/tcpsock.go:259\r\n#\t0x7fd234\tgithub.com/improbable-eng/thanos/vendor/google.golang.org/grpc.(*Server).Serve+0x1d4\t/go/src/github.com/improbable-eng/thanos/vendor/google.golang.org/grpc/server.go:504\r\n#\t0xc1eb8b\tmain.runQuery.func8+0x3b\t\t\t\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/cmd/thanos/query.go:194\r\n#\t0x8cd2d6\tgithub.com/improbable-eng/thanos/vendor/github.com/oklog/run.(*Group).Run.func1+0x26\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/oklog/run/group.go:38\r\n\r\n1 @ 0x42d16a 0x42d21e 0x403a7b 0x403815 0x884d6f 0x45ad21\r\n#\t0x884d6e\tgithub.com/improbable-eng/thanos/vendor/github.com/prometheus/client_golang/prometheus.computeApproximateRequestSize+0x12e\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/prometheus/client_golang/prometheus/http.go:316\r\n\r\n1 @ 0x42d16a 0x42d21e 0x4046d2 0x40438b 0x8cd1ac 0xc02e34 0x42cd12 0x45ad21\r\n#\t0x8cd1ab\tgithub.com/improbable-eng/thanos/vendor/github.com/oklog/run.(*Group).Run+0xeb\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/oklog/run/group.go:43\r\n#\t0xc02e33\tmain.main+0x1153\t\t\t\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/cmd/thanos/main.go:146\r\n#\t0x42cd11\truntime.main+0x211\t\t\t\t\t\t\t\t/usr/local/go/src/runtime/proc.go:198\r\n\r\n1 @ 0x42d16a 0x42d21e 0x4046d2 0x40438b 0xc1def0 0x8cd2d7 0x45ad21\r\n#\t0xc1deef\tmain.main.func1+0x4f\t\t\t\t\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/cmd/thanos/main.go:123\r\n#\t0x8cd2d6\tgithub.com/improbable-eng/thanos/vendor/github.com/oklog/run.(*Group).Run.func1+0x26\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/oklog/run/group.go:38\r\n\r\n1 @ 0x42d16a 0x42d21e 0x43dc34 0x43d859 0x471d92 0xbaadcc 0xb4d60d 0xb8b2e1 0xb8e2d4 0xb8d7c5 0x6b71a4 0xb50a29 0x6b71a4 0xa01d27 0x6b71a4 0x6d76ad 0x89217f 0x996262 0x992501 0x99602c 0x6b8e10 0x6b9f6c 0x6b61c1 0x45ad21\r\n#\t0x43d858\tsync.runtime_Semacquire+0x38\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/runtime/sema.go:56\r\n#\t0x471d91\tsync.(*WaitGroup).Wait+0x71\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/sync/waitgroup.go:129\r\n#\t0xbaadcb\tgithub.com/improbable-eng/thanos/pkg/store.(*ProxyStore).LabelValues+0x1cb\t\t\t\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/pkg/store/proxy.go:270\r\n#\t0xb4d60c\tgithub.com/improbable-eng/thanos/pkg/query.(*querier).LabelValues+0x13c\t\t\t\t\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/pkg/query/querier.go:236\r\n#\t0xb8b2e0\tgithub.com/improbable-eng/thanos/pkg/query/api.(*API).labelValues+0x300\t\t\t\t\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/pkg/query/api/v1.go:368\r\n#\t0xb8e2d3\tgithub.com/improbable-eng/thanos/pkg/query/api.(*API).(github.com/improbable-eng/thanos/pkg/query/api.labelValues)-fm+0x33\t\t/go/src/github.com/improbable-eng/thanos/pkg/query/api/v1.go:163\r\n#\t0xb8d7c4\tgithub.com/improbable-eng/thanos/pkg/query/api.(*API).Register.func1.1+0x54\t\t\t\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/pkg/query/api/v1.go:147\r\n#\t0x6b71a3\tnet/http.HandlerFunc.ServeHTTP+0x43\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/net/http/server.go:1947\r\n#\t0xb50a28\tgithub.com/improbable-eng/thanos/vendor/github.com/NYTimes/gziphandler.GzipHandlerWithOpts.func1.1+0x178\t\t\t\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/NYTimes/gziphandler/gzip.go:277\r\n#\t0x6b71a3\tnet/http.HandlerFunc.ServeHTTP+0x43\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/net/http/server.go:1947\r\n#\t0xa01d26\tgithub.com/improbable-eng/thanos/pkg/tracing.HTTPMiddleware.func1+0x576\t\t\t\t\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/pkg/tracing/http.go:37\r\n#\t0x6b71a3\tnet/http.HandlerFunc.ServeHTTP+0x43\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/net/http/server.go:1947\r\n#\t0x6d76ac\tnet/http.(Handler).ServeHTTP-fm+0x4c\t\t\t\t\t\t\t\t\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/prometheus/client_golang/prometheus/http.go:179\r\n#\t0x89217e\tgithub.com/improbable-eng/thanos/vendor/github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1+0x26e\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/prometheus/client_golang/prometheus/http.go:287\r\n#\t0x996261\tgithub.com/improbable-eng/thanos/vendor/github.com/prometheus/common/route.(*Router).handle.func1+0x221\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/prometheus/common/route/route.go:50\r\n#\t0x992500\tgithub.com/improbable-eng/thanos/vendor/github.com/julienschmidt/httprouter.(*Router).ServeHTTP+0x6c0\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/julienschmidt/httprouter/router.go:299\r\n#\t0x99602b\tgithub.com/improbable-eng/thanos/vendor/github.com/prometheus/common/route.(*Router).ServeHTTP+0x4b\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/prometheus/common/route/route.go:88\r\n#\t0x6b8e0f\tnet/http.(*ServeMux).ServeHTTP+0x12f\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/net/http/server.go:2337\r\n#\t0x6b9f6b\tnet/http.serverHandler.ServeHTTP+0xbb\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/net/http/server.go:2694\r\n#\t0x6b61c0\tnet/http.(*conn).serve+0x650\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/net/http/server.go:1830\r\n\r\n1 @ 0x42d16a 0x43cfb0 0x457924 0x45ad21\r\n#\t0x42d169\truntime.gopark+0x119\t\t/usr/local/go/src/runtime/proc.go:291\r\n#\t0x43cfaf\truntime.selectgo+0xe4f\t\t/usr/local/go/src/runtime/select.go:392\r\n#\t0x457923\truntime.ensureSigM.func1+0x1f3\t/usr/local/go/src/runtime/signal_unix.go:549\r\n\r\n1 @ 0x42d16a 0x43cfb0 0x9a2841 0xc1e880 0x8cd2d7 0x45ad21\r\n#\t0x9a2840\tgithub.com/improbable-eng/thanos/pkg/runutil.Repeat+0x130\t\t\t\t/go/src/github.com/improbable-eng/thanos/pkg/runutil/runutil.go:15\r\n#\t0xc1e87f\tmain.runQuery.func2+0x9f\t\t\t\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/cmd/thanos/query.go:138\r\n#\t0x8cd2d6\tgithub.com/improbable-eng/thanos/vendor/github.com/oklog/run.(*Group).Run.func1+0x26\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/oklog/run/group.go:38\r\n\r\n1 @ 0x42d16a 0x43cfb0 0x9a2841 0xc1e9f0 0x8cd2d7 0x45ad21\r\n#\t0x9a2840\tgithub.com/improbable-eng/thanos/pkg/runutil.Repeat+0x130\t\t\t\t/go/src/github.com/improbable-eng/thanos/pkg/runutil/runutil.go:15\r\n#\t0xc1e9ef\tmain.runQuery.func4+0x9f\t\t\t\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/cmd/thanos/query.go:150\r\n#\t0x8cd2d6\tgithub.com/improbable-eng/thanos/vendor/github.com/oklog/run.(*Group).Run.func1+0x26\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/oklog/run/group.go:38\r\n\r\n1 @ 0x42d16a 0x43cfb0 0xaf97e7 0x45ad21\r\n#\t0xaf97e6\tgithub.com/improbable-eng/thanos/vendor/github.com/hashicorp/memberlist.(*Memberlist).streamListen+0x136\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/hashicorp/memberlist/net.go:190\r\n\r\n1 @ 0x42d16a 0x43cfb0 0xafa9ff 0x45ad21\r\n#\t0xafa9fe\tgithub.com/improbable-eng/thanos/vendor/github.com/hashicorp/memberlist.(*Memberlist).packetListen+0x15e\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/hashicorp/memberlist/net.go:270\r\n\r\n1 @ 0x42d16a 0x43cfb0 0xafb4d9 0x45ad21\r\n#\t0xafb4d8\tgithub.com/improbable-eng/thanos/vendor/github.com/hashicorp/memberlist.(*Memberlist).packetHandler+0x108\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/hashicorp/memberlist/net.go:352\r\n\r\n1 @ 0x42d16a 0x43cfb0 0xb05366 0x45ad21\r\n#\t0xb05365\tgithub.com/improbable-eng/thanos/vendor/github.com/hashicorp/memberlist.(*Memberlist).pushPullTrigger+0x1f5\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/hashicorp/memberlist/state.go:155\r\n\r\n1 @ 0x42d16a 0x43cfb0 0xb3f0f5 0x45ad21\r\n#\t0xb3f0f4\tgithub.com/improbable-eng/thanos/pkg/cluster.(*Peer).warnIfAlone+0x114\t/go/src/github.com/improbable-eng/thanos/pkg/cluster/cluster.go:191\r\n\r\n1 @ 0x42d16a 0x43cfb0 0xc03bd8 0xc1dfaa 0x8cd2d7 0x45ad21\r\n#\t0xc03bd7\tmain.interrupt+0x137\t\t\t\t\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/cmd/thanos/main.go:156\r\n#\t0xc1dfa9\tmain.main.func3+0x29\t\t\t\t\t\t\t\t\t/go/src/github.com/improbable-eng/thanos/cmd/thanos/main.go:140\r\n#\t0x8cd2d6\tgithub.com/improbable-eng/thanos/vendor/github.com/oklog/run.(*Group).Run.func1+0x26\t/go/src/github.com/improbable-eng/thanos/vendor/github.com/oklog/run/group.go:38\r\n\r\n1 @ 0x9aede8 0x9aebf0 0x9ab734 0x9b6f78 0x9b74b3 0x6b8e10 0x6b9f6c 0x6b61c1 0x45ad21\r\n#\t0x9aede7\truntime/pprof.writeRuntimeProfile+0x97\t/usr/local/go/src/runtime/pprof/pprof.go:679\r\n#\t0x9aebef\truntime/pprof.writeGoroutine+0x9f\t/usr/local/go/src/runtime/pprof/pprof.go:641\r\n#\t0x9ab733\truntime/pprof.(*Profile).WriteTo+0x3e3\t/usr/local/go/src/runtime/pprof/pprof.go:310\r\n#\t0x9b6f77\tnet/http/pprof.handler.ServeHTTP+0x1b7\t/usr/local/go/src/net/http/pprof/pprof.go:237\r\n#\t0x6b8e0f\tnet/http.(*ServeMux).ServeHTTP+0x12f\t/usr/local/go/src/net/http/server.go:2337\r\n#\t0x6b9f6b\tnet/http.serverHandler.ServeHTTP+0xbb\t/usr/local/go/src/net/http/server.go:2694\r\n#\t0x6b61c0\tnet/http.(*conn).serve+0x650\t\t/usr/local/go/src/net/http/server.go:1830\r\n```\r\n</details>\r\n\r\n<details><summary>Query instance heap profile (in-use; taken on a separate occasion)</summary>\r\n\r\n```\r\n(pprof) top20\r\nShowing nodes accounting for 1528.66MB, 96.91% of 1577.37MB total\r\nDropped 37 nodes (cum <= 7.89MB)\r\nShowing top 20 nodes out of 83\r\n      flat  flat%   sum%        cum   cum%\r\n  319.55MB 20.26% 20.26%   352.55MB 22.35%  github.com/improbable-eng/thanos/vendor/google.golang.org/grpc/transport.(*http2Client).newStream\r\n  228.54MB 14.49% 34.75%   443.57MB 28.12%  github.com/improbable-eng/thanos/vendor/google.golang.org/grpc/transport.(*http2Server).operateHeaders\r\n  211.08MB 13.38% 48.13%   211.08MB 13.38%  runtime.malg\r\n  118.51MB  7.51% 55.64%   118.51MB  7.51%  context.WithValue\r\n  105.51MB  6.69% 62.33%   105.51MB  6.69%  runtime.acquireSudog\r\n  102.01MB  6.47% 68.80%   323.07MB 20.48%  runtime.systemstack\r\n   93.53MB  5.93% 74.73%    93.53MB  5.93%  runtime.mapassign_faststr\r\n   62.51MB  3.96% 78.69%   536.50MB 34.01%  github.com/improbable-eng/thanos/vendor/google.golang.org/grpc.invoke\r\n   61.01MB  3.87% 82.56%    61.01MB  3.87%  runtime.makechan\r\n   29.50MB  1.87% 84.43%    29.50MB  1.87%  github.com/improbable-eng/thanos/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.newServerReporter\r\n      29MB  1.84% 86.27%       29MB  1.84%  runtime.makemap_small\r\n      29MB  1.84% 88.10%       60MB  3.80%  github.com/improbable-eng/thanos/pkg/store.(*ProxyStore).LabelValues\r\n   24.44MB  1.55% 89.65%    24.44MB  1.55%  runtime.makeBucketArray\r\n      24MB  1.52% 91.18%       24MB  1.52%  github.com/improbable-eng/thanos/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.newClientReporter\r\n   23.50MB  1.49% 92.67%    23.50MB  1.49%  context.(*cancelCtx).Done\r\n   18.50MB  1.17% 93.84%    28.06MB  1.78%  context.WithCancel\r\n      16MB  1.01% 94.85%   619.51MB 39.27%  github.com/improbable-eng/thanos/pkg/store/storepb.(*storeClient).LabelValues\r\n   11.50MB  0.73% 95.58%    11.50MB  0.73%  github.com/improbable-eng/thanos/vendor/google.golang.org/grpc/transport.(*recvBuffer).put\r\n      11MB   0.7% 96.28%       11MB   0.7%  github.com/improbable-eng/thanos/vendor/github.com/grpc-ecosystem/go-grpc-middleware/tracing/opentracing.WithTracer\r\n    9.99MB  0.63% 96.91%     9.99MB  0.63%  runtime.allgadd\r\n(pprof)\r\n```\r\n</details>"},"comment":{"url":"https://api.github.com/repos/improbable-eng/thanos/issues/comments/373038147","html_url":"https://github.com/improbable-eng/thanos/issues/232#issuecomment-373038147","issue_url":"https://api.github.com/repos/improbable-eng/thanos/issues/232","id":373038147,"user":{"login":"fabxc","id":4948210,"avatar_url":"https://avatars2.githubusercontent.com/u/4948210?v=4","gravatar_id":"","url":"https://api.github.com/users/fabxc","html_url":"https://github.com/fabxc","followers_url":"https://api.github.com/users/fabxc/followers","following_url":"https://api.github.com/users/fabxc/following{/other_user}","gists_url":"https://api.github.com/users/fabxc/gists{/gist_id}","starred_url":"https://api.github.com/users/fabxc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fabxc/subscriptions","organizations_url":"https://api.github.com/users/fabxc/orgs","repos_url":"https://api.github.com/users/fabxc/repos","events_url":"https://api.github.com/users/fabxc/events{/privacy}","received_events_url":"https://api.github.com/users/fabxc/received_events","type":"User","site_admin":false},"created_at":"2018-03-14T14:21:53Z","updated_at":"2018-03-14T14:21:53Z","author_association":"COLLABORATOR","body":"Mh, is there no more trivial way for an instance just not querying itself to begin with?"}}
{"actor":{"display_login":"fabxc"}}