{"action":"created","issue":{"url":"https://api.github.com/repos/improbable-eng/thanos/issues/928","repository_url":"https://api.github.com/repos/improbable-eng/thanos","labels_url":"https://api.github.com/repos/improbable-eng/thanos/issues/928/labels{/name}","comments_url":"https://api.github.com/repos/improbable-eng/thanos/issues/928/comments","events_url":"https://api.github.com/repos/improbable-eng/thanos/issues/928/events","html_url":"https://github.com/improbable-eng/thanos/pull/928","id":421486806,"node_id":"MDExOlB1bGxSZXF1ZXN0MjYxNTI1Nzk4","number":928,"title":"query: Add store.response-timeout","user":{"login":"povilasv","id":22289110,"node_id":"MDQ6VXNlcjIyMjg5MTEw","avatar_url":"https://avatars1.githubusercontent.com/u/22289110?v=4","gravatar_id":"","url":"https://api.github.com/users/povilasv","html_url":"https://github.com/povilasv","followers_url":"https://api.github.com/users/povilasv/followers","following_url":"https://api.github.com/users/povilasv/following{/other_user}","gists_url":"https://api.github.com/users/povilasv/gists{/gist_id}","starred_url":"https://api.github.com/users/povilasv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/povilasv/subscriptions","organizations_url":"https://api.github.com/users/povilasv/orgs","repos_url":"https://api.github.com/users/povilasv/repos","events_url":"https://api.github.com/users/povilasv/events{/privacy}","received_events_url":"https://api.github.com/users/povilasv/received_events","type":"User","site_admin":false},"labels":[{"id":1272929195,"node_id":"MDU6TGFiZWwxMjcyOTI5MTk1","url":"https://api.github.com/repos/improbable-eng/thanos/labels/component:%20store","name":"component: store","color":"8fad16","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2019-03-15T12:04:18Z","updated_at":"2019-03-22T13:14:26Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/improbable-eng/thanos/pulls/928","html_url":"https://github.com/improbable-eng/thanos/pull/928","diff_url":"https://github.com/improbable-eng/thanos/pull/928.diff","patch_url":"https://github.com/improbable-eng/thanos/pull/928.patch"},"body":"<!--\r\n    Keep PR title verbose enough and add prefix telling\r\n    about what components it touches e.g \"query:\" or \".*:\"\r\n-->\r\n\r\n## Changes\r\n\r\nImproves handling of timed out data/\r\nAdds `--store.response-timeout` in order to have a timeout on a single stream operation.\r\n\r\nSome times GRPC streams get stuck and don't send any data for long periods of time\r\nE.G. Simple GRPC Store API, which just sleeps.\r\n\r\nRight now the only option to time them out in GRPC is to set `context.WithTimeout()`\r\nThe issue is that it is a global timeout,\r\n if we set it to 10s, we won't ever get results from Thanos Store (backed by object store)\r\nif we set it to 120s, we will always be waiting 120s for the HTTP queries to finish, which is :/\r\nAs we will be waiting for the slowest one to finish\r\n\r\nThis fix adds a timeout per single data receive operation\r\n\r\nRef: https://github.com/improbable-eng/thanos/pull/928\r\n\r\nAlso adds stream duration metrics per backend store:\r\n\r\n```\r\nthanos_query_stream_duration_seconds_bucket{store=\"prometheus-system-0.thanos-sidecar.sys-prom:10901\",le=\"0.01\"} 3327.0\r\nthanos_query_stream_duration_seconds_bucket{store=\"prometheus-system-0.thanos-sidecar.sys-prom:10901\",le=\"0.02\"} 3441.0\r\n...\r\nthanos_query_stream_duration_seconds_bucket{store=\"prometheus-system-0.thanos-sidecar.sys-prom:10901\",le=\"20.0\"} 3678.0\r\nthanos_query_stream_duration_seconds_bucket{store=\"prometheus-system-0.thanos-sidecar.sys-prom:10901\",le=\"+Inf\"} 3678.0\r\n```\r\n\r\nAnd a single stream recv duration:\r\n\r\n```\r\nthanos_query_stream_read_duration_seconds_sum{store=\"thanos-rule.sys-mon:10901\"} 3.151294896000005\r\nthanos_query_stream_read_duration_seconds_count{store=\"thanos-rule.sys-mon:10901\"} 3692.0\r\nthanos_query_stream_read_duration_seconds_bucket{store=\"thanos-rule.sys-prom:10901\",le=\"0.005\"} 3685.0\r\nthanos_query_stream_read_duration_seconds_bucket{store=\"thanos-rule.sys-prom:10901\",le=\"0.01\"} 3689.0\r\nthanos_query_stream_read_duration_seconds_bucket{store=\"thanos-rule.sys-prom:10901\",le=\"0.02\"} 3692.0\r\n...\r\nthanos_query_stream_read_duration_seconds_bucket{store=\"thanos-rule.sys-prom:10901\",le=\"5.0\"} 3692.0\r\nthanos_query_stream_read_duration_seconds_bucket{store=\"thanos-rule.sys-prom:10901\",le=\"+Inf\"} 3692.0\r\n```\r\n\r\nThanos Query will log if timeout was reached:\r\n\r\n```\r\nthanos-query-66f5864584-kh6wp thanos-query level=warn ts=2019-03-19T11:12:41.659624416Z caller=proxy.go:373 err=\"failed to receive any data in 500ms from Addr: prometheus-0.thanos-sidecar.sys-mon:10901 Labels: [{cloud_provider aws {} [] 0} {kubernetes_cluster dev-aws {} [] 0} {monitor sys-mon-prometheus {} [] 0} {replica sys-mon-prometheus-0 {} [] 0} {uw_environment dev {} [] 0}] Mint: 1552824000000 Maxt: 9223372036854775807:: context deadline exceeded\" msg=\"returning partial response\"\r\nthanos-query-66f5864584-kh6wp thanos-query level=warn ts=2019-03-19T11:13:50.291321886Z caller=proxy.go:373 err=\"failed to receive any data in 500ms from Addr: thanos-store.telecom:10901 Labels: [] Mint: 1535673600000 Maxt: 1552989600000:: context deadline exceeded\" msg=\"returning partial response\"\r\nthanos-query-66f5864584-kh6wp thanos-query level=warn ts=2019-03-19T11:15:37.430210112Z caller=proxy.go:373 err=\"failed to receive any data in 500ms from Addr: thanos-store.customer-platform:10901 Labels: [] Mint: 1549807838621 Maxt: 1552989600000:: context deadline exceeded\" msg=\"returning partial response\"\r\n```\r\n\r\n## Verification\r\n\r\nTested locally on our cluster, both turned on and turned off (default).\r\n\r\nUnit tests.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/improbable-eng/thanos/issues/comments/475616149","html_url":"https://github.com/improbable-eng/thanos/pull/928#issuecomment-475616149","issue_url":"https://api.github.com/repos/improbable-eng/thanos/issues/928","id":475616149,"node_id":"MDEyOklzc3VlQ29tbWVudDQ3NTYxNjE0OQ==","user":{"login":"povilasv","id":22289110,"node_id":"MDQ6VXNlcjIyMjg5MTEw","avatar_url":"https://avatars1.githubusercontent.com/u/22289110?v=4","gravatar_id":"","url":"https://api.github.com/users/povilasv","html_url":"https://github.com/povilasv","followers_url":"https://api.github.com/users/povilasv/followers","following_url":"https://api.github.com/users/povilasv/following{/other_user}","gists_url":"https://api.github.com/users/povilasv/gists{/gist_id}","starred_url":"https://api.github.com/users/povilasv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/povilasv/subscriptions","organizations_url":"https://api.github.com/users/povilasv/orgs","repos_url":"https://api.github.com/users/povilasv/repos","events_url":"https://api.github.com/users/povilasv/events{/privacy}","received_events_url":"https://api.github.com/users/povilasv/received_events","type":"User","site_admin":false},"created_at":"2019-03-22T13:14:26Z","updated_at":"2019-03-22T13:14:26Z","author_association":"CONTRIBUTOR","body":"> Also I am happy to add this periodic test in CircleCi for this env, can you add Github issue on me for that?\r\n\r\n@bwplotka https://github.com/improbable-eng/thanos/issues/960"}}
{"actor":{"display_login":"povilasv"}}