{"action":"created","comment":{"url":"https://api.github.com/repos/thanos-io/thanos/pulls/comments/351333635","pull_request_review_id":323734436,"id":351333635,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDM1MTMzMzYzNQ==","diff_hunk":"@@ -344,11 +398,55 @@ func startStreamSeriesSet(\n \n \twg.Add(1)\n \tgo func() {\n+\t\tisFirstRecv := true\n \t\tdefer wg.Done()\n \t\tdefer close(s.recvCh)\n-\n \t\tfor {\n-\t\t\tr, err := s.stream.Recv()\n+\t\t\tvar cancel context.CancelFunc\n+\t\t\tif s.responseTimeout != 0 {","path":"pkg/store/proxy.go","position":141,"original_position":141,"commit_id":"79774094944d9bf3cad9d2fca6b716e054644068","original_commit_id":"a97de522165a41403a3583b16f34b2e6ac76af89","user":{"login":"IKSIN","id":2960110,"node_id":"MDQ6VXNlcjI5NjAxMTA=","avatar_url":"https://avatars3.githubusercontent.com/u/2960110?v=4","gravatar_id":"","url":"https://api.github.com/users/IKSIN","html_url":"https://github.com/IKSIN","followers_url":"https://api.github.com/users/IKSIN/followers","following_url":"https://api.github.com/users/IKSIN/following{/other_user}","gists_url":"https://api.github.com/users/IKSIN/gists{/gist_id}","starred_url":"https://api.github.com/users/IKSIN/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/IKSIN/subscriptions","organizations_url":"https://api.github.com/users/IKSIN/orgs","repos_url":"https://api.github.com/users/IKSIN/repos","events_url":"https://api.github.com/users/IKSIN/events{/privacy}","received_events_url":"https://api.github.com/users/IKSIN/received_events","type":"User","site_admin":false},"body":"If we want to use timeout for a whole response:\r\n1. We must to set such timeout ~query response. in this case we lose opportunity to fast cancel response from slow store (for example)\r\n2. We need to increase channel buffer (for real parallel read from stores)","created_at":"2019-11-27T14:56:15Z","updated_at":"2019-11-27T14:56:15Z","html_url":"https://github.com/thanos-io/thanos/pull/1789#discussion_r351333635","pull_request_url":"https://api.github.com/repos/thanos-io/thanos/pulls/1789","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/thanos-io/thanos/pulls/comments/351333635"},"html":{"href":"https://github.com/thanos-io/thanos/pull/1789#discussion_r351333635"},"pull_request":{"href":"https://api.github.com/repos/thanos-io/thanos/pulls/1789"}},"in_reply_to_id":351241819},"pull_request":{"url":"https://api.github.com/repos/thanos-io/thanos/pulls/1789","id":345666274,"node_id":"MDExOlB1bGxSZXF1ZXN0MzQ1NjY2Mjc0","html_url":"https://github.com/thanos-io/thanos/pull/1789","diff_url":"https://github.com/thanos-io/thanos/pull/1789.diff","patch_url":"https://github.com/thanos-io/thanos/pull/1789.patch","issue_url":"https://api.github.com/repos/thanos-io/thanos/issues/1789","number":1789,"state":"open","locked":false,"title":"Improve store timeouts","user":{"login":"IKSIN","id":2960110,"node_id":"MDQ6VXNlcjI5NjAxMTA=","avatar_url":"https://avatars3.githubusercontent.com/u/2960110?v=4","gravatar_id":"","url":"https://api.github.com/users/IKSIN","html_url":"https://github.com/IKSIN","followers_url":"https://api.github.com/users/IKSIN/followers","following_url":"https://api.github.com/users/IKSIN/following{/other_user}","gists_url":"https://api.github.com/users/IKSIN/gists{/gist_id}","starred_url":"https://api.github.com/users/IKSIN/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/IKSIN/subscriptions","organizations_url":"https://api.github.com/users/IKSIN/orgs","repos_url":"https://api.github.com/users/IKSIN/repos","events_url":"https://api.github.com/users/IKSIN/events{/privacy}","received_events_url":"https://api.github.com/users/IKSIN/received_events","type":"User","site_admin":false},"body":"<!--\r\n    Keep PR title verbose enough and add prefix telling\r\n    about what components it touches e.g \"query:\" or \".*:\"\r\n-->\r\n\r\n<!-- \r\n    Don't forget about CHANGELOG! \r\n    \r\n    Changelog entry format:\r\n    - [#<PR-id>](<PR-URL>) Thanos <Component> ...\r\n    \r\n    <PR-id> Id of your pull request.\r\n    <PR-URL> URL of your PR such as https://github.com/thanos-io/thanos/pull/<PR-id>\r\n    <Component> Component affected by your changes such as Query, Store, Receive.\r\n-->\r\n\r\nWe want to increase stability Thanos infrastructure with many stores/sidecars (49 in our case) when some stores respond slowly.\r\n\r\n## Changes\r\n\r\n* [ ] Fix and improve TestProxyStore_SeriesSlowStores:\r\n  * [ ] Fixed broken test\r\n  * [ ] Add some use-cases\r\n  * [ ]  check on elapsed time on get data from stores.\r\n* [ ] Change logic in ProxyStore:\r\n  * [ ] store response timeout moved from MergedSet.Next() function to goroutine in startStreamSeriesSet to check timeouts in parallel\r\n  * [ ] add ProxyStore metrics\r\n\r\nMore details in comments.\r\n\r\n## Verification\r\n\r\n* [ ] Tests\r\n* [ ] On production in our system with 49 store backends already 2 weeks.\r\n","created_at":"2019-11-26T10:52:47Z","updated_at":"2019-11-27T14:56:15Z","closed_at":null,"merged_at":null,"merge_commit_sha":"414a95adcf73339947f8ce0674ecbee0f7bc4758","assignee":null,"assignees":[],"requested_reviewers":[{"login":"GiedriusS","id":2377233,"node_id":"MDQ6VXNlcjIzNzcyMzM=","avatar_url":"https://avatars2.githubusercontent.com/u/2377233?v=4","gravatar_id":"","url":"https://api.github.com/users/GiedriusS","html_url":"https://github.com/GiedriusS","followers_url":"https://api.github.com/users/GiedriusS/followers","following_url":"https://api.github.com/users/GiedriusS/following{/other_user}","gists_url":"https://api.github.com/users/GiedriusS/gists{/gist_id}","starred_url":"https://api.github.com/users/GiedriusS/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GiedriusS/subscriptions","organizations_url":"https://api.github.com/users/GiedriusS/orgs","repos_url":"https://api.github.com/users/GiedriusS/repos","events_url":"https://api.github.com/users/GiedriusS/events{/privacy}","received_events_url":"https://api.github.com/users/GiedriusS/received_events","type":"User","site_admin":false},{"login":"povilasv","id":22289110,"node_id":"MDQ6VXNlcjIyMjg5MTEw","avatar_url":"https://avatars1.githubusercontent.com/u/22289110?v=4","gravatar_id":"","url":"https://api.github.com/users/povilasv","html_url":"https://github.com/povilasv","followers_url":"https://api.github.com/users/povilasv/followers","following_url":"https://api.github.com/users/povilasv/following{/other_user}","gists_url":"https://api.github.com/users/povilasv/gists{/gist_id}","starred_url":"https://api.github.com/users/povilasv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/povilasv/subscriptions","organizations_url":"https://api.github.com/users/povilasv/orgs","repos_url":"https://api.github.com/users/povilasv/repos","events_url":"https://api.github.com/users/povilasv/events{/privacy}","received_events_url":"https://api.github.com/users/povilasv/received_events","type":"User","site_admin":false}],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/thanos-io/thanos/pulls/1789/commits","review_comments_url":"https://api.github.com/repos/thanos-io/thanos/pulls/1789/comments","review_comment_url":"https://api.github.com/repos/thanos-io/thanos/pulls/comments{/number}","comments_url":"https://api.github.com/repos/thanos-io/thanos/issues/1789/comments","statuses_url":"https://api.github.com/repos/thanos-io/thanos/statuses/79774094944d9bf3cad9d2fca6b716e054644068","head":{"label":"monitoring-tools:slow_tests","ref":"slow_tests","sha":"79774094944d9bf3cad9d2fca6b716e054644068","user":{"login":"monitoring-tools","id":26260389,"node_id":"MDEyOk9yZ2FuaXphdGlvbjI2MjYwMzg5","avatar_url":"https://avatars2.githubusercontent.com/u/26260389?v=4","gravatar_id":"","url":"https://api.github.com/users/monitoring-tools","html_url":"https://github.com/monitoring-tools","followers_url":"https://api.github.com/users/monitoring-tools/followers","following_url":"https://api.github.com/users/monitoring-tools/following{/other_user}","gists_url":"https://api.github.com/users/monitoring-tools/gists{/gist_id}","starred_url":"https://api.github.com/users/monitoring-tools/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/monitoring-tools/subscriptions","organizations_url":"https://api.github.com/users/monitoring-tools/orgs","repos_url":"https://api.github.com/users/monitoring-tools/repos","events_url":"https://api.github.com/users/monitoring-tools/events{/privacy}","received_events_url":"https://api.github.com/users/monitoring-tools/received_events","type":"Organization","site_admin":false},"repo":{"id":220025090,"node_id":"MDEwOlJlcG9zaXRvcnkyMjAwMjUwOTA=","name":"thanos","full_name":"monitoring-tools/thanos","private":false,"owner":{"login":"monitoring-tools","id":26260389,"node_id":"MDEyOk9yZ2FuaXphdGlvbjI2MjYwMzg5","avatar_url":"https://avatars2.githubusercontent.com/u/26260389?v=4","gravatar_id":"","url":"https://api.github.com/users/monitoring-tools","html_url":"https://github.com/monitoring-tools","followers_url":"https://api.github.com/users/monitoring-tools/followers","following_url":"https://api.github.com/users/monitoring-tools/following{/other_user}","gists_url":"https://api.github.com/users/monitoring-tools/gists{/gist_id}","starred_url":"https://api.github.com/users/monitoring-tools/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/monitoring-tools/subscriptions","organizations_url":"https://api.github.com/users/monitoring-tools/orgs","repos_url":"https://api.github.com/users/monitoring-tools/repos","events_url":"https://api.github.com/users/monitoring-tools/events{/privacy}","received_events_url":"https://api.github.com/users/monitoring-tools/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/monitoring-tools/thanos","description":"Highly available Prometheus setup with long term storage capabilities. CNCF Sandbox project.","fork":true,"url":"https://api.github.com/repos/monitoring-tools/thanos","forks_url":"https://api.github.com/repos/monitoring-tools/thanos/forks","keys_url":"https://api.github.com/repos/monitoring-tools/thanos/keys{/key_id}","collaborators_url":"https://api.github.com/repos/monitoring-tools/thanos/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/monitoring-tools/thanos/teams","hooks_url":"https://api.github.com/repos/monitoring-tools/thanos/hooks","issue_events_url":"https://api.github.com/repos/monitoring-tools/thanos/issues/events{/number}","events_url":"https://api.github.com/repos/monitoring-tools/thanos/events","assignees_url":"https://api.github.com/repos/monitoring-tools/thanos/assignees{/user}","branches_url":"https://api.github.com/repos/monitoring-tools/thanos/branches{/branch}","tags_url":"https://api.github.com/repos/monitoring-tools/thanos/tags","blobs_url":"https://api.github.com/repos/monitoring-tools/thanos/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/monitoring-tools/thanos/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/monitoring-tools/thanos/git/refs{/sha}","trees_url":"https://api.github.com/repos/monitoring-tools/thanos/git/trees{/sha}","statuses_url":"https://api.github.com/repos/monitoring-tools/thanos/statuses/{sha}","languages_url":"https://api.github.com/repos/monitoring-tools/thanos/languages","stargazers_url":"https://api.github.com/repos/monitoring-tools/thanos/stargazers","contributors_url":"https://api.github.com/repos/monitoring-tools/thanos/contributors","subscribers_url":"https://api.github.com/repos/monitoring-tools/thanos/subscribers","subscription_url":"https://api.github.com/repos/monitoring-tools/thanos/subscription","commits_url":"https://api.github.com/repos/monitoring-tools/thanos/commits{/sha}","git_commits_url":"https://api.github.com/repos/monitoring-tools/thanos/git/commits{/sha}","comments_url":"https://api.github.com/repos/monitoring-tools/thanos/comments{/number}","issue_comment_url":"https://api.github.com/repos/monitoring-tools/thanos/issues/comments{/number}","contents_url":"https://api.github.com/repos/monitoring-tools/thanos/contents/{+path}","compare_url":"https://api.github.com/repos/monitoring-tools/thanos/compare/{base}...{head}","merges_url":"https://api.github.com/repos/monitoring-tools/thanos/merges","archive_url":"https://api.github.com/repos/monitoring-tools/thanos/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/monitoring-tools/thanos/downloads","issues_url":"https://api.github.com/repos/monitoring-tools/thanos/issues{/number}","pulls_url":"https://api.github.com/repos/monitoring-tools/thanos/pulls{/number}","milestones_url":"https://api.github.com/repos/monitoring-tools/thanos/milestones{/number}","notifications_url":"https://api.github.com/repos/monitoring-tools/thanos/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/monitoring-tools/thanos/labels{/name}","releases_url":"https://api.github.com/repos/monitoring-tools/thanos/releases{/id}","deployments_url":"https://api.github.com/repos/monitoring-tools/thanos/deployments","created_at":"2019-11-06T15:10:54Z","updated_at":"2019-11-22T12:26:42Z","pushed_at":"2019-11-27T14:24:38Z","git_url":"git://github.com/monitoring-tools/thanos.git","ssh_url":"git@github.com:monitoring-tools/thanos.git","clone_url":"https://github.com/monitoring-tools/thanos.git","svn_url":"https://github.com/monitoring-tools/thanos","homepage":"https://thanos.io","size":63263,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"disabled":false,"open_issues_count":1,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"base":{"label":"thanos-io:master","ref":"master","sha":"4e06e2e6c2effa9d86b0e80d1b431e74d5276469","user":{"login":"thanos-io","id":49725059,"node_id":"MDEyOk9yZ2FuaXphdGlvbjQ5NzI1MDU5","avatar_url":"https://avatars3.githubusercontent.com/u/49725059?v=4","gravatar_id":"","url":"https://api.github.com/users/thanos-io","html_url":"https://github.com/thanos-io","followers_url":"https://api.github.com/users/thanos-io/followers","following_url":"https://api.github.com/users/thanos-io/following{/other_user}","gists_url":"https://api.github.com/users/thanos-io/gists{/gist_id}","starred_url":"https://api.github.com/users/thanos-io/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thanos-io/subscriptions","organizations_url":"https://api.github.com/users/thanos-io/orgs","repos_url":"https://api.github.com/users/thanos-io/repos","events_url":"https://api.github.com/users/thanos-io/events{/privacy}","received_events_url":"https://api.github.com/users/thanos-io/received_events","type":"Organization","site_admin":false},"repo":{"id":109162639,"node_id":"MDEwOlJlcG9zaXRvcnkxMDkxNjI2Mzk=","name":"thanos","full_name":"thanos-io/thanos","private":false,"owner":{"login":"thanos-io","id":49725059,"node_id":"MDEyOk9yZ2FuaXphdGlvbjQ5NzI1MDU5","avatar_url":"https://avatars3.githubusercontent.com/u/49725059?v=4","gravatar_id":"","url":"https://api.github.com/users/thanos-io","html_url":"https://github.com/thanos-io","followers_url":"https://api.github.com/users/thanos-io/followers","following_url":"https://api.github.com/users/thanos-io/following{/other_user}","gists_url":"https://api.github.com/users/thanos-io/gists{/gist_id}","starred_url":"https://api.github.com/users/thanos-io/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thanos-io/subscriptions","organizations_url":"https://api.github.com/users/thanos-io/orgs","repos_url":"https://api.github.com/users/thanos-io/repos","events_url":"https://api.github.com/users/thanos-io/events{/privacy}","received_events_url":"https://api.github.com/users/thanos-io/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/thanos-io/thanos","description":"Highly available Prometheus setup with long term storage capabilities. CNCF Sandbox project.","fork":false,"url":"https://api.github.com/repos/thanos-io/thanos","forks_url":"https://api.github.com/repos/thanos-io/thanos/forks","keys_url":"https://api.github.com/repos/thanos-io/thanos/keys{/key_id}","collaborators_url":"https://api.github.com/repos/thanos-io/thanos/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/thanos-io/thanos/teams","hooks_url":"https://api.github.com/repos/thanos-io/thanos/hooks","issue_events_url":"https://api.github.com/repos/thanos-io/thanos/issues/events{/number}","events_url":"https://api.github.com/repos/thanos-io/thanos/events","assignees_url":"https://api.github.com/repos/thanos-io/thanos/assignees{/user}","branches_url":"https://api.github.com/repos/thanos-io/thanos/branches{/branch}","tags_url":"https://api.github.com/repos/thanos-io/thanos/tags","blobs_url":"https://api.github.com/repos/thanos-io/thanos/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/thanos-io/thanos/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/thanos-io/thanos/git/refs{/sha}","trees_url":"https://api.github.com/repos/thanos-io/thanos/git/trees{/sha}","statuses_url":"https://api.github.com/repos/thanos-io/thanos/statuses/{sha}","languages_url":"https://api.github.com/repos/thanos-io/thanos/languages","stargazers_url":"https://api.github.com/repos/thanos-io/thanos/stargazers","contributors_url":"https://api.github.com/repos/thanos-io/thanos/contributors","subscribers_url":"https://api.github.com/repos/thanos-io/thanos/subscribers","subscription_url":"https://api.github.com/repos/thanos-io/thanos/subscription","commits_url":"https://api.github.com/repos/thanos-io/thanos/commits{/sha}","git_commits_url":"https://api.github.com/repos/thanos-io/thanos/git/commits{/sha}","comments_url":"https://api.github.com/repos/thanos-io/thanos/comments{/number}","issue_comment_url":"https://api.github.com/repos/thanos-io/thanos/issues/comments{/number}","contents_url":"https://api.github.com/repos/thanos-io/thanos/contents/{+path}","compare_url":"https://api.github.com/repos/thanos-io/thanos/compare/{base}...{head}","merges_url":"https://api.github.com/repos/thanos-io/thanos/merges","archive_url":"https://api.github.com/repos/thanos-io/thanos/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/thanos-io/thanos/downloads","issues_url":"https://api.github.com/repos/thanos-io/thanos/issues{/number}","pulls_url":"https://api.github.com/repos/thanos-io/thanos/pulls{/number}","milestones_url":"https://api.github.com/repos/thanos-io/thanos/milestones{/number}","notifications_url":"https://api.github.com/repos/thanos-io/thanos/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/thanos-io/thanos/labels{/name}","releases_url":"https://api.github.com/repos/thanos-io/thanos/releases{/id}","deployments_url":"https://api.github.com/repos/thanos-io/thanos/deployments","created_at":"2017-11-01T17:38:03Z","updated_at":"2019-11-27T10:12:32Z","pushed_at":"2019-11-27T14:27:30Z","git_url":"git://github.com/thanos-io/thanos.git","ssh_url":"git@github.com:thanos-io/thanos.git","clone_url":"https://github.com/thanos-io/thanos.git","svn_url":"https://github.com/thanos-io/thanos","homepage":"https://thanos.io","size":63257,"stargazers_count":4714,"watchers_count":4714,"language":"Go","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":583,"mirror_url":null,"archived":false,"disabled":false,"open_issues_count":255,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":583,"open_issues":255,"watchers":4714,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/thanos-io/thanos/pulls/1789"},"html":{"href":"https://github.com/thanos-io/thanos/pull/1789"},"issue":{"href":"https://api.github.com/repos/thanos-io/thanos/issues/1789"},"comments":{"href":"https://api.github.com/repos/thanos-io/thanos/issues/1789/comments"},"review_comments":{"href":"https://api.github.com/repos/thanos-io/thanos/pulls/1789/comments"},"review_comment":{"href":"https://api.github.com/repos/thanos-io/thanos/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/thanos-io/thanos/pulls/1789/commits"},"statuses":{"href":"https://api.github.com/repos/thanos-io/thanos/statuses/79774094944d9bf3cad9d2fca6b716e054644068"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"IKSIN"}}