{"action":"opened","number":17539,"pull_request":{"url":"https://api.github.com/repos/grpc/grpc/pulls/17539","id":239419889,"node_id":"MDExOlB1bGxSZXF1ZXN0MjM5NDE5ODg5","html_url":"https://github.com/grpc/grpc/pull/17539","diff_url":"https://github.com/grpc/grpc/pull/17539.diff","patch_url":"https://github.com/grpc/grpc/pull/17539.patch","issue_url":"https://api.github.com/repos/grpc/grpc/issues/17539","number":17539,"state":"open","locked":false,"title":"Increase json run localhost timeout under tsan, preparation for enabling c-ares","user":{"login":"apolcyn","id":9566254,"node_id":"MDQ6VXNlcjk1NjYyNTQ=","avatar_url":"https://avatars1.githubusercontent.com/u/9566254?v=4","gravatar_id":"","url":"https://api.github.com/users/apolcyn","html_url":"https://github.com/apolcyn","followers_url":"https://api.github.com/users/apolcyn/followers","following_url":"https://api.github.com/users/apolcyn/following{/other_user}","gists_url":"https://api.github.com/users/apolcyn/gists{/gist_id}","starred_url":"https://api.github.com/users/apolcyn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apolcyn/subscriptions","organizations_url":"https://api.github.com/users/apolcyn/orgs","repos_url":"https://api.github.com/users/apolcyn/repos","events_url":"https://api.github.com/users/apolcyn/events{/privacy}","received_events_url":"https://api.github.com/users/apolcyn/received_events","type":"User","site_admin":false},"body":"The 600 channel scenario under TSAN fails almost every time under c-ares (when ran under the bazel RBE TSAN suite). This is because QPS workers hit an [assertion failure](https://github.com/grpc/grpc/blob/master/test/cpp/qps/client.h#L506) because they fail to connect all channels to servers within the 10 second deadline in this configuration about every time.\r\n\r\nAs observable with logging, in this scenario/config, channels can sometimes spend huge amounts of time in [subchannel_index_register](https://github.com/grpc/grpc/blob/master/src/core/ext/filters/client_channel/subchannel_index.cc#L160). I've seen the slowest-connecting channels spend just over a minute within a single call to `subchannel_index_register`, whie taking ~200 [passes](https://github.com/grpc/grpc/blob/master/src/core/ext/filters/client_channel/subchannel_index.cc#L192) through the while loop before finishing. However, I think this slowness makes sense here, and so increasing the test timeout is the right fix:\r\n  a) all 600 channels have [different args](https://github.com/grpc/grpc/blob/master/test/cpp/qps/client.h#L458), so there's no subchannel sharing\r\n  b) Bazel RBE tests don't run on a lot of cores\r\n  c) slow TSAN\r\n  d) There's more room for concurrency with subchannel creation when c-ares is used than with the native resolver. When the native resolver is used, the number of concurrent DNS resolutions (and subsequent subchannel creations), is constrained by the number of resolver executor threads, but c-ares can run on all threads. Thus there are more chances for races during `subchannel_register_index` - e.g., [this PR](https://github.com/grpc/grpc/pull/17529) is an alternative \"fix\" for this issue.","created_at":"2018-12-18T09:36:47Z","updated_at":"2018-12-18T09:36:47Z","closed_at":null,"merged_at":null,"merge_commit_sha":null,"assignee":{"login":"jtattermusch","id":9939684,"node_id":"MDQ6VXNlcjk5Mzk2ODQ=","avatar_url":"https://avatars1.githubusercontent.com/u/9939684?v=4","gravatar_id":"","url":"https://api.github.com/users/jtattermusch","html_url":"https://github.com/jtattermusch","followers_url":"https://api.github.com/users/jtattermusch/followers","following_url":"https://api.github.com/users/jtattermusch/following{/other_user}","gists_url":"https://api.github.com/users/jtattermusch/gists{/gist_id}","starred_url":"https://api.github.com/users/jtattermusch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jtattermusch/subscriptions","organizations_url":"https://api.github.com/users/jtattermusch/orgs","repos_url":"https://api.github.com/users/jtattermusch/repos","events_url":"https://api.github.com/users/jtattermusch/events{/privacy}","received_events_url":"https://api.github.com/users/jtattermusch/received_events","type":"User","site_admin":false},"assignees":[{"login":"jtattermusch","id":9939684,"node_id":"MDQ6VXNlcjk5Mzk2ODQ=","avatar_url":"https://avatars1.githubusercontent.com/u/9939684?v=4","gravatar_id":"","url":"https://api.github.com/users/jtattermusch","html_url":"https://github.com/jtattermusch","followers_url":"https://api.github.com/users/jtattermusch/followers","following_url":"https://api.github.com/users/jtattermusch/following{/other_user}","gists_url":"https://api.github.com/users/jtattermusch/gists{/gist_id}","starred_url":"https://api.github.com/users/jtattermusch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jtattermusch/subscriptions","organizations_url":"https://api.github.com/users/jtattermusch/orgs","repos_url":"https://api.github.com/users/jtattermusch/repos","events_url":"https://api.github.com/users/jtattermusch/events{/privacy}","received_events_url":"https://api.github.com/users/jtattermusch/received_events","type":"User","site_admin":false}],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/grpc/grpc/pulls/17539/commits","review_comments_url":"https://api.github.com/repos/grpc/grpc/pulls/17539/comments","review_comment_url":"https://api.github.com/repos/grpc/grpc/pulls/comments{/number}","comments_url":"https://api.github.com/repos/grpc/grpc/issues/17539/comments","statuses_url":"https://api.github.com/repos/grpc/grpc/statuses/1e58ccf1a6dcaac30c5b7650bb5ebd99c68e608a","head":{"label":"apolcyn:json_run_localhost_under_tsan","ref":"json_run_localhost_under_tsan","sha":"1e58ccf1a6dcaac30c5b7650bb5ebd99c68e608a","user":{"login":"apolcyn","id":9566254,"node_id":"MDQ6VXNlcjk1NjYyNTQ=","avatar_url":"https://avatars1.githubusercontent.com/u/9566254?v=4","gravatar_id":"","url":"https://api.github.com/users/apolcyn","html_url":"https://github.com/apolcyn","followers_url":"https://api.github.com/users/apolcyn/followers","following_url":"https://api.github.com/users/apolcyn/following{/other_user}","gists_url":"https://api.github.com/users/apolcyn/gists{/gist_id}","starred_url":"https://api.github.com/users/apolcyn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apolcyn/subscriptions","organizations_url":"https://api.github.com/users/apolcyn/orgs","repos_url":"https://api.github.com/users/apolcyn/repos","events_url":"https://api.github.com/users/apolcyn/events{/privacy}","received_events_url":"https://api.github.com/users/apolcyn/received_events","type":"User","site_admin":false},"repo":{"id":61862132,"node_id":"MDEwOlJlcG9zaXRvcnk2MTg2MjEzMg==","name":"grpc","full_name":"apolcyn/grpc","private":false,"owner":{"login":"apolcyn","id":9566254,"node_id":"MDQ6VXNlcjk1NjYyNTQ=","avatar_url":"https://avatars1.githubusercontent.com/u/9566254?v=4","gravatar_id":"","url":"https://api.github.com/users/apolcyn","html_url":"https://github.com/apolcyn","followers_url":"https://api.github.com/users/apolcyn/followers","following_url":"https://api.github.com/users/apolcyn/following{/other_user}","gists_url":"https://api.github.com/users/apolcyn/gists{/gist_id}","starred_url":"https://api.github.com/users/apolcyn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apolcyn/subscriptions","organizations_url":"https://api.github.com/users/apolcyn/orgs","repos_url":"https://api.github.com/users/apolcyn/repos","events_url":"https://api.github.com/users/apolcyn/events{/privacy}","received_events_url":"https://api.github.com/users/apolcyn/received_events","type":"User","site_admin":false},"html_url":"https://github.com/apolcyn/grpc","description":"The C based gRPC (C++, Node.js, Python, Ruby, Objective-C, PHP, C#)","fork":true,"url":"https://api.github.com/repos/apolcyn/grpc","forks_url":"https://api.github.com/repos/apolcyn/grpc/forks","keys_url":"https://api.github.com/repos/apolcyn/grpc/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apolcyn/grpc/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apolcyn/grpc/teams","hooks_url":"https://api.github.com/repos/apolcyn/grpc/hooks","issue_events_url":"https://api.github.com/repos/apolcyn/grpc/issues/events{/number}","events_url":"https://api.github.com/repos/apolcyn/grpc/events","assignees_url":"https://api.github.com/repos/apolcyn/grpc/assignees{/user}","branches_url":"https://api.github.com/repos/apolcyn/grpc/branches{/branch}","tags_url":"https://api.github.com/repos/apolcyn/grpc/tags","blobs_url":"https://api.github.com/repos/apolcyn/grpc/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apolcyn/grpc/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apolcyn/grpc/git/refs{/sha}","trees_url":"https://api.github.com/repos/apolcyn/grpc/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apolcyn/grpc/statuses/{sha}","languages_url":"https://api.github.com/repos/apolcyn/grpc/languages","stargazers_url":"https://api.github.com/repos/apolcyn/grpc/stargazers","contributors_url":"https://api.github.com/repos/apolcyn/grpc/contributors","subscribers_url":"https://api.github.com/repos/apolcyn/grpc/subscribers","subscription_url":"https://api.github.com/repos/apolcyn/grpc/subscription","commits_url":"https://api.github.com/repos/apolcyn/grpc/commits{/sha}","git_commits_url":"https://api.github.com/repos/apolcyn/grpc/git/commits{/sha}","comments_url":"https://api.github.com/repos/apolcyn/grpc/comments{/number}","issue_comment_url":"https://api.github.com/repos/apolcyn/grpc/issues/comments{/number}","contents_url":"https://api.github.com/repos/apolcyn/grpc/contents/{+path}","compare_url":"https://api.github.com/repos/apolcyn/grpc/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apolcyn/grpc/merges","archive_url":"https://api.github.com/repos/apolcyn/grpc/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apolcyn/grpc/downloads","issues_url":"https://api.github.com/repos/apolcyn/grpc/issues{/number}","pulls_url":"https://api.github.com/repos/apolcyn/grpc/pulls{/number}","milestones_url":"https://api.github.com/repos/apolcyn/grpc/milestones{/number}","notifications_url":"https://api.github.com/repos/apolcyn/grpc/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apolcyn/grpc/labels{/name}","releases_url":"https://api.github.com/repos/apolcyn/grpc/releases{/id}","deployments_url":"https://api.github.com/repos/apolcyn/grpc/deployments","created_at":"2016-06-24T06:32:25Z","updated_at":"2016-06-24T06:32:34Z","pushed_at":"2018-12-18T08:56:26Z","git_url":"git://github.com/apolcyn/grpc.git","ssh_url":"git@github.com:apolcyn/grpc.git","clone_url":"https://github.com/apolcyn/grpc.git","svn_url":"https://github.com/apolcyn/grpc","homepage":"","size":374714,"stargazers_count":0,"watchers_count":0,"language":"C","has_issues":false,"has_projects":true,"has_downloads":false,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"other","name":"Other","spdx_id":"NOASSERTION","url":null,"node_id":"MDc6TGljZW5zZTA="},"forks":1,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"grpc:master","ref":"master","sha":"70a4915fa16ae6ef994f258b2c4781269b0317ea","user":{"login":"grpc","id":7802525,"node_id":"MDEyOk9yZ2FuaXphdGlvbjc4MDI1MjU=","avatar_url":"https://avatars1.githubusercontent.com/u/7802525?v=4","gravatar_id":"","url":"https://api.github.com/users/grpc","html_url":"https://github.com/grpc","followers_url":"https://api.github.com/users/grpc/followers","following_url":"https://api.github.com/users/grpc/following{/other_user}","gists_url":"https://api.github.com/users/grpc/gists{/gist_id}","starred_url":"https://api.github.com/users/grpc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/grpc/subscriptions","organizations_url":"https://api.github.com/users/grpc/orgs","repos_url":"https://api.github.com/users/grpc/repos","events_url":"https://api.github.com/users/grpc/events{/privacy}","received_events_url":"https://api.github.com/users/grpc/received_events","type":"Organization","site_admin":false},"repo":{"id":27729880,"node_id":"MDEwOlJlcG9zaXRvcnkyNzcyOTg4MA==","name":"grpc","full_name":"grpc/grpc","private":false,"owner":{"login":"grpc","id":7802525,"node_id":"MDEyOk9yZ2FuaXphdGlvbjc4MDI1MjU=","avatar_url":"https://avatars1.githubusercontent.com/u/7802525?v=4","gravatar_id":"","url":"https://api.github.com/users/grpc","html_url":"https://github.com/grpc","followers_url":"https://api.github.com/users/grpc/followers","following_url":"https://api.github.com/users/grpc/following{/other_user}","gists_url":"https://api.github.com/users/grpc/gists{/gist_id}","starred_url":"https://api.github.com/users/grpc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/grpc/subscriptions","organizations_url":"https://api.github.com/users/grpc/orgs","repos_url":"https://api.github.com/users/grpc/repos","events_url":"https://api.github.com/users/grpc/events{/privacy}","received_events_url":"https://api.github.com/users/grpc/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/grpc/grpc","description":"The C based gRPC (C++, Python, Ruby, Objective-C, PHP, C#)","fork":false,"url":"https://api.github.com/repos/grpc/grpc","forks_url":"https://api.github.com/repos/grpc/grpc/forks","keys_url":"https://api.github.com/repos/grpc/grpc/keys{/key_id}","collaborators_url":"https://api.github.com/repos/grpc/grpc/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/grpc/grpc/teams","hooks_url":"https://api.github.com/repos/grpc/grpc/hooks","issue_events_url":"https://api.github.com/repos/grpc/grpc/issues/events{/number}","events_url":"https://api.github.com/repos/grpc/grpc/events","assignees_url":"https://api.github.com/repos/grpc/grpc/assignees{/user}","branches_url":"https://api.github.com/repos/grpc/grpc/branches{/branch}","tags_url":"https://api.github.com/repos/grpc/grpc/tags","blobs_url":"https://api.github.com/repos/grpc/grpc/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/grpc/grpc/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/grpc/grpc/git/refs{/sha}","trees_url":"https://api.github.com/repos/grpc/grpc/git/trees{/sha}","statuses_url":"https://api.github.com/repos/grpc/grpc/statuses/{sha}","languages_url":"https://api.github.com/repos/grpc/grpc/languages","stargazers_url":"https://api.github.com/repos/grpc/grpc/stargazers","contributors_url":"https://api.github.com/repos/grpc/grpc/contributors","subscribers_url":"https://api.github.com/repos/grpc/grpc/subscribers","subscription_url":"https://api.github.com/repos/grpc/grpc/subscription","commits_url":"https://api.github.com/repos/grpc/grpc/commits{/sha}","git_commits_url":"https://api.github.com/repos/grpc/grpc/git/commits{/sha}","comments_url":"https://api.github.com/repos/grpc/grpc/comments{/number}","issue_comment_url":"https://api.github.com/repos/grpc/grpc/issues/comments{/number}","contents_url":"https://api.github.com/repos/grpc/grpc/contents/{+path}","compare_url":"https://api.github.com/repos/grpc/grpc/compare/{base}...{head}","merges_url":"https://api.github.com/repos/grpc/grpc/merges","archive_url":"https://api.github.com/repos/grpc/grpc/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/grpc/grpc/downloads","issues_url":"https://api.github.com/repos/grpc/grpc/issues{/number}","pulls_url":"https://api.github.com/repos/grpc/grpc/pulls{/number}","milestones_url":"https://api.github.com/repos/grpc/grpc/milestones{/number}","notifications_url":"https://api.github.com/repos/grpc/grpc/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/grpc/grpc/labels{/name}","releases_url":"https://api.github.com/repos/grpc/grpc/releases{/id}","deployments_url":"https://api.github.com/repos/grpc/grpc/deployments","created_at":"2014-12-08T18:58:53Z","updated_at":"2018-12-18T09:17:16Z","pushed_at":"2018-12-18T08:56:28Z","git_url":"git://github.com/grpc/grpc.git","ssh_url":"git@github.com:grpc/grpc.git","clone_url":"https://github.com/grpc/grpc.git","svn_url":"https://github.com/grpc/grpc","homepage":"https://grpc.io","size":202632,"stargazers_count":18552,"watchers_count":18552,"language":"C++","has_issues":true,"has_projects":true,"has_downloads":false,"has_wiki":true,"has_pages":true,"forks_count":4289,"mirror_url":null,"archived":false,"open_issues_count":1143,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":4289,"open_issues":1143,"watchers":18552,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/grpc/grpc/pulls/17539"},"html":{"href":"https://github.com/grpc/grpc/pull/17539"},"issue":{"href":"https://api.github.com/repos/grpc/grpc/issues/17539"},"comments":{"href":"https://api.github.com/repos/grpc/grpc/issues/17539/comments"},"review_comments":{"href":"https://api.github.com/repos/grpc/grpc/pulls/17539/comments"},"review_comment":{"href":"https://api.github.com/repos/grpc/grpc/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/grpc/grpc/pulls/17539/commits"},"statuses":{"href":"https://api.github.com/repos/grpc/grpc/statuses/1e58ccf1a6dcaac30c5b7650bb5ebd99c68e608a"}},"author_association":"CONTRIBUTOR","merged":false,"mergeable":null,"rebaseable":null,"mergeable_state":"unknown","merged_by":null,"comments":0,"review_comments":0,"maintainer_can_modify":true,"commits":1,"additions":15,"deletions":0,"changed_files":2}}
{"actor":{"display_login":"apolcyn"}}