{"action":"created","comment":{"url":"https://api.github.com/repos/solana-labs/solana/pulls/comments/240449783","pull_request_review_id":183482717,"id":240449783,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDI0MDQ0OTc4Mw==","diff_hunk":"@@ -37,13 +38,47 @@ fn retransmit(\n             .to_owned(),\n     );\n \n-    for b in &mut dq {\n-        ClusterInfo::retransmit(&cluster_info, b, sock)?;\n+    let peers = cluster_info.read().unwrap().sorted_tvu_peers(bank);\n+    let my_id = cluster_info.read().unwrap().id();\n+    //calc num_layers and num_neighborhoods using the total number of nodes\n+    let (num_layers, layer_indices) = ClusterInfo::describe_data_plane(\n+        peers.len(),\n+        DATA_PLANE_FANOUT,\n+        NEIGHBORHOOD_SIZE,\n+        GROW_LAYER_CAPACITY,\n+    );\n+    if num_layers <= 1 {\n+        /* single layer data plane */\n+        for b in &mut dq {\n+            ClusterInfo::retransmit(&cluster_info, b, sock)?;\n+        }\n+    } else {\n+        //find my index (my ix is the same as the first node with smaller stake)\n+        let my_index = peers\n+            .iter()\n+            .position(|ci| bank.get_stake(&ci.id) <= bank.get_stake(&my_id));\n+        //find my layer\n+        let locality = ClusterInfo::localize(\n+            &layer_indices,\n+            NEIGHBORHOOD_SIZE,\n+            my_index.unwrap_or(peers.len() - 1),\n+        );\n+        let mut retransmit_peers =\n+            peers[locality.neighbor_bounds.0..locality.neighbor_bounds.1].to_vec();\n+        locality.child_layer_peers.iter().cloned().for_each(|ix| {\n+            if let Some(peer) = peers.get(ix) {\n+                retransmit_peers.push(peer.clone());\n+            }\n+        });\n+\n+        for b in &mut dq {\n+            ClusterInfo::retransmit_to(&cluster_info, retransmit_peers.to_vec(), b, sock)?;","path":"src/retransmit_stage.rs","position":61,"original_position":61,"commit_id":"cc7dab3edb1d13b7d2363dd13b06aebe3879460d","original_commit_id":"cc7dab3edb1d13b7d2363dd13b06aebe3879460d","user":{"login":"sagar-solana","id":44789711,"node_id":"MDQ6VXNlcjQ0Nzg5NzEx","avatar_url":"https://avatars0.githubusercontent.com/u/44789711?v=4","gravatar_id":"","url":"https://api.github.com/users/sagar-solana","html_url":"https://github.com/sagar-solana","followers_url":"https://api.github.com/users/sagar-solana/followers","following_url":"https://api.github.com/users/sagar-solana/following{/other_user}","gists_url":"https://api.github.com/users/sagar-solana/gists{/gist_id}","starred_url":"https://api.github.com/users/sagar-solana/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sagar-solana/subscriptions","organizations_url":"https://api.github.com/users/sagar-solana/orgs","repos_url":"https://api.github.com/users/sagar-solana/repos","events_url":"https://api.github.com/users/sagar-solana/events{/privacy}","received_events_url":"https://api.github.com/users/sagar-solana/received_events","type":"User","site_admin":false},"body":"Yeah this looks weird. I'll check it out. ","created_at":"2018-12-11T01:58:37Z","updated_at":"2018-12-11T01:58:37Z","html_url":"https://github.com/solana-labs/solana/pull/2058#discussion_r240449783","pull_request_url":"https://api.github.com/repos/solana-labs/solana/pulls/2058","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/solana-labs/solana/pulls/comments/240449783"},"html":{"href":"https://github.com/solana-labs/solana/pull/2058#discussion_r240449783"},"pull_request":{"href":"https://api.github.com/repos/solana-labs/solana/pulls/2058"}},"in_reply_to_id":240448171},"pull_request":{"url":"https://api.github.com/repos/solana-labs/solana/pulls/2058","id":237010275,"node_id":"MDExOlB1bGxSZXF1ZXN0MjM3MDEwMjc1","html_url":"https://github.com/solana-labs/solana/pull/2058","diff_url":"https://github.com/solana-labs/solana/pull/2058.diff","patch_url":"https://github.com/solana-labs/solana/pull/2058.patch","issue_url":"https://api.github.com/repos/solana-labs/solana/issues/2058","number":2058,"state":"open","locked":false,"title":"Dynamic N layer 'avalanche' broadcast and retransmit","user":{"login":"sagar-solana","id":44789711,"node_id":"MDQ6VXNlcjQ0Nzg5NzEx","avatar_url":"https://avatars0.githubusercontent.com/u/44789711?v=4","gravatar_id":"","url":"https://api.github.com/users/sagar-solana","html_url":"https://github.com/sagar-solana","followers_url":"https://api.github.com/users/sagar-solana/followers","following_url":"https://api.github.com/users/sagar-solana/following{/other_user}","gists_url":"https://api.github.com/users/sagar-solana/gists{/gist_id}","starred_url":"https://api.github.com/users/sagar-solana/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sagar-solana/subscriptions","organizations_url":"https://api.github.com/users/sagar-solana/orgs","repos_url":"https://api.github.com/users/sagar-solana/repos","events_url":"https://api.github.com/users/sagar-solana/events{/privacy}","received_events_url":"https://api.github.com/users/sagar-solana/received_events","type":"User","site_admin":false},"body":"#### Problem\r\nIn a large network that tasks the leader with broadcasting to all validators, the leader's bandwidth can get saturated. To counter this we implemented a single layer avalanche where the leader doesn't have to replay all messages to each validator, instead validators can share the leader's messages with each other. But even this single layer avalanche can be crippled when network sizes get larger than a few hundred. We need more layers.\r\n\r\n#### Summary of Changes\r\nImplemented an N layer avalanche mechanism to distribute leader messages across the entire network. \r\nThere are a couple of knobs to tweak here to figure out if the layers should be increasing, or constant in capacity and how many nodes are allowed in each \"neighborhood\". \r\nThe general idea behind these changes is described in #2008 but that RFC still needs updating.\r\n\r\nFixes #\r\n","created_at":"2018-12-07T22:23:10Z","updated_at":"2018-12-11T01:58:37Z","closed_at":null,"merged_at":null,"merge_commit_sha":"e849a0195107e30efbe2e1e39a8e696f94eaaca2","assignee":null,"assignees":[],"requested_reviewers":[{"login":"garious","id":55449,"node_id":"MDQ6VXNlcjU1NDQ5","avatar_url":"https://avatars1.githubusercontent.com/u/55449?v=4","gravatar_id":"","url":"https://api.github.com/users/garious","html_url":"https://github.com/garious","followers_url":"https://api.github.com/users/garious/followers","following_url":"https://api.github.com/users/garious/following{/other_user}","gists_url":"https://api.github.com/users/garious/gists{/gist_id}","starred_url":"https://api.github.com/users/garious/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/garious/subscriptions","organizations_url":"https://api.github.com/users/garious/orgs","repos_url":"https://api.github.com/users/garious/repos","events_url":"https://api.github.com/users/garious/events{/privacy}","received_events_url":"https://api.github.com/users/garious/received_events","type":"User","site_admin":false}],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/solana-labs/solana/pulls/2058/commits","review_comments_url":"https://api.github.com/repos/solana-labs/solana/pulls/2058/comments","review_comment_url":"https://api.github.com/repos/solana-labs/solana/pulls/comments{/number}","comments_url":"https://api.github.com/repos/solana-labs/solana/issues/2058/comments","statuses_url":"https://api.github.com/repos/solana-labs/solana/statuses/cc7dab3edb1d13b7d2363dd13b06aebe3879460d","head":{"label":"sagar-solana:avalanche","ref":"avalanche","sha":"cc7dab3edb1d13b7d2363dd13b06aebe3879460d","user":{"login":"sagar-solana","id":44789711,"node_id":"MDQ6VXNlcjQ0Nzg5NzEx","avatar_url":"https://avatars0.githubusercontent.com/u/44789711?v=4","gravatar_id":"","url":"https://api.github.com/users/sagar-solana","html_url":"https://github.com/sagar-solana","followers_url":"https://api.github.com/users/sagar-solana/followers","following_url":"https://api.github.com/users/sagar-solana/following{/other_user}","gists_url":"https://api.github.com/users/sagar-solana/gists{/gist_id}","starred_url":"https://api.github.com/users/sagar-solana/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sagar-solana/subscriptions","organizations_url":"https://api.github.com/users/sagar-solana/orgs","repos_url":"https://api.github.com/users/sagar-solana/repos","events_url":"https://api.github.com/users/sagar-solana/events{/privacy}","received_events_url":"https://api.github.com/users/sagar-solana/received_events","type":"User","site_admin":false},"repo":{"id":156295308,"node_id":"MDEwOlJlcG9zaXRvcnkxNTYyOTUzMDg=","name":"solana","full_name":"sagar-solana/solana","private":false,"owner":{"login":"sagar-solana","id":44789711,"node_id":"MDQ6VXNlcjQ0Nzg5NzEx","avatar_url":"https://avatars0.githubusercontent.com/u/44789711?v=4","gravatar_id":"","url":"https://api.github.com/users/sagar-solana","html_url":"https://github.com/sagar-solana","followers_url":"https://api.github.com/users/sagar-solana/followers","following_url":"https://api.github.com/users/sagar-solana/following{/other_user}","gists_url":"https://api.github.com/users/sagar-solana/gists{/gist_id}","starred_url":"https://api.github.com/users/sagar-solana/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sagar-solana/subscriptions","organizations_url":"https://api.github.com/users/sagar-solana/orgs","repos_url":"https://api.github.com/users/sagar-solana/repos","events_url":"https://api.github.com/users/sagar-solana/events{/privacy}","received_events_url":"https://api.github.com/users/sagar-solana/received_events","type":"User","site_admin":false},"html_url":"https://github.com/sagar-solana/solana","description":"Blockchain, Rebuilt for Scale","fork":true,"url":"https://api.github.com/repos/sagar-solana/solana","forks_url":"https://api.github.com/repos/sagar-solana/solana/forks","keys_url":"https://api.github.com/repos/sagar-solana/solana/keys{/key_id}","collaborators_url":"https://api.github.com/repos/sagar-solana/solana/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/sagar-solana/solana/teams","hooks_url":"https://api.github.com/repos/sagar-solana/solana/hooks","issue_events_url":"https://api.github.com/repos/sagar-solana/solana/issues/events{/number}","events_url":"https://api.github.com/repos/sagar-solana/solana/events","assignees_url":"https://api.github.com/repos/sagar-solana/solana/assignees{/user}","branches_url":"https://api.github.com/repos/sagar-solana/solana/branches{/branch}","tags_url":"https://api.github.com/repos/sagar-solana/solana/tags","blobs_url":"https://api.github.com/repos/sagar-solana/solana/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/sagar-solana/solana/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/sagar-solana/solana/git/refs{/sha}","trees_url":"https://api.github.com/repos/sagar-solana/solana/git/trees{/sha}","statuses_url":"https://api.github.com/repos/sagar-solana/solana/statuses/{sha}","languages_url":"https://api.github.com/repos/sagar-solana/solana/languages","stargazers_url":"https://api.github.com/repos/sagar-solana/solana/stargazers","contributors_url":"https://api.github.com/repos/sagar-solana/solana/contributors","subscribers_url":"https://api.github.com/repos/sagar-solana/solana/subscribers","subscription_url":"https://api.github.com/repos/sagar-solana/solana/subscription","commits_url":"https://api.github.com/repos/sagar-solana/solana/commits{/sha}","git_commits_url":"https://api.github.com/repos/sagar-solana/solana/git/commits{/sha}","comments_url":"https://api.github.com/repos/sagar-solana/solana/comments{/number}","issue_comment_url":"https://api.github.com/repos/sagar-solana/solana/issues/comments{/number}","contents_url":"https://api.github.com/repos/sagar-solana/solana/contents/{+path}","compare_url":"https://api.github.com/repos/sagar-solana/solana/compare/{base}...{head}","merges_url":"https://api.github.com/repos/sagar-solana/solana/merges","archive_url":"https://api.github.com/repos/sagar-solana/solana/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/sagar-solana/solana/downloads","issues_url":"https://api.github.com/repos/sagar-solana/solana/issues{/number}","pulls_url":"https://api.github.com/repos/sagar-solana/solana/pulls{/number}","milestones_url":"https://api.github.com/repos/sagar-solana/solana/milestones{/number}","notifications_url":"https://api.github.com/repos/sagar-solana/solana/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/sagar-solana/solana/labels{/name}","releases_url":"https://api.github.com/repos/sagar-solana/solana/releases{/id}","deployments_url":"https://api.github.com/repos/sagar-solana/solana/deployments","created_at":"2018-11-05T23:04:42Z","updated_at":"2018-12-10T22:35:14Z","pushed_at":"2018-12-10T23:00:09Z","git_url":"git://github.com/sagar-solana/solana.git","ssh_url":"git@github.com:sagar-solana/solana.git","clone_url":"https://github.com/sagar-solana/solana.git","svn_url":"https://github.com/sagar-solana/solana","homepage":"https://solana.com","size":13983,"stargazers_count":0,"watchers_count":0,"language":"Rust","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"other","name":"Other","spdx_id":"NOASSERTION","url":null,"node_id":"MDc6TGljZW5zZTA="},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"solana-labs:master","ref":"master","sha":"9556a9be17ed70f404827a14b89152f3f0fe05df","user":{"login":"solana-labs","id":35608259,"node_id":"MDEyOk9yZ2FuaXphdGlvbjM1NjA4MjU5","avatar_url":"https://avatars3.githubusercontent.com/u/35608259?v=4","gravatar_id":"","url":"https://api.github.com/users/solana-labs","html_url":"https://github.com/solana-labs","followers_url":"https://api.github.com/users/solana-labs/followers","following_url":"https://api.github.com/users/solana-labs/following{/other_user}","gists_url":"https://api.github.com/users/solana-labs/gists{/gist_id}","starred_url":"https://api.github.com/users/solana-labs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/solana-labs/subscriptions","organizations_url":"https://api.github.com/users/solana-labs/orgs","repos_url":"https://api.github.com/users/solana-labs/repos","events_url":"https://api.github.com/users/solana-labs/events{/privacy}","received_events_url":"https://api.github.com/users/solana-labs/received_events","type":"Organization","site_admin":false},"repo":{"id":121470383,"node_id":"MDEwOlJlcG9zaXRvcnkxMjE0NzAzODM=","name":"solana","full_name":"solana-labs/solana","private":false,"owner":{"login":"solana-labs","id":35608259,"node_id":"MDEyOk9yZ2FuaXphdGlvbjM1NjA4MjU5","avatar_url":"https://avatars3.githubusercontent.com/u/35608259?v=4","gravatar_id":"","url":"https://api.github.com/users/solana-labs","html_url":"https://github.com/solana-labs","followers_url":"https://api.github.com/users/solana-labs/followers","following_url":"https://api.github.com/users/solana-labs/following{/other_user}","gists_url":"https://api.github.com/users/solana-labs/gists{/gist_id}","starred_url":"https://api.github.com/users/solana-labs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/solana-labs/subscriptions","organizations_url":"https://api.github.com/users/solana-labs/orgs","repos_url":"https://api.github.com/users/solana-labs/repos","events_url":"https://api.github.com/users/solana-labs/events{/privacy}","received_events_url":"https://api.github.com/users/solana-labs/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/solana-labs/solana","description":"Blockchain Rebuilt for Scale","fork":false,"url":"https://api.github.com/repos/solana-labs/solana","forks_url":"https://api.github.com/repos/solana-labs/solana/forks","keys_url":"https://api.github.com/repos/solana-labs/solana/keys{/key_id}","collaborators_url":"https://api.github.com/repos/solana-labs/solana/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/solana-labs/solana/teams","hooks_url":"https://api.github.com/repos/solana-labs/solana/hooks","issue_events_url":"https://api.github.com/repos/solana-labs/solana/issues/events{/number}","events_url":"https://api.github.com/repos/solana-labs/solana/events","assignees_url":"https://api.github.com/repos/solana-labs/solana/assignees{/user}","branches_url":"https://api.github.com/repos/solana-labs/solana/branches{/branch}","tags_url":"https://api.github.com/repos/solana-labs/solana/tags","blobs_url":"https://api.github.com/repos/solana-labs/solana/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/solana-labs/solana/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/solana-labs/solana/git/refs{/sha}","trees_url":"https://api.github.com/repos/solana-labs/solana/git/trees{/sha}","statuses_url":"https://api.github.com/repos/solana-labs/solana/statuses/{sha}","languages_url":"https://api.github.com/repos/solana-labs/solana/languages","stargazers_url":"https://api.github.com/repos/solana-labs/solana/stargazers","contributors_url":"https://api.github.com/repos/solana-labs/solana/contributors","subscribers_url":"https://api.github.com/repos/solana-labs/solana/subscribers","subscription_url":"https://api.github.com/repos/solana-labs/solana/subscription","commits_url":"https://api.github.com/repos/solana-labs/solana/commits{/sha}","git_commits_url":"https://api.github.com/repos/solana-labs/solana/git/commits{/sha}","comments_url":"https://api.github.com/repos/solana-labs/solana/comments{/number}","issue_comment_url":"https://api.github.com/repos/solana-labs/solana/issues/comments{/number}","contents_url":"https://api.github.com/repos/solana-labs/solana/contents/{+path}","compare_url":"https://api.github.com/repos/solana-labs/solana/compare/{base}...{head}","merges_url":"https://api.github.com/repos/solana-labs/solana/merges","archive_url":"https://api.github.com/repos/solana-labs/solana/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/solana-labs/solana/downloads","issues_url":"https://api.github.com/repos/solana-labs/solana/issues{/number}","pulls_url":"https://api.github.com/repos/solana-labs/solana/pulls{/number}","milestones_url":"https://api.github.com/repos/solana-labs/solana/milestones{/number}","notifications_url":"https://api.github.com/repos/solana-labs/solana/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/solana-labs/solana/labels{/name}","releases_url":"https://api.github.com/repos/solana-labs/solana/releases{/id}","deployments_url":"https://api.github.com/repos/solana-labs/solana/deployments","created_at":"2018-02-14T04:26:27Z","updated_at":"2018-12-11T00:10:49Z","pushed_at":"2018-12-11T01:21:35Z","git_url":"git://github.com/solana-labs/solana.git","ssh_url":"git@github.com:solana-labs/solana.git","clone_url":"https://github.com/solana-labs/solana.git","svn_url":"https://github.com/solana-labs/solana","homepage":"https://solana.com","size":14586,"stargazers_count":397,"watchers_count":397,"language":"Rust","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":68,"mirror_url":null,"archived":false,"open_issues_count":177,"license":{"key":"other","name":"Other","spdx_id":"NOASSERTION","url":null,"node_id":"MDc6TGljZW5zZTA="},"forks":68,"open_issues":177,"watchers":397,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/solana-labs/solana/pulls/2058"},"html":{"href":"https://github.com/solana-labs/solana/pull/2058"},"issue":{"href":"https://api.github.com/repos/solana-labs/solana/issues/2058"},"comments":{"href":"https://api.github.com/repos/solana-labs/solana/issues/2058/comments"},"review_comments":{"href":"https://api.github.com/repos/solana-labs/solana/pulls/2058/comments"},"review_comment":{"href":"https://api.github.com/repos/solana-labs/solana/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/solana-labs/solana/pulls/2058/commits"},"statuses":{"href":"https://api.github.com/repos/solana-labs/solana/statuses/cc7dab3edb1d13b7d2363dd13b06aebe3879460d"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"sagar-solana"}}