{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/240792081","pull_request_review_id":183904901,"id":240792081,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDI0MDc5MjA4MQ==","diff_hunk":"@@ -95,6 +111,38 @@ public static String getRemoteAddress(Channel channel) {\n     return \"<unknown remote>\";\n   }\n \n+  /**\n+   * Returns the default number of threads for both the Netty client and server thread pools.\n+   * If numUsableCores is 0, we will use Runtime get an approximate number of available cores.\n+   */\n+  public static int defaultNumThreads(int numUsableCores) {\n+    final int availableCores;\n+    if (numUsableCores > 0) {\n+      availableCores = numUsableCores;\n+    } else {\n+      availableCores = Runtime.getRuntime().availableProcessors();\n+    }\n+    return Math.min(availableCores, MAX_DEFAULT_NETTY_THREADS);\n+  }\n+\n+  /**\n+   * Returns the lazily created shared pooled ByteBuf allocator for the specified allowCache\n+   * parameter value.\n+   */\n+  public static synchronized PooledByteBufAllocator getSharedPooledByteBufAllocator(\n+      boolean allowDirectBufs,\n+      boolean allowCache) {\n+    final int index = allowCache ? 0 : 1;","path":"common/network-common/src/main/java/org/apache/spark/network/util/NettyUtils.java","position":48,"original_position":48,"commit_id":"ae8fa8e81510c69f84444c368cad25788f5423b4","original_commit_id":"ae8fa8e81510c69f84444c368cad25788f5423b4","user":{"login":"ankuriitg","id":1723600,"node_id":"MDQ6VXNlcjE3MjM2MDA=","avatar_url":"https://avatars1.githubusercontent.com/u/1723600?v=4","gravatar_id":"","url":"https://api.github.com/users/ankuriitg","html_url":"https://github.com/ankuriitg","followers_url":"https://api.github.com/users/ankuriitg/followers","following_url":"https://api.github.com/users/ankuriitg/following{/other_user}","gists_url":"https://api.github.com/users/ankuriitg/gists{/gist_id}","starred_url":"https://api.github.com/users/ankuriitg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ankuriitg/subscriptions","organizations_url":"https://api.github.com/users/ankuriitg/orgs","repos_url":"https://api.github.com/users/ankuriitg/repos","events_url":"https://api.github.com/users/ankuriitg/events{/privacy}","received_events_url":"https://api.github.com/users/ankuriitg/received_events","type":"User","site_admin":false},"body":"I don't think that you need to change createPooledByteBufAllocator if you change this method. allowCache is used to determine whether to create a pool with or without cache in createPooledByteBufAllocator. This is different from getSharedPooledByteBufAllocator which is using allowCache to determine this plus, it is also using allowCache to determine whether to return a pooledAllocator for a client or a server.\r\n\r\nYou may still want to keep this change if allowCache is the only thing that distinguishes the two pooled allocators but if it is not then it may be better to create another variable.","created_at":"2018-12-11T21:10:55Z","updated_at":"2018-12-11T21:10:55Z","html_url":"https://github.com/apache/spark/pull/23278#discussion_r240792081","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/23278","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/240792081"},"html":{"href":"https://github.com/apache/spark/pull/23278#discussion_r240792081"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/23278"}},"in_reply_to_id":240755466},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/23278","id":237405286,"node_id":"MDExOlB1bGxSZXF1ZXN0MjM3NDA1Mjg2","html_url":"https://github.com/apache/spark/pull/23278","diff_url":"https://github.com/apache/spark/pull/23278.diff","patch_url":"https://github.com/apache/spark/pull/23278.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/23278","number":23278,"state":"open","locked":false,"title":"[SPARK-24920][Core] Allow sharing Netty's memory pool allocators","user":{"login":"attilapiros","id":2017933,"node_id":"MDQ6VXNlcjIwMTc5MzM=","avatar_url":"https://avatars1.githubusercontent.com/u/2017933?v=4","gravatar_id":"","url":"https://api.github.com/users/attilapiros","html_url":"https://github.com/attilapiros","followers_url":"https://api.github.com/users/attilapiros/followers","following_url":"https://api.github.com/users/attilapiros/following{/other_user}","gists_url":"https://api.github.com/users/attilapiros/gists{/gist_id}","starred_url":"https://api.github.com/users/attilapiros/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/attilapiros/subscriptions","organizations_url":"https://api.github.com/users/attilapiros/orgs","repos_url":"https://api.github.com/users/attilapiros/repos","events_url":"https://api.github.com/users/attilapiros/events{/privacy}","received_events_url":"https://api.github.com/users/attilapiros/received_events","type":"User","site_admin":false},"body":"## What changes were proposed in this pull request?\r\n\r\nIntroducing shared polled ByteBuf allocators. \r\nThis feature can be enabled via the \"spark.network.sharedByteBufAllocators.enabled\" configuration. \r\n\r\nWhen it is on then only two pooled ByteBuf allocators are created: \r\n- one for transport servers where caching is allowed and\r\n- one for transport clients where caching is disabled\r\n\r\nThis way the cache allowance remains as before. \r\nBoth shareable pools are created with numCores parameter set to 0 (which defaults to the available processors) as conf.serverThreads() and conf.clientThreads() are module dependant and the lazy creation of this allocators would lead to unpredicted behaviour.\r\n\r\nWhen \"spark.network.sharedByteBufAllocators\" is false then a new allocator is created for every transport client and server separately as was before this PR.\r\n\r\n## How was this patch tested?\r\n\r\nExisting unit tests.\r\n","created_at":"2018-12-10T17:31:51Z","updated_at":"2018-12-11T21:10:55Z","closed_at":null,"merged_at":null,"merge_commit_sha":"479bad350ba21ff8a3b76ead94c3fa0bb3790437","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/23278/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/23278/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/23278/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/ae8fa8e81510c69f84444c368cad25788f5423b4","head":{"label":"attilapiros:SPARK-24920","ref":"SPARK-24920","sha":"ae8fa8e81510c69f84444c368cad25788f5423b4","user":{"login":"attilapiros","id":2017933,"node_id":"MDQ6VXNlcjIwMTc5MzM=","avatar_url":"https://avatars1.githubusercontent.com/u/2017933?v=4","gravatar_id":"","url":"https://api.github.com/users/attilapiros","html_url":"https://github.com/attilapiros","followers_url":"https://api.github.com/users/attilapiros/followers","following_url":"https://api.github.com/users/attilapiros/following{/other_user}","gists_url":"https://api.github.com/users/attilapiros/gists{/gist_id}","starred_url":"https://api.github.com/users/attilapiros/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/attilapiros/subscriptions","organizations_url":"https://api.github.com/users/attilapiros/orgs","repos_url":"https://api.github.com/users/attilapiros/repos","events_url":"https://api.github.com/users/attilapiros/events{/privacy}","received_events_url":"https://api.github.com/users/attilapiros/received_events","type":"User","site_admin":false},"repo":{"id":114023960,"node_id":"MDEwOlJlcG9zaXRvcnkxMTQwMjM5NjA=","name":"spark","full_name":"attilapiros/spark","private":false,"owner":{"login":"attilapiros","id":2017933,"node_id":"MDQ6VXNlcjIwMTc5MzM=","avatar_url":"https://avatars1.githubusercontent.com/u/2017933?v=4","gravatar_id":"","url":"https://api.github.com/users/attilapiros","html_url":"https://github.com/attilapiros","followers_url":"https://api.github.com/users/attilapiros/followers","following_url":"https://api.github.com/users/attilapiros/following{/other_user}","gists_url":"https://api.github.com/users/attilapiros/gists{/gist_id}","starred_url":"https://api.github.com/users/attilapiros/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/attilapiros/subscriptions","organizations_url":"https://api.github.com/users/attilapiros/orgs","repos_url":"https://api.github.com/users/attilapiros/repos","events_url":"https://api.github.com/users/attilapiros/events{/privacy}","received_events_url":"https://api.github.com/users/attilapiros/received_events","type":"User","site_admin":false},"html_url":"https://github.com/attilapiros/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/attilapiros/spark","forks_url":"https://api.github.com/repos/attilapiros/spark/forks","keys_url":"https://api.github.com/repos/attilapiros/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/attilapiros/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/attilapiros/spark/teams","hooks_url":"https://api.github.com/repos/attilapiros/spark/hooks","issue_events_url":"https://api.github.com/repos/attilapiros/spark/issues/events{/number}","events_url":"https://api.github.com/repos/attilapiros/spark/events","assignees_url":"https://api.github.com/repos/attilapiros/spark/assignees{/user}","branches_url":"https://api.github.com/repos/attilapiros/spark/branches{/branch}","tags_url":"https://api.github.com/repos/attilapiros/spark/tags","blobs_url":"https://api.github.com/repos/attilapiros/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/attilapiros/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/attilapiros/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/attilapiros/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/attilapiros/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/attilapiros/spark/languages","stargazers_url":"https://api.github.com/repos/attilapiros/spark/stargazers","contributors_url":"https://api.github.com/repos/attilapiros/spark/contributors","subscribers_url":"https://api.github.com/repos/attilapiros/spark/subscribers","subscription_url":"https://api.github.com/repos/attilapiros/spark/subscription","commits_url":"https://api.github.com/repos/attilapiros/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/attilapiros/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/attilapiros/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/attilapiros/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/attilapiros/spark/contents/{+path}","compare_url":"https://api.github.com/repos/attilapiros/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/attilapiros/spark/merges","archive_url":"https://api.github.com/repos/attilapiros/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/attilapiros/spark/downloads","issues_url":"https://api.github.com/repos/attilapiros/spark/issues{/number}","pulls_url":"https://api.github.com/repos/attilapiros/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/attilapiros/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/attilapiros/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/attilapiros/spark/labels{/name}","releases_url":"https://api.github.com/repos/attilapiros/spark/releases{/id}","deployments_url":"https://api.github.com/repos/attilapiros/spark/deployments","created_at":"2017-12-12T18:15:59Z","updated_at":"2018-12-10T09:59:34Z","pushed_at":"2018-12-10T23:12:15Z","git_url":"git://github.com/attilapiros/spark.git","ssh_url":"git@github.com:attilapiros/spark.git","clone_url":"https://github.com/attilapiros/spark.git","svn_url":"https://github.com/attilapiros/spark","homepage":null,"size":288056,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":1,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"90c77ea3132d0b7a12c316bd42fb8d0f59bee253","user":{"login":"apache","id":47359,"node_id":"MDEyOk9yZ2FuaXphdGlvbjQ3MzU5","avatar_url":"https://avatars0.githubusercontent.com/u/47359?v=4","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"node_id":"MDEwOlJlcG9zaXRvcnkxNzE2NTY1OA==","name":"spark","full_name":"apache/spark","private":false,"owner":{"login":"apache","id":47359,"node_id":"MDEyOk9yZ2FuaXphdGlvbjQ3MzU5","avatar_url":"https://avatars0.githubusercontent.com/u/47359?v=4","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/apache/spark","description":"Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","deployments_url":"https://api.github.com/repos/apache/spark/deployments","created_at":"2014-02-25T08:00:08Z","updated_at":"2018-12-11T17:16:23Z","pushed_at":"2018-12-11T20:40:14Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":"","size":289208,"stargazers_count":19824,"watchers_count":19824,"language":"Scala","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":17671,"mirror_url":null,"archived":false,"open_issues_count":488,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":17671,"open_issues":488,"watchers":19824,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/23278"},"html":{"href":"https://github.com/apache/spark/pull/23278"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/23278"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/23278/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/23278/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/23278/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/ae8fa8e81510c69f84444c368cad25788f5423b4"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"ankuriitg"}}