{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/21933","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/21933/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/21933/comments","events_url":"https://api.github.com/repos/apache/spark/issues/21933/events","html_url":"https://github.com/apache/spark/pull/21933","id":346262016,"node_id":"MDExOlB1bGxSZXF1ZXN0MjA1MTY5NTE5","number":21933,"title":"[SPARK-24917][CORE] make chunk size configurable","user":{"login":"vincent-grosbois","id":6704932,"node_id":"MDQ6VXNlcjY3MDQ5MzI=","avatar_url":"https://avatars2.githubusercontent.com/u/6704932?v=4","gravatar_id":"","url":"https://api.github.com/users/vincent-grosbois","html_url":"https://github.com/vincent-grosbois","followers_url":"https://api.github.com/users/vincent-grosbois/followers","following_url":"https://api.github.com/users/vincent-grosbois/following{/other_user}","gists_url":"https://api.github.com/users/vincent-grosbois/gists{/gist_id}","starred_url":"https://api.github.com/users/vincent-grosbois/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vincent-grosbois/subscriptions","organizations_url":"https://api.github.com/users/vincent-grosbois/orgs","repos_url":"https://api.github.com/users/vincent-grosbois/repos","events_url":"https://api.github.com/users/vincent-grosbois/events{/privacy}","received_events_url":"https://api.github.com/users/vincent-grosbois/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":15,"created_at":"2018-07-31T16:22:35Z","updated_at":"2018-08-07T08:38:21Z","closed_at":null,"author_association":"NONE","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/21933","html_url":"https://github.com/apache/spark/pull/21933","diff_url":"https://github.com/apache/spark/pull/21933.diff","patch_url":"https://github.com/apache/spark/pull/21933.patch"},"body":"Add an option in Spark configuration to change the chunk size (which is by default 4 Mb).\r\n\r\nThis would allow to bypass the issue mentionned in SPARK-24917 by allowing users to define larger chunks.\r\nExplanation:\r\ncurrently, using netty < 4.1.28 (before this patch https://github.com/netty/netty/commit/9b95b8ee628983e3e4434da93fffb893edff4aa2), sending a ChunkedByteBuffer with more than 16 chunks over the network will trigger a \"merge\" of all the blocks into one big transient array that is then sent over the network. This is problematic as the total memory for all chunks can be high (2GB) and this would then trigger an allocation of 2GB to merge everything, which will create  OOM errors.\r\nA possibility to bypass this netty behavior is to make sure that they data is never split into more than 16 chunks. One way to do this is to create bigger chunks, which is currently fixed to 4MB. In this commit I'm allowing users to define bigger chunk sizes for their job, which allowed us to bypass this OOM error.\r\n\r\n## What changes were proposed in this pull request?\r\nI'm introducing a configuration parameter to define the chunk size\r\n\r\n## How was this patch tested?\r\nTested on several spark jobs, the changes actually work and generates \"chunks\" of the indicated size"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/410980062","html_url":"https://github.com/apache/spark/pull/21933#issuecomment-410980062","issue_url":"https://api.github.com/repos/apache/spark/issues/21933","id":410980062,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMDk4MDA2Mg==","user":{"login":"HyukjinKwon","id":6477701,"node_id":"MDQ6VXNlcjY0Nzc3MDE=","avatar_url":"https://avatars0.githubusercontent.com/u/6477701?v=4","gravatar_id":"","url":"https://api.github.com/users/HyukjinKwon","html_url":"https://github.com/HyukjinKwon","followers_url":"https://api.github.com/users/HyukjinKwon/followers","following_url":"https://api.github.com/users/HyukjinKwon/following{/other_user}","gists_url":"https://api.github.com/users/HyukjinKwon/gists{/gist_id}","starred_url":"https://api.github.com/users/HyukjinKwon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HyukjinKwon/subscriptions","organizations_url":"https://api.github.com/users/HyukjinKwon/orgs","repos_url":"https://api.github.com/users/HyukjinKwon/repos","events_url":"https://api.github.com/users/HyukjinKwon/events{/privacy}","received_events_url":"https://api.github.com/users/HyukjinKwon/received_events","type":"User","site_admin":false},"created_at":"2018-08-07T08:38:21Z","updated_at":"2018-08-07T08:38:21Z","author_association":"MEMBER","body":"retest this please"}}
{"actor":{"display_login":"HyukjinKwon"}}