{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/21212","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/21212/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/21212/comments","events_url":"https://api.github.com/repos/apache/spark/issues/21212/events","html_url":"https://github.com/apache/spark/pull/21212","id":319417300,"number":21212,"title":"[SPARK-24143] filter empty blocks when convert mapstatus to (blockId, size) pair","user":{"login":"jinxing64","id":4058918,"avatar_url":"https://avatars0.githubusercontent.com/u/4058918?v=4","gravatar_id":"","url":"https://api.github.com/users/jinxing64","html_url":"https://github.com/jinxing64","followers_url":"https://api.github.com/users/jinxing64/followers","following_url":"https://api.github.com/users/jinxing64/following{/other_user}","gists_url":"https://api.github.com/users/jinxing64/gists{/gist_id}","starred_url":"https://api.github.com/users/jinxing64/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jinxing64/subscriptions","organizations_url":"https://api.github.com/users/jinxing64/orgs","repos_url":"https://api.github.com/users/jinxing64/repos","events_url":"https://api.github.com/users/jinxing64/events{/privacy}","received_events_url":"https://api.github.com/users/jinxing64/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":11,"created_at":"2018-05-02T05:48:56Z","updated_at":"2018-05-04T02:31:43Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/21212","html_url":"https://github.com/apache/spark/pull/21212","diff_url":"https://github.com/apache/spark/pull/21212.diff","patch_url":"https://github.com/apache/spark/pull/21212.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nIn current code(`MapOutputTracker.convertMapStatuses`), mapstatus are converted to (blockId, size) pair for all blocks – no matter the block is empty or not, which result in OOM when there are lots of consecutive empty blocks, especially when adaptive execution is enabled.\r\n\r\n(blockId, size) pair is only used in `ShuffleBlockFetcherIterator` to control shuffle-read and only non-empty block request is sent. Can we just filter out the empty blocks in MapOutputTracker.convertMapStatuses and save memory?\r\n\r\n\r\n## How was this patch tested?\r\n\r\nnot added yet."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/386491833","html_url":"https://github.com/apache/spark/pull/21212#issuecomment-386491833","issue_url":"https://api.github.com/repos/apache/spark/issues/21212","id":386491833,"user":{"login":"cloud-fan","id":3182036,"avatar_url":"https://avatars3.githubusercontent.com/u/3182036?v=4","gravatar_id":"","url":"https://api.github.com/users/cloud-fan","html_url":"https://github.com/cloud-fan","followers_url":"https://api.github.com/users/cloud-fan/followers","following_url":"https://api.github.com/users/cloud-fan/following{/other_user}","gists_url":"https://api.github.com/users/cloud-fan/gists{/gist_id}","starred_url":"https://api.github.com/users/cloud-fan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cloud-fan/subscriptions","organizations_url":"https://api.github.com/users/cloud-fan/orgs","repos_url":"https://api.github.com/users/cloud-fan/repos","events_url":"https://api.github.com/users/cloud-fan/events{/privacy}","received_events_url":"https://api.github.com/users/cloud-fan/received_events","type":"User","site_admin":false},"created_at":"2018-05-04T02:31:43Z","updated_at":"2018-05-04T02:31:43Z","author_association":"CONTRIBUTOR","body":"> do you mean optimize space usage for MapStatus when there are lots of consecutive empty-blocks ?\r\n\r\nYea, something like doing an RLE for the size array in `CompressedMapStatus`. But this can be done as a followup."}}
{"actor":{"display_login":"cloud-fan"}}