{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/21219","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/21219/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/21219/comments","events_url":"https://api.github.com/repos/apache/spark/issues/21219/events","html_url":"https://github.com/apache/spark/pull/21219","id":319730131,"number":21219,"title":"[SPARK-24160] ShuffleBlockFetcherIterator should fail if it receives zero-size blocks","user":{"login":"JoshRosen","id":50748,"avatar_url":"https://avatars0.githubusercontent.com/u/50748?v=4","gravatar_id":"","url":"https://api.github.com/users/JoshRosen","html_url":"https://github.com/JoshRosen","followers_url":"https://api.github.com/users/JoshRosen/followers","following_url":"https://api.github.com/users/JoshRosen/following{/other_user}","gists_url":"https://api.github.com/users/JoshRosen/gists{/gist_id}","starred_url":"https://api.github.com/users/JoshRosen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JoshRosen/subscriptions","organizations_url":"https://api.github.com/users/JoshRosen/orgs","repos_url":"https://api.github.com/users/JoshRosen/repos","events_url":"https://api.github.com/users/JoshRosen/events{/privacy}","received_events_url":"https://api.github.com/users/JoshRosen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-05-02T22:12:33Z","updated_at":"2018-05-03T07:05:45Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/21219","html_url":"https://github.com/apache/spark/pull/21219","diff_url":"https://github.com/apache/spark/pull/21219.diff","patch_url":"https://github.com/apache/spark/pull/21219.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThis patch modifies `ShuffleBlockFetcherIterator` so that the receipt of zero-size blocks is treated as an error. This is done as a preventative measure to guard against a potential source of data loss bugs.\r\n\r\nIn the shuffle layer, we guarantee that zero-size blocks will never be requested (a block containing zero records is always 0 bytes in size and is marked as empty such that it will never be legitimately requested by executors). However, the existing code does not fully take advantage of this invariant in the shuffle-read path: the existing code did not explicitly check whether blocks are non-zero-size.\r\n\r\nAdditionally, our decompression and deserialization streams treat zero-size inputs as empty streams rather than errors (EOF might actually be treated as \"end-of-stream\" in certain layers (longstanding behavior dating to earliest versions of Spark) and decompressors like Snappy may be tolerant to zero-size inputs).\r\n\r\nAs a result, if some other bug causes legitimate buffers to be replaced with zero-sized buffers (due to corruption on either the send or receive sides) then this would translate into silent data loss rather than an explicit fail-fast error. \r\n\r\nThis patch addresses this problem by adding a `buf.size != 0` check. See code comments for pointers to tests which guarantee the invariants relied on here.\r\n\r\n## How was this patch tested?\r\n\r\nExisting tests (which required modifications, since some were creating empty buffers in mocks). I also added a test to make sure we fail on zero-size blocks.\r\n\r\nTo test that the zero-size blocks are indeed a potential corruption source, I manually ran a workload in `spark-shell` with a modified build which replaces all buffers with zero-size buffers in the receive path."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/386207645","html_url":"https://github.com/apache/spark/pull/21219#issuecomment-386207645","issue_url":"https://api.github.com/repos/apache/spark/issues/21219","id":386207645,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2018-05-03T07:05:45Z","updated_at":"2018-05-03T07:05:45Z","author_association":"NONE","body":"Test FAILed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90093/\nTest FAILed."}}
{"actor":{"display_login":"AmplabJenkins"}}