{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/15043","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/15043/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/15043/comments","events_url":"https://api.github.com/repos/apache/spark/issues/15043/events","html_url":"https://github.com/apache/spark/pull/15043","id":176163604,"number":15043,"title":"[SPARK-17491] Close serialization stream to fix wrong answer bug in putIteratorAsBytes()","user":{"login":"JoshRosen","id":50748,"avatar_url":"https://avatars.githubusercontent.com/u/50748?v=3","gravatar_id":"","url":"https://api.github.com/users/JoshRosen","html_url":"https://github.com/JoshRosen","followers_url":"https://api.github.com/users/JoshRosen/followers","following_url":"https://api.github.com/users/JoshRosen/following{/other_user}","gists_url":"https://api.github.com/users/JoshRosen/gists{/gist_id}","starred_url":"https://api.github.com/users/JoshRosen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JoshRosen/subscriptions","organizations_url":"https://api.github.com/users/JoshRosen/orgs","repos_url":"https://api.github.com/users/JoshRosen/repos","events_url":"https://api.github.com/users/JoshRosen/events{/privacy}","received_events_url":"https://api.github.com/users/JoshRosen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2016-09-10T06:55:03Z","updated_at":"2016-09-14T01:56:36Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/15043","html_url":"https://github.com/apache/spark/pull/15043","diff_url":"https://github.com/apache/spark/pull/15043.diff","patch_url":"https://github.com/apache/spark/pull/15043.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\n`MemoryStore.putIteratorAsBytes()` may silently lose values when used with `KryoSerializer` because it does not properly close the serialization stream before attempting to deserialize the already-serialized values, which may cause values buffered in Kryo's internal buffers to not be read.\r\n\r\nThis is the root cause behind a user-reported \"wrong answer\" bug in PySpark caching reported by @bennoleslie on the Spark user mailing list in a thread titled \"pyspark persist MEMORY_ONLY vs MEMORY_AND_DISK\". Due to Spark 2.0's automatic use of KryoSerializer for \"safe\" types (such as byte arrays, primitives, etc.) this misuse of serializers manifested itself as silent data corruption rather than a StreamCorrupted error (which you might get from JavaSerializer). \r\n\r\nThe minimal fix, implemented here, is to close the serialization stream before attempting to deserialize written values.\r\n\r\n## How was this patch tested?\r\n\r\nThe original bug was masked by an invalid assert in the memory store test cases: the old assert compared two results record-by-record with `zip` but didn't first check that the lengths of the two collections were equal, causing missing records to go unnoticed.\r\n\r\nTODO: I also should add a dedicated test for PartiallySerializedBlock in its own suite."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/246882740","html_url":"https://github.com/apache/spark/pull/15043#issuecomment-246882740","issue_url":"https://api.github.com/repos/apache/spark/issues/15043","id":246882740,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-09-14T01:56:36Z","updated_at":"2016-09-14T01:56:36Z","body":"Merged build finished. Test FAILed."}}
{"actor":{"display_login":"AmplabJenkins"}}