{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/15026","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/15026/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/15026/comments","events_url":"https://api.github.com/repos/apache/spark/issues/15026/events","html_url":"https://github.com/apache/spark/pull/15026","id":176057571,"number":15026,"title":"[SPARK-17472] [PYSPARK] Better error message for serialization failures of large objects in Python","user":{"login":"ericl","id":14922,"avatar_url":"https://avatars.githubusercontent.com/u/14922?v=3","gravatar_id":"","url":"https://api.github.com/users/ericl","html_url":"https://github.com/ericl","followers_url":"https://api.github.com/users/ericl/followers","following_url":"https://api.github.com/users/ericl/following{/other_user}","gists_url":"https://api.github.com/users/ericl/gists{/gist_id}","starred_url":"https://api.github.com/users/ericl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ericl/subscriptions","organizations_url":"https://api.github.com/users/ericl/orgs","repos_url":"https://api.github.com/users/ericl/repos","events_url":"https://api.github.com/users/ericl/events{/privacy}","received_events_url":"https://api.github.com/users/ericl/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-09-09T16:57:32Z","updated_at":"2016-09-10T01:07:43Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/15026","html_url":"https://github.com/apache/spark/pull/15026","diff_url":"https://github.com/apache/spark/pull/15026.diff","patch_url":"https://github.com/apache/spark/pull/15026.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nFor large objects, pickle does not raise useful error messages. However, we can wrap them to be slightly more user friendly:\r\n\r\nExample 1:\r\n```\r\ndef run():\r\n  import numpy.random as nr\r\n  b = nr.bytes(8 * 1000000000)\r\n  sc.parallelize(range(1000), 1000).map(lambda x: len(b)).count()\r\n\r\nrun()\r\n```\r\n\r\nBefore:\r\n```\r\nerror: 'i' format requires -2147483648 <= number <= 2147483647\r\n```\r\n\r\nAfter:\r\n```\r\npickle.PicklingError: Object too large to serialize: 'i' format requires -2147483648 <= number <= 2147483647\r\n```\r\n\r\nExample 2:\r\n```\r\ndef run():\r\n  import numpy.random as nr\r\n  b = sc.broadcast(nr.bytes(8 * 1000000000))\r\n  sc.parallelize(range(1000), 1000).map(lambda x: len(b.value)).count()\r\n\r\nrun()\r\n```\r\n\r\nBefore:\r\n```\r\nSystemError: error return without exception set\r\n```\r\n\r\nAfter:\r\n```\r\ncPickle.PicklingError: Could not serialize broadcast: SystemError: error return without exception set\r\n```\r\n\r\n## How was this patch tested?\r\n\r\nManually tried out these cases\r\n\r\ncc @davies "},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/246077816","html_url":"https://github.com/apache/spark/pull/15026#issuecomment-246077816","issue_url":"https://api.github.com/repos/apache/spark/issues/15026","id":246077816,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-09-10T01:07:43Z","updated_at":"2016-09-10T01:07:43Z","body":"**[Test build #65184 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65184/consoleFull)** for PR 15026 at commit [`fdaed3f`](https://github.com/apache/spark/commit/fdaed3fdfa5d47a3cba03504a82ce94b26b7eae6)."}}
{"actor":{"display_login":"SparkQA"}}