{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/15722","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/15722/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/15722/comments","events_url":"https://api.github.com/repos/apache/spark/issues/15722/events","html_url":"https://github.com/apache/spark/pull/15722","id":186670374,"number":15722,"title":"[SPARK-18208] [Shuffle] Executor OOM due to a growing LongArray in BytesToBytesMap","user":{"login":"jiexiong","id":6044805,"avatar_url":"https://avatars.githubusercontent.com/u/6044805?v=3","gravatar_id":"","url":"https://api.github.com/users/jiexiong","html_url":"https://github.com/jiexiong","followers_url":"https://api.github.com/users/jiexiong/followers","following_url":"https://api.github.com/users/jiexiong/following{/other_user}","gists_url":"https://api.github.com/users/jiexiong/gists{/gist_id}","starred_url":"https://api.github.com/users/jiexiong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jiexiong/subscriptions","organizations_url":"https://api.github.com/users/jiexiong/orgs","repos_url":"https://api.github.com/users/jiexiong/repos","events_url":"https://api.github.com/users/jiexiong/events{/privacy}","received_events_url":"https://api.github.com/users/jiexiong/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":17,"created_at":"2016-11-01T22:57:31Z","updated_at":"2016-11-08T15:13:32Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/15722","html_url":"https://github.com/apache/spark/pull/15722","diff_url":"https://github.com/apache/spark/pull/15722.diff","patch_url":"https://github.com/apache/spark/pull/15722.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nFixed the OOM problem in BytesToBytesMap. We have a production query which joins two large tables. It failed because of \"No enough memory for aggregation\". This is because when we reset BytesToBytesMap after spilling, we did not reset longArray. This change would reset the longArray to its initial capacity whenever the map is reset.\r\n\r\n## How was this patch tested?\r\n\r\n build the package and submit a query to spark-shell with the new package. The query that failed before succeeded now. I also tested some other query who suffer from the same OOM problem and it works after the fix. \r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/259162990","html_url":"https://github.com/apache/spark/pull/15722#issuecomment-259162990","issue_url":"https://api.github.com/repos/apache/spark/issues/15722","id":259162990,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-11-08T15:13:32Z","updated_at":"2016-11-08T15:13:32Z","body":"**[Test build #68338 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68338/consoleFull)** for PR 15722 at commit [`7a95697`](https://github.com/apache/spark/commit/7a956978189953590fcfd482ff96083d2cf91383).\n * This patch passes all tests.\n * This patch merges cleanly.\n * This patch adds no public classes."}}
{"actor":{"display_login":"SparkQA"}}