{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/15976","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/15976/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/15976/comments","events_url":"https://api.github.com/repos/apache/spark/issues/15976/events","html_url":"https://github.com/apache/spark/pull/15976","id":190919756,"number":15976,"title":"[SPARK-18403][SQL] Fix unsafe data false sharing issue in ObjectHashAggregateExec","user":{"login":"liancheng","id":230655,"avatar_url":"https://avatars.githubusercontent.com/u/230655?v=3","gravatar_id":"","url":"https://api.github.com/users/liancheng","html_url":"https://github.com/liancheng","followers_url":"https://api.github.com/users/liancheng/followers","following_url":"https://api.github.com/users/liancheng/following{/other_user}","gists_url":"https://api.github.com/users/liancheng/gists{/gist_id}","starred_url":"https://api.github.com/users/liancheng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liancheng/subscriptions","organizations_url":"https://api.github.com/users/liancheng/orgs","repos_url":"https://api.github.com/users/liancheng/repos","events_url":"https://api.github.com/users/liancheng/events{/privacy}","received_events_url":"https://api.github.com/users/liancheng/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":13,"created_at":"2016-11-22T06:40:21Z","updated_at":"2016-11-28T08:05:50Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/15976","html_url":"https://github.com/apache/spark/pull/15976","diff_url":"https://github.com/apache/spark/pull/15976.diff","patch_url":"https://github.com/apache/spark/pull/15976.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThis PR fixes a random OOM issue occurred while running `ObjectHashAggregateSuite`.\r\n\r\nThis issue can be steadily reproduced under the following conditions:\r\n\r\n1. The aggregation must be evaluated using `ObjectHashAggregateExec`;\r\n2. There must be an input column whose data type involves `ArrayType` (an input column of `MapType` may even cause SIGSEGV);\r\n3. Sort-based aggregation fallback must be triggered during evaluation.\r\n\r\nThe root cause is that while falling back to sort-based aggregation, we must sort and feed already evaluated partial aggregation buffers living in the hash map to the sort-based aggregator using an external sorter. However, the underlying mutable byte buffer of `UnsafeRow`s produced by the iterator of the external sorter is reused and may get overwritten when the iterator steps forward. After the last entry is consumed, the byte buffer points to a block of uninitialized memory filled by `5a`. Therefore, while reading an `UnsafeArrayData` out of the `UnsafeRow`, `5a5a5a5a` is treated as array size and triggers a memory allocation for a ridiculously large array and immediately blows up the JVM with an OOM.\r\n\r\nTo fix this issue, we only need to add `.copy()` accordingly.\r\n\r\n## How was this patch tested?\r\n\r\nNew regression test case added in `ObjectHashAggregateSuite`."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/263206743","html_url":"https://github.com/apache/spark/pull/15976#issuecomment-263206743","issue_url":"https://api.github.com/repos/apache/spark/issues/15976","id":263206743,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-11-28T08:05:50Z","updated_at":"2016-11-28T08:05:50Z","body":"Merged build finished. Test FAILed."}}
{"actor":{"display_login":"AmplabJenkins"}}