{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/13680","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/13680/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/13680/comments","events_url":"https://api.github.com/repos/apache/spark/issues/13680/events","html_url":"https://github.com/apache/spark/pull/13680","id":160346862,"number":13680,"title":"[SPARK-15962][SQL] Introduce additonal implementation with a dense format for UnsafeArrayData","user":{"login":"kiszk","id":1315079,"avatar_url":"https://avatars.githubusercontent.com/u/1315079?v=3","gravatar_id":"","url":"https://api.github.com/users/kiszk","html_url":"https://github.com/kiszk","followers_url":"https://api.github.com/users/kiszk/followers","following_url":"https://api.github.com/users/kiszk/following{/other_user}","gists_url":"https://api.github.com/users/kiszk/gists{/gist_id}","starred_url":"https://api.github.com/users/kiszk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kiszk/subscriptions","organizations_url":"https://api.github.com/users/kiszk/orgs","repos_url":"https://api.github.com/users/kiszk/repos","events_url":"https://api.github.com/users/kiszk/events{/privacy}","received_events_url":"https://api.github.com/users/kiszk/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2016-06-15T06:22:50Z","updated_at":"2016-06-23T05:09:50Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/13680","html_url":"https://github.com/apache/spark/pull/13680","diff_url":"https://github.com/apache/spark/pull/13680.diff","patch_url":"https://github.com/apache/spark/pull/13680.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThis PR introduces two implementations for ```UnsafeArrayData```. One is an sparse array ```UnsafeArrayDataSparse``` that is the original format of ```UnsafeArrayData```. The other is a dense array ```UnsafeArrayDataDense``` that is a new format of ```UnsafeArrayData```.\r\n\r\n```UnsafeArrayDataSparse``` accepts ```null``` value in each entry of an array. However, it increases memory footprint (e.g. 2x for an integer array) due to existence of [```offsets``` area](https://github.com/apache/spark/blob/43b149fb885a27f9467aab28e5195f6f03aadcf0/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java#L35). In some cases (e.g. an unsafe array is created from an primitive array by a method ```fromPrimitiveArray```, we know there is no ```null``` value in an array. ```UnsafeArrayDataDense``` can reduce memory footprint since it does not have ```offsets``` area.\r\n\r\nExample of memory foot comparision:\r\n1024x1024 elements integer array\r\nSize of ```baseObject``` for ```UnsafeArrayDataSparse```: 4 + 1024x1024 + 1024x1024 = 2M bytes\r\nSize of ```baseObject``` for ```UnsafeArrayDataDense```: 4 + 4 + 1024x1024 = 1M bytes\r\n\r\n## How was this patch tested?\r\n\r\nTested by existing unit tests\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/227951331","html_url":"https://github.com/apache/spark/pull/13680#issuecomment-227951331","issue_url":"https://api.github.com/repos/apache/spark/issues/13680","id":227951331,"user":{"login":"kiszk","id":1315079,"avatar_url":"https://avatars.githubusercontent.com/u/1315079?v=3","gravatar_id":"","url":"https://api.github.com/users/kiszk","html_url":"https://github.com/kiszk","followers_url":"https://api.github.com/users/kiszk/followers","following_url":"https://api.github.com/users/kiszk/following{/other_user}","gists_url":"https://api.github.com/users/kiszk/gists{/gist_id}","starred_url":"https://api.github.com/users/kiszk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kiszk/subscriptions","organizations_url":"https://api.github.com/users/kiszk/orgs","repos_url":"https://api.github.com/users/kiszk/repos","events_url":"https://api.github.com/users/kiszk/events{/privacy}","received_events_url":"https://api.github.com/users/kiszk/received_events","type":"User","site_admin":false},"created_at":"2016-06-23T05:09:50Z","updated_at":"2016-06-23T05:09:50Z","body":"@cloud-fan thank you for your good comment. I also read [previous proposal](https://github.com/apache/spark/pull/12640#discussion_r61539393).\r\nI love to have only single format (or implementation). Since I thought that there are some reasons to keep the old format, I introduced a new dense format.\r\n\r\nIMHO, a new unified format should have three properties.\r\n1. Remove indirect offset (for performance and footprint)\r\n2. Have capability of presence of nullbit (for generality)\r\n3. Quickly get information on existence of null value in an array (for performance, in particular, primitive array)\r\n\r\nBased on them, how about this single format?\r\n```\r\n[numElements] [all zero in null bits?] [null bits] [values] [variable length portion]\r\n``` \r\nIf we want to reduce memory footprint in the case of primitive array, we can drop ```[null bits]``` part if ```[all zero in null bits?]``` has a special value."}}
{"actor":{"display_login":"kiszk"}}