{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/13758","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/13758/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/13758/comments","events_url":"https://api.github.com/repos/apache/spark/issues/13758/events","html_url":"https://github.com/apache/spark/pull/13758","id":161010843,"number":13758,"title":"[SPARK-16043][SQL] Prepare GenericArrayData implementation specialized for a primitive array","user":{"login":"kiszk","id":1315079,"avatar_url":"https://avatars.githubusercontent.com/u/1315079?v=3","gravatar_id":"","url":"https://api.github.com/users/kiszk","html_url":"https://github.com/kiszk","followers_url":"https://api.github.com/users/kiszk/followers","following_url":"https://api.github.com/users/kiszk/following{/other_user}","gists_url":"https://api.github.com/users/kiszk/gists{/gist_id}","starred_url":"https://api.github.com/users/kiszk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kiszk/subscriptions","organizations_url":"https://api.github.com/users/kiszk/orgs","repos_url":"https://api.github.com/users/kiszk/repos","events_url":"https://api.github.com/users/kiszk/events{/privacy}","received_events_url":"https://api.github.com/users/kiszk/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":35,"created_at":"2016-06-18T06:56:25Z","updated_at":"2016-06-26T17:10:42Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/13758","html_url":"https://github.com/apache/spark/pull/13758","diff_url":"https://github.com/apache/spark/pull/13758.diff","patch_url":"https://github.com/apache/spark/pull/13758.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThis PR addresses a ToDo of ```GenericArrayData``` class. Current implementation of ```GenericArrayData``` leads to boxing/unboxing if type of array elements are primitive. It would be good to eliminate boxing/unboxing from the view of runtime memory footprint and performance.\r\n\r\nThis PR eliminattes boxing/unboxing by preparing sub classes of ```GenericArrayData``` to specialize operations in these classes. This PR prepare a new method ```GenericArrayData.allocate(...) ``` that can return generic ```GenericArrayData``` instance or a ```Generic<Type>ArrayData``` instance specialized for ```<Type>``` type array.\r\n\r\nHere are major improvements:\r\n1. Hold an array in a primitive array (previously ```Object[]``` is used and boxing happened in a constructor)\r\n2. a method \"get```<Type>```()\" gets a value from an primitive array (previously unboxing happened)\r\n3. a method \"to```<Type>```Array\" performs data copy using ```System.arraycopy``` (previously unboxing happened)\r\n\r\nHere are performance results of micro benchmarks\r\n```\r\nRunning benchmark: Allocate GenericArrayData for int\r\nOpenJDK 64-Bit Server VM 1.8.0_91-b14 on Linux 4.0.4-301.fc22.x86_64\r\nIntel Xeon E3-12xx v2 (Ivy Bridge)\r\nAllocate GenericArrayData for int:       Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\r\n------------------------------------------------------------------------------------------------\r\nSparse                                          40 /   43        522.2           1.9       1.0X\r\nDense                                            0 /    0  209715200.0           0.0  401598.7X\r\n\r\nRunning benchmark: Allocate GenericArrayData for double\r\nOpenJDK 64-Bit Server VM 1.8.0_91-b14 on Linux 4.0.4-301.fc22.x86_64\r\nIntel Xeon E3-12xx v2 (Ivy Bridge)\r\nAllocate GenericArrayData for double:    Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\r\n------------------------------------------------------------------------------------------------\r\nSparse                                          40 /   44        523.2           1.9       1.0X\r\nDense                                            0 /    0  225500215.1           0.0  431013.0X\r\n\r\nRunning benchmark: Get int primitive array\r\nOpenJDK 64-Bit Server VM 1.8.0_91-b14 on Linux 4.0.4-301.fc22.x86_64\r\nIntel Xeon E3-12xx v2 (Ivy Bridge)\r\nGet int primitive array:                 Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\r\n------------------------------------------------------------------------------------------------\r\nSparse int                                      67 /   70        783.9           1.3       1.0X\r\nDense  int                                      41 /   43       1263.8           0.8       1.6X\r\n\r\nRunning benchmark: Get double primitive array\r\nOpenJDK 64-Bit Server VM 1.8.0_91-b14 on Linux 4.0.4-301.fc22.x86_64\r\nIntel Xeon E3-12xx v2 (Ivy Bridge)\r\nGet double primitive array:              Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\r\n------------------------------------------------------------------------------------------------\r\nSparse double                                  211 /  217        248.6           4.0       1.0X\r\nDense  double                                   95 /  100        554.1           1.8       2.2X\r\n\r\nRunning benchmark: get primitive array\r\nOpenJDK 64-Bit Server VM 1.8.0_91-b14 on Linux 4.0.4-301.fc22.x86_64\r\nIntel Xeon E3-12xx v2 (Ivy Bridge)\r\nRead GenericArrayData Int:               Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\r\n------------------------------------------------------------------------------------------------\r\nSparse                                         160 /  163       1314.5           0.8       1.0X\r\nDense                                           68 /   69       3080.0           0.3       2.3X\r\n\r\nRunning benchmark: Read GenericArrayData Double\r\nOpenJDK 64-Bit Server VM 1.8.0_91-b14 on Linux 4.0.4-301.fc22.x86_64\r\nIntel Xeon E3-12xx v2 (Ivy Bridge)\r\nRead GenericArrayData Double:            Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\r\n------------------------------------------------------------------------------------------------\r\nSparse                                         611 /  613        343.3           2.9       1.0X\r\nDense                                          199 /  202       1051.5           1.0       3.1X\r\n```\r\n\r\n## How was this patch tested?\r\n\r\n(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)\r\n\r\nadd unit tests"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/228611414","html_url":"https://github.com/apache/spark/pull/13758#issuecomment-228611414","issue_url":"https://api.github.com/repos/apache/spark/issues/13758","id":228611414,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-06-26T17:10:41Z","updated_at":"2016-06-26T17:10:41Z","body":"Test FAILed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61270/\nTest FAILed."}}
{"actor":{"display_login":"AmplabJenkins"}}