{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/13065","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/13065/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/13065/comments","events_url":"https://api.github.com/repos/apache/spark/issues/13065/events","html_url":"https://github.com/apache/spark/pull/13065","id":154358883,"number":13065,"title":"[SPARK-15214][SQL] Code-generation for Generate","user":{"login":"hvanhovell","id":9616802,"avatar_url":"https://avatars.githubusercontent.com/u/9616802?v=3","gravatar_id":"","url":"https://api.github.com/users/hvanhovell","html_url":"https://github.com/hvanhovell","followers_url":"https://api.github.com/users/hvanhovell/followers","following_url":"https://api.github.com/users/hvanhovell/following{/other_user}","gists_url":"https://api.github.com/users/hvanhovell/gists{/gist_id}","starred_url":"https://api.github.com/users/hvanhovell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hvanhovell/subscriptions","organizations_url":"https://api.github.com/users/hvanhovell/orgs","repos_url":"https://api.github.com/users/hvanhovell/repos","events_url":"https://api.github.com/users/hvanhovell/events{/privacy}","received_events_url":"https://api.github.com/users/hvanhovell/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":22,"created_at":"2016-05-11T22:38:27Z","updated_at":"2016-08-30T14:48:46Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/13065","html_url":"https://github.com/apache/spark/pull/13065","diff_url":"https://github.com/apache/spark/pull/13065.diff","patch_url":"https://github.com/apache/spark/pull/13065.patch"},"body":"## What changes were proposed in this pull request?\r\nThis PR adds code generation to `Generate`. It supports two code paths:\r\n- General `TraversableOnce` based iteration. This used for regular `Generator` (code generation supporting) expressions. This code path expects the expression to return a `TraversableOnce[InternalRow]` and it will iterate over the returned collection. This PR adds code generation for the `json_tuple` generator.\r\n- Specialized `ArrayData/MapData` based iteration. This is used for the `explode` function and operates directly on the `ArrayData`/`MapData` result that the child of the `explode` expression returns.\r\n\r\nI have added some benchmarks and it seems we can create a nice speedup for explode:\r\n##### explode(array)\r\n```\r\nJava HotSpot(TM) 64-Bit Server VM 1.8.0_92-b14 on Mac OS X 10.11.4\r\nIntel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz\r\n\r\ngenerate explode array:                  Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\r\n------------------------------------------------------------------------------------------------\r\ngenerate explode array wholestage off         7095 / 7331          2.4         422.9       1.0X\r\ngenerate explode array wholestage on           631 /  641         26.6          37.6      11.2X\r\n```\r\n##### explode(map)\r\n```\r\nJava HotSpot(TM) 64-Bit Server VM 1.8.0_92-b14 on Mac OS X 10.11.4\r\nIntel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz\r\n\r\ngenerate explode map:                    Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\r\n------------------------------------------------------------------------------------------------\r\ngenerate explode map wholestage off         12856 / 13569          1.3         766.3       1.0X\r\ngenerate explode map wholestage on             865 /  873         19.4          51.6      14.9X\r\n```\r\n##### json_tuple\r\nThis isn't actually that interesting because json parsing dominates the runtime. It is however the only native generator and the only way to test this code path.\r\n```\r\nJava HotSpot(TM) 64-Bit Server VM 1.8.0_92-b14 on Mac OS X 10.11.4\r\nIntel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz\r\n\r\ngenerate json_tuple:                     Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative\r\n------------------------------------------------------------------------------------------------\r\ngenerate json_tuple wholestage off          13097 / 13323          1.3         780.6       1.0X\r\ngenerate json_tuple wholestage on           11815 / 11899          1.4         704.2       1.1X\r\n```\r\n## How was this patch tested?\r\nExisting tests.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/243465185","html_url":"https://github.com/apache/spark/pull/13065#issuecomment-243465185","issue_url":"https://api.github.com/repos/apache/spark/issues/13065","id":243465185,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-08-30T14:48:46Z","updated_at":"2016-08-30T14:48:46Z","body":"Merged build finished. Test PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}