{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/14091","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/14091/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/14091/comments","events_url":"https://api.github.com/repos/apache/spark/issues/14091/events","html_url":"https://github.com/apache/spark/pull/14091","id":164318693,"number":14091,"title":"[SPARK-16412][SQL] Generate Java code that gets an array in each column of CachedBatch when DataFrame.cache() is called","user":{"login":"kiszk","id":1315079,"avatar_url":"https://avatars.githubusercontent.com/u/1315079?v=3","gravatar_id":"","url":"https://api.github.com/users/kiszk","html_url":"https://github.com/kiszk","followers_url":"https://api.github.com/users/kiszk/followers","following_url":"https://api.github.com/users/kiszk/following{/other_user}","gists_url":"https://api.github.com/users/kiszk/gists{/gist_id}","starred_url":"https://api.github.com/users/kiszk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kiszk/subscriptions","organizations_url":"https://api.github.com/users/kiszk/orgs","repos_url":"https://api.github.com/users/kiszk/repos","events_url":"https://api.github.com/users/kiszk/events{/privacy}","received_events_url":"https://api.github.com/users/kiszk/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-07-07T14:12:28Z","updated_at":"2016-07-07T14:21:51Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/14091","html_url":"https://github.com/apache/spark/pull/14091","diff_url":"https://github.com/apache/spark/pull/14091.diff","patch_url":"https://github.com/apache/spark/pull/14091.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nWaiting #11956 to be merged.\r\n\r\nThis PR generates Java code to directly get an array of each column from `CachedBatch` when `DataFrame.cache()` is called. This is done in whole stage code generation.\r\n\r\nWhen DataFrame.cache()` is called, data is stored as column-oriented storage (columnar cache) in `CachedBatch`. This PR avoid conversion from column-oriented storage to row-oriented storage. This PR handles an array type that is stored into a column. \r\n\r\nThis PR generates code both for row-oriented storage and column-oriented storage only if\r\n - `InMemoryColumnarTableScan` exists in a plan sub-tree. A decision is performed by checking an given iterator is `ColumnaIterator` at runtime\r\n - Sort or join does not exist in a plan sub-tree. \r\n\r\nThis PR generates Java code for columnar cache only if types in all columns, which are accessed in operations, are primitive or an array\r\n\r\nI will add benchmark suites into  [here](https://github.com/kiszk/spark/blob/SPARK-14098/sql/core/src/test/scala/org/apache/spark/sql/DataFrameCacheBenchmark.scala)\r\n\r\n\r\n\r\n## How was this patch tested?\r\n\r\nAdded new tests into `DataFrameCacheSuite.scala`\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/231092385","html_url":"https://github.com/apache/spark/pull/14091#issuecomment-231092385","issue_url":"https://api.github.com/repos/apache/spark/issues/14091","id":231092385,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-07-07T14:21:51Z","updated_at":"2016-07-07T14:21:51Z","body":"Test FAILed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61913/\nTest FAILed."}}
{"actor":{"display_login":"AmplabJenkins"}}