{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/21447","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/21447/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/21447/comments","events_url":"https://api.github.com/repos/apache/spark/issues/21447/events","html_url":"https://github.com/apache/spark/pull/21447","id":327179709,"node_id":"MDExOlB1bGxSZXF1ZXN0MTkxMDM5OTcx","number":21447,"title":"[SPARK-24339][SQL]Add project for transform/map/reduce sql to prune column","user":{"login":"xdcjie","id":6291466,"node_id":"MDQ6VXNlcjYyOTE0NjY=","avatar_url":"https://avatars2.githubusercontent.com/u/6291466?v=4","gravatar_id":"","url":"https://api.github.com/users/xdcjie","html_url":"https://github.com/xdcjie","followers_url":"https://api.github.com/users/xdcjie/followers","following_url":"https://api.github.com/users/xdcjie/following{/other_user}","gists_url":"https://api.github.com/users/xdcjie/gists{/gist_id}","starred_url":"https://api.github.com/users/xdcjie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xdcjie/subscriptions","organizations_url":"https://api.github.com/users/xdcjie/orgs","repos_url":"https://api.github.com/users/xdcjie/repos","events_url":"https://api.github.com/users/xdcjie/events{/privacy}","received_events_url":"https://api.github.com/users/xdcjie/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":18,"created_at":"2018-05-29T06:00:40Z","updated_at":"2018-07-18T19:03:09Z","closed_at":"2018-07-17T08:09:59Z","author_association":"NONE","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/21447","html_url":"https://github.com/apache/spark/pull/21447","diff_url":"https://github.com/apache/spark/pull/21447.diff","patch_url":"https://github.com/apache/spark/pull/21447.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nTransform query do not have Project Node, query like:\r\n\r\n` select transform(a, b) using 'func' from e` \r\n\r\nand it' logic plan is:\r\n\r\n> 'ScriptTransformation['a, 'b], func, [key#0, value#0] \r\n        +- 'UnresolvedRelation `e` (PlanTest.scala:97)\r\n\r\nso that it will scan all the column data of relation.\r\n\r\nIn this PR, I propose to add Project Node for transform query, so that it scan required data by prune columns, for above transform query,  it's logic plan will be:\r\n\r\n> 'ScriptTransformation['a, 'b], func, [key#0, value#0] \r\n   +- 'Project ['a, 'b]    \r\n        +- 'UnresolvedRelation `e`\r\n\r\nit will scan only two column data of relation.\r\n\r\nIn summary, Add Project Node for transform query can reduce the time of scan and assemble data.(In our scenario, the relation(table) have 700 columns.)\r\n\r\n## How was this patch tested?\r\n\r\nModify existing test (\"transform query spec\")\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/406040064","html_url":"https://github.com/apache/spark/pull/21447#issuecomment-406040064","issue_url":"https://api.github.com/repos/apache/spark/issues/21447","id":406040064,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNjA0MDA2NA==","user":{"login":"gatorsmile","id":11567269,"node_id":"MDQ6VXNlcjExNTY3MjY5","avatar_url":"https://avatars1.githubusercontent.com/u/11567269?v=4","gravatar_id":"","url":"https://api.github.com/users/gatorsmile","html_url":"https://github.com/gatorsmile","followers_url":"https://api.github.com/users/gatorsmile/followers","following_url":"https://api.github.com/users/gatorsmile/following{/other_user}","gists_url":"https://api.github.com/users/gatorsmile/gists{/gist_id}","starred_url":"https://api.github.com/users/gatorsmile/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gatorsmile/subscriptions","organizations_url":"https://api.github.com/users/gatorsmile/orgs","repos_url":"https://api.github.com/users/gatorsmile/repos","events_url":"https://api.github.com/users/gatorsmile/events{/privacy}","received_events_url":"https://api.github.com/users/gatorsmile/received_events","type":"User","site_admin":false},"created_at":"2018-07-18T19:03:09Z","updated_at":"2018-07-18T19:03:09Z","author_association":"MEMBER","body":"@maropu Do you want to take this over and add such a project in `ColumnPruning`?  "}}
{"actor":{"display_login":"gatorsmile"}}