{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/212476268","pull_request_review_id":149120864,"id":212476268,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIxMjQ3NjI2OA==","diff_hunk":"@@ -31,6 +32,7 @@ import org.apache.spark.sql.types.StructType\n class ParquetSchemaPruningSuite\n     extends QueryTest\n     with ParquetTest\n+    with SchemaPruningTest","path":"sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruningSuite.scala","position":null,"original_position":12,"commit_id":"e6baf681e06e229d740af120491d1bf0f426af99","original_commit_id":"e6baf681e06e229d740af120491d1bf0f426af99","user":{"login":"gatorsmile","id":11567269,"node_id":"MDQ6VXNlcjExNTY3MjY5","avatar_url":"https://avatars1.githubusercontent.com/u/11567269?v=4","gravatar_id":"","url":"https://api.github.com/users/gatorsmile","html_url":"https://github.com/gatorsmile","followers_url":"https://api.github.com/users/gatorsmile/followers","following_url":"https://api.github.com/users/gatorsmile/following{/other_user}","gists_url":"https://api.github.com/users/gatorsmile/gists{/gist_id}","starred_url":"https://api.github.com/users/gatorsmile/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gatorsmile/subscriptions","organizations_url":"https://api.github.com/users/gatorsmile/orgs","repos_url":"https://api.github.com/users/gatorsmile/repos","events_url":"https://api.github.com/users/gatorsmile/events{/privacy}","received_events_url":"https://api.github.com/users/gatorsmile/received_events","type":"User","site_admin":false},"body":"Why this is removed?","created_at":"2018-08-23T22:28:57Z","updated_at":"2018-08-23T22:28:57Z","html_url":"https://github.com/apache/spark/pull/21320#discussion_r212476268","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/21320","author_association":"MEMBER","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/212476268"},"html":{"href":"https://github.com/apache/spark/pull/21320#discussion_r212476268"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/21320"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/21320","id":187775702,"node_id":"MDExOlB1bGxSZXF1ZXN0MTg3Nzc1NzAy","html_url":"https://github.com/apache/spark/pull/21320","diff_url":"https://github.com/apache/spark/pull/21320.diff","patch_url":"https://github.com/apache/spark/pull/21320.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/21320","number":21320,"state":"open","locked":false,"title":"[SPARK-4502][SQL] Parquet nested column pruning - foundation","user":{"login":"mallman","id":833693,"node_id":"MDQ6VXNlcjgzMzY5Mw==","avatar_url":"https://avatars3.githubusercontent.com/u/833693?v=4","gravatar_id":"","url":"https://api.github.com/users/mallman","html_url":"https://github.com/mallman","followers_url":"https://api.github.com/users/mallman/followers","following_url":"https://api.github.com/users/mallman/following{/other_user}","gists_url":"https://api.github.com/users/mallman/gists{/gist_id}","starred_url":"https://api.github.com/users/mallman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mallman/subscriptions","organizations_url":"https://api.github.com/users/mallman/orgs","repos_url":"https://api.github.com/users/mallman/repos","events_url":"https://api.github.com/users/mallman/events{/privacy}","received_events_url":"https://api.github.com/users/mallman/received_events","type":"User","site_admin":false},"body":"(Link to Jira: https://issues.apache.org/jira/browse/SPARK-4502)\r\n\r\n_N.B. This is a restart of PR #16578 which includes a subset of that code. Relevant review comments from that PR should be considered incorporated by reference. Please avoid duplication in review by reviewing that PR first. The summary below is an edited copy of the summary of the previous PR._\r\n\r\n## What changes were proposed in this pull request?\r\n\r\nOne of the hallmarks of a column-oriented data storage format is the ability to read data from a subset of columns, efficiently skipping reads from other columns. Spark has long had support for pruning unneeded top-level schema fields from the scan of a parquet file. For example, consider a table, `contacts`, backed by parquet with the following Spark SQL schema:\r\n\r\n```\r\nroot\r\n |-- name: struct\r\n |    |-- first: string\r\n |    |-- last: string\r\n |-- address: string\r\n```\r\n\r\nParquet stores this table's data in three physical columns: `name.first`, `name.last` and `address`. To answer the query\r\n\r\n```SQL\r\nselect address from contacts\r\n```\r\n\r\nSpark will read only from the `address` column of parquet data. However, to answer the query\r\n\r\n```SQL\r\nselect name.first from contacts\r\n```\r\n\r\nSpark will read `name.first` and `name.last` from parquet.\r\n\r\nThis PR modifies Spark SQL to support a finer-grain of schema pruning. With this patch, Spark reads only the `name.first` column to answer the previous query.\r\n\r\n### Implementation\r\n\r\nThere are two main components of this patch. First, there is a `ParquetSchemaPruning` optimizer rule for gathering the required schema fields of a `PhysicalOperation` over a parquet file, constructing a new schema based on those required fields and rewriting the plan in terms of that pruned schema. The pruned schema fields are pushed down to the parquet requested read schema. `ParquetSchemaPruning` uses a new `ProjectionOverSchema` extractor for rewriting a catalyst expression in terms of a pruned schema.\r\n\r\nSecond, the `ParquetRowConverter` has been patched to ensure the ordinals of the parquet columns read are correct for the pruned schema. `ParquetReadSupport` has been patched to address a compatibility mismatch between Spark's built in vectorized reader and the parquet-mr library's reader.\r\n\r\n### Limitation\r\n\r\nAmong the complex Spark SQL data types, this patch supports parquet column pruning of nested sequences of struct fields only.\r\n\r\n## How was this patch tested?\r\n\r\nCare has been taken to ensure correctness and prevent regressions. A more advanced version of this patch incorporating optimizations for rewriting queries involving aggregations and joins has been running on a production Spark cluster at VideoAmp for several years. In that time, one bug was found and fixed early on, and we added a regression test for that bug.\r\n\r\nWe forward-ported this patch to Spark master in June 2016 and have been running this patch against Spark 2.x branches on ad-hoc clusters since then.","created_at":"2018-05-14T09:59:33Z","updated_at":"2018-08-23T22:28:57Z","closed_at":null,"merged_at":null,"merge_commit_sha":"9c7803aab37646e7bbcaff5118b14c43fd411a8f","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/21320/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/21320/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/21320/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/e6baf681e06e229d740af120491d1bf0f426af99","head":{"label":"VideoAmp:spark-4502-parquet_column_pruning-foundation","ref":"spark-4502-parquet_column_pruning-foundation","sha":"e6baf681e06e229d740af120491d1bf0f426af99","user":{"login":"VideoAmp","id":7621972,"node_id":"MDEyOk9yZ2FuaXphdGlvbjc2MjE5NzI=","avatar_url":"https://avatars0.githubusercontent.com/u/7621972?v=4","gravatar_id":"","url":"https://api.github.com/users/VideoAmp","html_url":"https://github.com/VideoAmp","followers_url":"https://api.github.com/users/VideoAmp/followers","following_url":"https://api.github.com/users/VideoAmp/following{/other_user}","gists_url":"https://api.github.com/users/VideoAmp/gists{/gist_id}","starred_url":"https://api.github.com/users/VideoAmp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/VideoAmp/subscriptions","organizations_url":"https://api.github.com/users/VideoAmp/orgs","repos_url":"https://api.github.com/users/VideoAmp/repos","events_url":"https://api.github.com/users/VideoAmp/events{/privacy}","received_events_url":"https://api.github.com/users/VideoAmp/received_events","type":"Organization","site_admin":false},"repo":{"id":49447912,"node_id":"MDEwOlJlcG9zaXRvcnk0OTQ0NzkxMg==","name":"spark-public","full_name":"VideoAmp/spark-public","owner":{"login":"VideoAmp","id":7621972,"node_id":"MDEyOk9yZ2FuaXphdGlvbjc2MjE5NzI=","avatar_url":"https://avatars0.githubusercontent.com/u/7621972?v=4","gravatar_id":"","url":"https://api.github.com/users/VideoAmp","html_url":"https://github.com/VideoAmp","followers_url":"https://api.github.com/users/VideoAmp/followers","following_url":"https://api.github.com/users/VideoAmp/following{/other_user}","gists_url":"https://api.github.com/users/VideoAmp/gists{/gist_id}","starred_url":"https://api.github.com/users/VideoAmp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/VideoAmp/subscriptions","organizations_url":"https://api.github.com/users/VideoAmp/orgs","repos_url":"https://api.github.com/users/VideoAmp/repos","events_url":"https://api.github.com/users/VideoAmp/events{/privacy}","received_events_url":"https://api.github.com/users/VideoAmp/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/VideoAmp/spark-public","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/VideoAmp/spark-public","forks_url":"https://api.github.com/repos/VideoAmp/spark-public/forks","keys_url":"https://api.github.com/repos/VideoAmp/spark-public/keys{/key_id}","collaborators_url":"https://api.github.com/repos/VideoAmp/spark-public/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/VideoAmp/spark-public/teams","hooks_url":"https://api.github.com/repos/VideoAmp/spark-public/hooks","issue_events_url":"https://api.github.com/repos/VideoAmp/spark-public/issues/events{/number}","events_url":"https://api.github.com/repos/VideoAmp/spark-public/events","assignees_url":"https://api.github.com/repos/VideoAmp/spark-public/assignees{/user}","branches_url":"https://api.github.com/repos/VideoAmp/spark-public/branches{/branch}","tags_url":"https://api.github.com/repos/VideoAmp/spark-public/tags","blobs_url":"https://api.github.com/repos/VideoAmp/spark-public/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/VideoAmp/spark-public/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/VideoAmp/spark-public/git/refs{/sha}","trees_url":"https://api.github.com/repos/VideoAmp/spark-public/git/trees{/sha}","statuses_url":"https://api.github.com/repos/VideoAmp/spark-public/statuses/{sha}","languages_url":"https://api.github.com/repos/VideoAmp/spark-public/languages","stargazers_url":"https://api.github.com/repos/VideoAmp/spark-public/stargazers","contributors_url":"https://api.github.com/repos/VideoAmp/spark-public/contributors","subscribers_url":"https://api.github.com/repos/VideoAmp/spark-public/subscribers","subscription_url":"https://api.github.com/repos/VideoAmp/spark-public/subscription","commits_url":"https://api.github.com/repos/VideoAmp/spark-public/commits{/sha}","git_commits_url":"https://api.github.com/repos/VideoAmp/spark-public/git/commits{/sha}","comments_url":"https://api.github.com/repos/VideoAmp/spark-public/comments{/number}","issue_comment_url":"https://api.github.com/repos/VideoAmp/spark-public/issues/comments{/number}","contents_url":"https://api.github.com/repos/VideoAmp/spark-public/contents/{+path}","compare_url":"https://api.github.com/repos/VideoAmp/spark-public/compare/{base}...{head}","merges_url":"https://api.github.com/repos/VideoAmp/spark-public/merges","archive_url":"https://api.github.com/repos/VideoAmp/spark-public/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/VideoAmp/spark-public/downloads","issues_url":"https://api.github.com/repos/VideoAmp/spark-public/issues{/number}","pulls_url":"https://api.github.com/repos/VideoAmp/spark-public/pulls{/number}","milestones_url":"https://api.github.com/repos/VideoAmp/spark-public/milestones{/number}","notifications_url":"https://api.github.com/repos/VideoAmp/spark-public/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/VideoAmp/spark-public/labels{/name}","releases_url":"https://api.github.com/repos/VideoAmp/spark-public/releases{/id}","deployments_url":"https://api.github.com/repos/VideoAmp/spark-public/deployments","created_at":"2016-01-11T19:11:30Z","updated_at":"2016-01-11T19:11:55Z","pushed_at":"2018-08-23T22:25:58Z","git_url":"git://github.com/VideoAmp/spark-public.git","ssh_url":"git@github.com:VideoAmp/spark-public.git","clone_url":"https://github.com/VideoAmp/spark-public.git","svn_url":"https://github.com/VideoAmp/spark-public","homepage":null,"size":264339,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"60af2501e1afc00192c779f2736a4e3de12428fa","user":{"login":"apache","id":47359,"node_id":"MDEyOk9yZ2FuaXphdGlvbjQ3MzU5","avatar_url":"https://avatars0.githubusercontent.com/u/47359?v=4","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"node_id":"MDEwOlJlcG9zaXRvcnkxNzE2NTY1OA==","name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"node_id":"MDEyOk9yZ2FuaXphdGlvbjQ3MzU5","avatar_url":"https://avatars0.githubusercontent.com/u/47359?v=4","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","deployments_url":"https://api.github.com/repos/apache/spark/deployments","created_at":"2014-02-25T08:00:08Z","updated_at":"2018-08-23T19:18:30Z","pushed_at":"2018-08-23T22:26:04Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":280310,"stargazers_count":18494,"watchers_count":18494,"language":"Scala","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":16734,"mirror_url":null,"archived":false,"open_issues_count":486,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":16734,"open_issues":486,"watchers":18494,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/21320"},"html":{"href":"https://github.com/apache/spark/pull/21320"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/21320"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/21320/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/21320/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/21320/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/e6baf681e06e229d740af120491d1bf0f426af99"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"gatorsmile"}}