{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/21889","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/21889/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/21889/comments","events_url":"https://api.github.com/repos/apache/spark/issues/21889/events","html_url":"https://github.com/apache/spark/pull/21889","id":345049966,"node_id":"MDExOlB1bGxSZXF1ZXN0MjA0MzAxMTA2","number":21889,"title":"[SPARK-4502][SQL] Parquet nested column pruning - foundation (2nd attempt)","user":{"login":"ajacques","id":945049,"node_id":"MDQ6VXNlcjk0NTA0OQ==","avatar_url":"https://avatars2.githubusercontent.com/u/945049?v=4","gravatar_id":"","url":"https://api.github.com/users/ajacques","html_url":"https://github.com/ajacques","followers_url":"https://api.github.com/users/ajacques/followers","following_url":"https://api.github.com/users/ajacques/following{/other_user}","gists_url":"https://api.github.com/users/ajacques/gists{/gist_id}","starred_url":"https://api.github.com/users/ajacques/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ajacques/subscriptions","organizations_url":"https://api.github.com/users/ajacques/orgs","repos_url":"https://api.github.com/users/ajacques/repos","events_url":"https://api.github.com/users/ajacques/events{/privacy}","received_events_url":"https://api.github.com/users/ajacques/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-07-27T00:29:33Z","updated_at":"2018-07-27T04:56:22Z","closed_at":null,"author_association":"NONE","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/21889","html_url":"https://github.com/apache/spark/pull/21889","diff_url":"https://github.com/apache/spark/pull/21889.diff","patch_url":"https://github.com/apache/spark/pull/21889.patch"},"body":"(Link to Jira: https://issues.apache.org/jira/browse/SPARK-4502)\r\n\r\n**This is a restart of apache/spark#21320. Most of the discussion has taken place over there. I've only taken it as a based to make stylistic changes based on the code review to help move things along.**\r\n\r\nDue to the urgency of the upcoming 2.4 code freeze, I'm going to open this PR to collect any feedback. This can be closed if you prefer to continue to the work in the original PR.\r\n\r\n### What changes were proposed in this pull request?\r\nOne of the hallmarks of a column-oriented data storage format is the ability to read data from a subset of columns, efficiently skipping reads from other columns. Spark has long had support for pruning unneeded top-level schema fields from the scan of a parquet file. For example, consider a table, contacts, backed by parquet with the following Spark SQL schema:\r\n\r\n```\r\nroot\r\n |-- name: struct\r\n |    |-- first: string\r\n |    |-- last: string\r\n |-- address: string\r\n```\r\nParquet stores this table's data in three physical columns: name.first, name.last and address. To answer the query\r\n\r\n`select address from contacts`\r\nSpark will read only from the address column of parquet data. However, to answer the query\r\n\r\n`select name.first from contacts`\r\nSpark will read name.first and name.last from parquet.\r\n\r\nThis PR modifies Spark SQL to support a finer-grain of schema pruning. With this patch, Spark reads only the name.first column to answer the previous query.\r\n\r\n### Implementation\r\nThere are two main components of this patch. First, there is a ParquetSchemaPruning optimizer rule for gathering the required schema fields of a PhysicalOperation over a parquet file, constructing a new schema based on those required fields and rewriting the plan in terms of that pruned schema. The pruned schema fields are pushed down to the parquet requested read schema. ParquetSchemaPruning uses a new ProjectionOverSchema extractor for rewriting a catalyst expression in terms of a pruned schema.\r\n\r\nSecond, the ParquetRowConverter has been patched to ensure the ordinals of the parquet columns read are correct for the pruned schema. ParquetReadSupport has been patched to address a compatibility mismatch between Spark's built in vectorized reader and the parquet-mr library's reader.\r\n\r\n### Limitation\r\nAmong the complex Spark SQL data types, this patch supports parquet column pruning of nested sequences of struct fields only.\r\n\r\n### How was this patch tested?\r\nCare has been taken to ensure correctness and prevent regressions. A more advanced version of this patch incorporating optimizations for rewriting queries involving aggregations and joins has been running on a production Spark cluster at VideoAmp for several years. In that time, one bug was found and fixed early on, and we added a regression test for that bug.\r\n\r\nWe forward-ported this patch to Spark master in June 2016 and have been running this patch against Spark 2.x branches on ad-hoc clusters since then.\r\n\r\n### Known Issues\r\nHighlighted in https://github.com/apache/spark/pull/21320#issuecomment-408271470"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/408311273","html_url":"https://github.com/apache/spark/pull/21889#issuecomment-408311273","issue_url":"https://api.github.com/repos/apache/spark/issues/21889","id":408311273,"node_id":"MDEyOklzc3VlQ29tbWVudDQwODMxMTI3Mw==","user":{"login":"AmplabJenkins","id":4031291,"node_id":"MDQ6VXNlcjQwMzEyOTE=","avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2018-07-27T04:56:22Z","updated_at":"2018-07-27T04:56:22Z","author_association":"NONE","body":"Test PASSed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93646/\nTest PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}