{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/14450","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/14450/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/14450/comments","events_url":"https://api.github.com/repos/apache/spark/issues/14450/events","html_url":"https://github.com/apache/spark/pull/14450","id":168797957,"number":14450,"title":"[SPARK-16847][SQL] Prevent to potentially read corrupt statstics on binary in Parquet vectorized reader","user":{"login":"HyukjinKwon","id":6477701,"avatar_url":"https://avatars.githubusercontent.com/u/6477701?v=3","gravatar_id":"","url":"https://api.github.com/users/HyukjinKwon","html_url":"https://github.com/HyukjinKwon","followers_url":"https://api.github.com/users/HyukjinKwon/followers","following_url":"https://api.github.com/users/HyukjinKwon/following{/other_user}","gists_url":"https://api.github.com/users/HyukjinKwon/gists{/gist_id}","starred_url":"https://api.github.com/users/HyukjinKwon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HyukjinKwon/subscriptions","organizations_url":"https://api.github.com/users/HyukjinKwon/orgs","repos_url":"https://api.github.com/users/HyukjinKwon/repos","events_url":"https://api.github.com/users/HyukjinKwon/events{/privacy}","received_events_url":"https://api.github.com/users/HyukjinKwon/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2016-08-02T04:40:46Z","updated_at":"2016-08-03T02:28:38Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/14450","html_url":"https://github.com/apache/spark/pull/14450","diff_url":"https://github.com/apache/spark/pull/14450.diff","patch_url":"https://github.com/apache/spark/pull/14450.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThis problem was found in [PARQUET-251](https://issues.apache.org/jira/browse/PARQUET-251) and we disabled filter pushdown on binary columns in Spark before. We enabled this after upgrading Parquet but it seems there is potential incompatibility for Parquet files written in lower Spark versions.\r\n\r\nCurrently, this does not happen in normal Parquet reader. However, In Spark, we implemented a vectorized reader, separately with Parquet's standard API. For normal Parquet reader this is being handled but not in the vectorized reader.\r\n\r\nIt is okay to just pass `FileMetaData`. This is being handled in parquet-mr (See https://github.com/apache/parquet-mr/commit/e3b95020f777eb5e0651977f654c1662e3ea1f29). This will prevent loading corrupt statistics in each page in Parquet.\r\n\r\nThis PR replaces the deprecated usage of constructor.\r\n\r\n## How was this patch tested?\r\n\r\nN/A\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/237115702","html_url":"https://github.com/apache/spark/pull/14450#issuecomment-237115702","issue_url":"https://api.github.com/repos/apache/spark/issues/14450","id":237115702,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-08-03T02:28:38Z","updated_at":"2016-08-03T02:28:38Z","body":"Test PASSed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63145/\nTest PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}