{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/90972771","pull_request_review_id":11486072,"id":90972771,"diff_hunk":"@@ -107,7 +107,16 @@ public void initialize(InputSplit inputSplit, TaskAttemptContext taskAttemptCont\n       footer = readFooter(configuration, file, range(split.getStart(), split.getEnd()));\n       MessageType fileSchema = footer.getFileMetaData().getSchema();\n       FilterCompat.Filter filter = getFilter(configuration);\n-      blocks = filterRowGroups(filter, footer.getBlocks(), fileSchema);\n+      try {\n+        blocks = filterRowGroups(filter, footer.getBlocks(), fileSchema);\n+      } catch (IllegalArgumentException e) {\n+        // In the case where a particular parquet files does not contain\n+        // the column(s) in the filter, we don't do filtering at this level\n+        // PARQUET-389 will resolve this issue in Parquet 1.9, which may be used\n+        // by future Spark versions. This is a workaround for current Spark version.\n+        // Also the assumption here is that the predicates will be applied later","path":"sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java","position":12,"original_position":12,"commit_id":"096ab18887c40761eb7ba79e9c406fe8ca6ce7c0","original_commit_id":"096ab18887c40761eb7ba79e9c406fe8ca6ce7c0","user":{"login":"dongjoon-hyun","id":9700541,"avatar_url":"https://avatars.githubusercontent.com/u/9700541?v=3","gravatar_id":"","url":"https://api.github.com/users/dongjoon-hyun","html_url":"https://github.com/dongjoon-hyun","followers_url":"https://api.github.com/users/dongjoon-hyun/followers","following_url":"https://api.github.com/users/dongjoon-hyun/following{/other_user}","gists_url":"https://api.github.com/users/dongjoon-hyun/gists{/gist_id}","starred_url":"https://api.github.com/users/dongjoon-hyun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dongjoon-hyun/subscriptions","organizations_url":"https://api.github.com/users/dongjoon-hyun/orgs","repos_url":"https://api.github.com/users/dongjoon-hyun/repos","events_url":"https://api.github.com/users/dongjoon-hyun/events{/privacy}","received_events_url":"https://api.github.com/users/dongjoon-hyun/received_events","type":"User","site_admin":false},"body":"Thank you, @xwu0226 !","created_at":"2016-12-05T22:40:58Z","updated_at":"2016-12-05T22:40:58Z","html_url":"https://github.com/apache/spark/pull/16156#discussion_r90972771","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/16156","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/90972771"},"html":{"href":"https://github.com/apache/spark/pull/16156#discussion_r90972771"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/16156"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/16156","id":96616296,"html_url":"https://github.com/apache/spark/pull/16156","diff_url":"https://github.com/apache/spark/pull/16156.diff","patch_url":"https://github.com/apache/spark/pull/16156.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/16156","number":16156,"state":"open","locked":false,"title":"[SPARK-18539][SQL]: tolerate pushed-down filter on non-existing parquet columns","user":{"login":"xwu0226","id":14225992,"avatar_url":"https://avatars.githubusercontent.com/u/14225992?v=3","gravatar_id":"","url":"https://api.github.com/users/xwu0226","html_url":"https://github.com/xwu0226","followers_url":"https://api.github.com/users/xwu0226/followers","following_url":"https://api.github.com/users/xwu0226/following{/other_user}","gists_url":"https://api.github.com/users/xwu0226/gists{/gist_id}","starred_url":"https://api.github.com/users/xwu0226/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xwu0226/subscriptions","organizations_url":"https://api.github.com/users/xwu0226/orgs","repos_url":"https://api.github.com/users/xwu0226/repos","events_url":"https://api.github.com/users/xwu0226/events{/privacy}","received_events_url":"https://api.github.com/users/xwu0226/received_events","type":"User","site_admin":false},"body":"## What changes were proposed in this pull request?\r\n\r\nWhen `spark.sql.parquet.filterPushdown` is on, SparkSQL's Parquet reader pushes down filters to Parquet file when creating reader, in order to start with filtered blocks. However, when the parquet file does not have the predicate column(s), Parquet-mr throw exceptions complaining the filter column does not existing. This issue will be fixed in parquet-mr 1.9. But Spark 2.1 is still on parquet 1.8. \r\n\r\nThis PR is to tolerate such exception thrown by Parquet-mr and just return all the blocks from the current parquet file to the created SparkSQL parquet reader. Filters will be applied again anyway in later physical plan operation. According to following example physical plan:\r\n\r\n```\r\n== Physical Plan ==\r\n*Project [a#2805, b#2806, c#2807]\r\n+- *Filter ((isnotnull(a#2805) && isnull(c#2807)) && (a#2805 < 2))\r\n  +- *FileScan parquet [a#2805,b#2806,c#2807] Batched: true, Format: ParquetFormat, Location: InMemoryFileIndex[file:/Users/xinwu/spark/target/tmp/spark-ed6f0c12-6494-4ac5-b485-5b986ef475cc], PartitionFilters: [], PushedFilters: [IsNotNull(a), IsNull(c), LessThan(a,2)], ReadSchema: struct<a:int,b:string,c:int>\r\n```\r\n\r\n## How was this patch tested?\r\nA unit test case is added. ","created_at":"2016-12-05T21:51:16Z","updated_at":"2016-12-05T22:40:58Z","closed_at":null,"merged_at":null,"merge_commit_sha":"5636721de3e02d5ce36badb0f5e3e3bf75f9ff78","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/16156/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/16156/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/16156/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/096ab18887c40761eb7ba79e9c406fe8ca6ce7c0","head":{"label":"xwu0226:SPARK-18539","ref":"SPARK-18539","sha":"096ab18887c40761eb7ba79e9c406fe8ca6ce7c0","user":{"login":"xwu0226","id":14225992,"avatar_url":"https://avatars.githubusercontent.com/u/14225992?v=3","gravatar_id":"","url":"https://api.github.com/users/xwu0226","html_url":"https://github.com/xwu0226","followers_url":"https://api.github.com/users/xwu0226/followers","following_url":"https://api.github.com/users/xwu0226/following{/other_user}","gists_url":"https://api.github.com/users/xwu0226/gists{/gist_id}","starred_url":"https://api.github.com/users/xwu0226/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xwu0226/subscriptions","organizations_url":"https://api.github.com/users/xwu0226/orgs","repos_url":"https://api.github.com/users/xwu0226/repos","events_url":"https://api.github.com/users/xwu0226/events{/privacy}","received_events_url":"https://api.github.com/users/xwu0226/received_events","type":"User","site_admin":false},"repo":{"id":42968936,"name":"spark","full_name":"xwu0226/spark","owner":{"login":"xwu0226","id":14225992,"avatar_url":"https://avatars.githubusercontent.com/u/14225992?v=3","gravatar_id":"","url":"https://api.github.com/users/xwu0226","html_url":"https://github.com/xwu0226","followers_url":"https://api.github.com/users/xwu0226/followers","following_url":"https://api.github.com/users/xwu0226/following{/other_user}","gists_url":"https://api.github.com/users/xwu0226/gists{/gist_id}","starred_url":"https://api.github.com/users/xwu0226/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xwu0226/subscriptions","organizations_url":"https://api.github.com/users/xwu0226/orgs","repos_url":"https://api.github.com/users/xwu0226/repos","events_url":"https://api.github.com/users/xwu0226/events{/privacy}","received_events_url":"https://api.github.com/users/xwu0226/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/xwu0226/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/xwu0226/spark","forks_url":"https://api.github.com/repos/xwu0226/spark/forks","keys_url":"https://api.github.com/repos/xwu0226/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/xwu0226/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/xwu0226/spark/teams","hooks_url":"https://api.github.com/repos/xwu0226/spark/hooks","issue_events_url":"https://api.github.com/repos/xwu0226/spark/issues/events{/number}","events_url":"https://api.github.com/repos/xwu0226/spark/events","assignees_url":"https://api.github.com/repos/xwu0226/spark/assignees{/user}","branches_url":"https://api.github.com/repos/xwu0226/spark/branches{/branch}","tags_url":"https://api.github.com/repos/xwu0226/spark/tags","blobs_url":"https://api.github.com/repos/xwu0226/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/xwu0226/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/xwu0226/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/xwu0226/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/xwu0226/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/xwu0226/spark/languages","stargazers_url":"https://api.github.com/repos/xwu0226/spark/stargazers","contributors_url":"https://api.github.com/repos/xwu0226/spark/contributors","subscribers_url":"https://api.github.com/repos/xwu0226/spark/subscribers","subscription_url":"https://api.github.com/repos/xwu0226/spark/subscription","commits_url":"https://api.github.com/repos/xwu0226/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/xwu0226/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/xwu0226/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/xwu0226/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/xwu0226/spark/contents/{+path}","compare_url":"https://api.github.com/repos/xwu0226/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/xwu0226/spark/merges","archive_url":"https://api.github.com/repos/xwu0226/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/xwu0226/spark/downloads","issues_url":"https://api.github.com/repos/xwu0226/spark/issues{/number}","pulls_url":"https://api.github.com/repos/xwu0226/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/xwu0226/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/xwu0226/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/xwu0226/spark/labels{/name}","releases_url":"https://api.github.com/repos/xwu0226/spark/releases{/id}","deployments_url":"https://api.github.com/repos/xwu0226/spark/deployments","created_at":"2015-09-23T00:30:54Z","updated_at":"2016-01-15T19:57:06Z","pushed_at":"2016-12-05T21:39:43Z","git_url":"git://github.com/xwu0226/spark.git","ssh_url":"git@github.com:xwu0226/spark.git","clone_url":"https://github.com/xwu0226/spark.git","svn_url":"https://github.com/xwu0226/spark","homepage":null,"size":248157,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"5a92dc76ab431d73275a2bdfbc2c0a8ceb0d75d1","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","deployments_url":"https://api.github.com/repos/apache/spark/deployments","created_at":"2014-02-25T08:00:08Z","updated_at":"2016-12-05T15:08:05Z","pushed_at":"2016-12-05T22:32:48Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":619817,"stargazers_count":10968,"watchers_count":10968,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":10243,"mirror_url":"git://git.apache.org/spark.git","open_issues_count":460,"forks":10243,"open_issues":460,"watchers":10968,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/16156"},"html":{"href":"https://github.com/apache/spark/pull/16156"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/16156"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/16156/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/16156/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/16156/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/096ab18887c40761eb7ba79e9c406fe8ca6ce7c0"}}}}
{"actor":{"display_login":"dongjoon-hyun"}}