{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/14912","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/14912/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/14912/comments","events_url":"https://api.github.com/repos/apache/spark/issues/14912/events","html_url":"https://github.com/apache/spark/pull/14912","id":174449358,"number":14912,"title":"[SPARK-17357][SQL] Fix current predicate pushdown","user":{"login":"viirya","id":68855,"avatar_url":"https://avatars.githubusercontent.com/u/68855?v=3","gravatar_id":"","url":"https://api.github.com/users/viirya","html_url":"https://github.com/viirya","followers_url":"https://api.github.com/users/viirya/followers","following_url":"https://api.github.com/users/viirya/following{/other_user}","gists_url":"https://api.github.com/users/viirya/gists{/gist_id}","starred_url":"https://api.github.com/users/viirya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/viirya/subscriptions","organizations_url":"https://api.github.com/users/viirya/orgs","repos_url":"https://api.github.com/users/viirya/repos","events_url":"https://api.github.com/users/viirya/events{/privacy}","received_events_url":"https://api.github.com/users/viirya/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2016-09-01T06:51:46Z","updated_at":"2016-09-09T06:57:18Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/14912","html_url":"https://github.com/apache/spark/pull/14912","diff_url":"https://github.com/apache/spark/pull/14912.diff","patch_url":"https://github.com/apache/spark/pull/14912.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nCurrently some predicates, which should be pushdown, will not be correctly pushdown in `Optimizer`.\r\n\r\n### Predicates will be simplified to the form that can't be pushdown\r\n\r\nIn `Optimizer`, `Filter` operator will go through the rules `PushDownPredicate`, `CombineFilters` and `BooleanSimplification`.\r\n\r\nUnder this rule order, it is possibly that some predicates that should be able to push down, can't be pushed down through operators.\r\n\r\nBecause `Filter` will not pushdown through another `Filter` node and will wait for combination in the rule `CombineFilters` later. After the `Filter`s are combined, `BooleanSimplification` will simplify conditions in the combined `Filter` and make some predicates, which are able to push down, become unable to push down.\r\n\r\n### Some predicates can't be pushdown due to the disjunctive form\r\n\r\nSome predicates can't be pushdown just because it is disjunctive form. If we convert it to conjunctive form, it can be pushdown.\r\n\r\n### Change in this patch\r\n\r\nWe convert predicates to conjunctive norm form in `CombineFilters` and then combine two `Filter`s. We perform `PushDownPredicates` rule on the combined `Filter` immediately to avoid the predicate to be simplified by `BooleanSimplification`.\r\n\r\n## How was this patch tested?\r\n\r\nJenkins tests.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/245835073","html_url":"https://github.com/apache/spark/pull/14912#issuecomment-245835073","issue_url":"https://api.github.com/repos/apache/spark/issues/14912","id":245835073,"user":{"login":"viirya","id":68855,"avatar_url":"https://avatars.githubusercontent.com/u/68855?v=3","gravatar_id":"","url":"https://api.github.com/users/viirya","html_url":"https://github.com/viirya","followers_url":"https://api.github.com/users/viirya/followers","following_url":"https://api.github.com/users/viirya/following{/other_user}","gists_url":"https://api.github.com/users/viirya/gists{/gist_id}","starred_url":"https://api.github.com/users/viirya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/viirya/subscriptions","organizations_url":"https://api.github.com/users/viirya/orgs","repos_url":"https://api.github.com/users/viirya/repos","events_url":"https://api.github.com/users/viirya/events{/privacy}","received_events_url":"https://api.github.com/users/viirya/received_events","type":"User","site_admin":false},"created_at":"2016-09-09T06:57:18Z","updated_at":"2016-09-09T06:57:18Z","body":"@gatorsmile I've described it in the pr description.\r\n\r\nSimply said, now a Filter will be stopped to pushdown once it encounters another Filter. `BooleanSimplification` rule will simplify the predicate to a form that can't be pushed down in next round of optimization. For example, (a > 10 || b > 2) && (a > 10 || c == 3) will be simplified as (a > 10) || (b > 2 && c == 3).\r\n\r\nThis patch does is to perform `PushDownPredicate` once the adjoining Filters are merged.\r\n"}}
{"actor":{"display_login":"viirya"}}