{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/11665","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/11665/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/11665/comments","events_url":"https://api.github.com/repos/apache/spark/issues/11665/events","html_url":"https://github.com/apache/spark/pull/11665","id":140319138,"number":11665,"title":"[SPARK-XXXX][SQL] Support for inferring filters from data constraints","user":{"login":"sameeragarwal","id":166454,"avatar_url":"https://avatars.githubusercontent.com/u/166454?v=3","gravatar_id":"","url":"https://api.github.com/users/sameeragarwal","html_url":"https://github.com/sameeragarwal","followers_url":"https://api.github.com/users/sameeragarwal/followers","following_url":"https://api.github.com/users/sameeragarwal/following{/other_user}","gists_url":"https://api.github.com/users/sameeragarwal/gists{/gist_id}","starred_url":"https://api.github.com/users/sameeragarwal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sameeragarwal/subscriptions","organizations_url":"https://api.github.com/users/sameeragarwal/orgs","repos_url":"https://api.github.com/users/sameeragarwal/repos","events_url":"https://api.github.com/users/sameeragarwal/events{/privacy}","received_events_url":"https://api.github.com/users/sameeragarwal/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-03-11T23:40:48Z","updated_at":"2016-03-12T01:03:36Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/11665","html_url":"https://github.com/apache/spark/pull/11665","diff_url":"https://github.com/apache/spark/pull/11665.diff","patch_url":"https://github.com/apache/spark/pull/11665.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\n**[I'll link it to the JIRA once ASF JIRA is back online]**\r\n\r\nThis PR generalizes the `NullFiltering` optimizer rule in catalyst to `InferFiltersFromConstraints` that can automatically infer all relevant filters based on an operator's constraints while making sure of 2 things:\r\n\r\n(a) no redundant filters are generated, and \r\n(b) filters that do not contribute to any further optimizations are not generated.\r\n\r\n## How was this patch tested?\r\n\r\nExtended all tests in `InferFiltersFromConstraintsSuite` (that were initially based on `NullFilteringSuite` to test filter inference in `Filter` and `Join` operators.\r\n\r\nIn particular the 2 tests ( `single inner join with pre-existing filters: filter out values on either side` and `multiple inner joins: filter out values on all sides on equi-join keys` attempts to highlight/test the real potential of this rule for join optimization."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/195620519","html_url":"https://github.com/apache/spark/pull/11665#issuecomment-195620519","issue_url":"https://api.github.com/repos/apache/spark/issues/11665","id":195620519,"user":{"login":"sameeragarwal","id":166454,"avatar_url":"https://avatars.githubusercontent.com/u/166454?v=3","gravatar_id":"","url":"https://api.github.com/users/sameeragarwal","html_url":"https://github.com/sameeragarwal","followers_url":"https://api.github.com/users/sameeragarwal/followers","following_url":"https://api.github.com/users/sameeragarwal/following{/other_user}","gists_url":"https://api.github.com/users/sameeragarwal/gists{/gist_id}","starred_url":"https://api.github.com/users/sameeragarwal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sameeragarwal/subscriptions","organizations_url":"https://api.github.com/users/sameeragarwal/orgs","repos_url":"https://api.github.com/users/sameeragarwal/repos","events_url":"https://api.github.com/users/sameeragarwal/events{/privacy}","received_events_url":"https://api.github.com/users/sameeragarwal/received_events","type":"User","site_admin":false},"created_at":"2016-03-12T01:03:36Z","updated_at":"2016-03-12T01:03:36Z","body":"cc @nongli @yhuai @gatorsmile "}}