{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/11585","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/11585/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/11585/comments","events_url":"https://api.github.com/repos/apache/spark/issues/11585/events","html_url":"https://github.com/apache/spark/pull/11585","id":139377828,"number":11585,"title":"[SPARK-13751] [SQL] generate better code for Filter","user":{"login":"davies","id":40902,"avatar_url":"https://avatars.githubusercontent.com/u/40902?v=3","gravatar_id":"","url":"https://api.github.com/users/davies","html_url":"https://github.com/davies","followers_url":"https://api.github.com/users/davies/followers","following_url":"https://api.github.com/users/davies/following{/other_user}","gists_url":"https://api.github.com/users/davies/gists{/gist_id}","starred_url":"https://api.github.com/users/davies/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davies/subscriptions","organizations_url":"https://api.github.com/users/davies/orgs","repos_url":"https://api.github.com/users/davies/repos","events_url":"https://api.github.com/users/davies/events{/privacy}","received_events_url":"https://api.github.com/users/davies/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":10,"created_at":"2016-03-08T19:57:35Z","updated_at":"2016-03-11T00:22:37Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/11585","html_url":"https://github.com/apache/spark/pull/11585","diff_url":"https://github.com/apache/spark/pull/11585.diff","patch_url":"https://github.com/apache/spark/pull/11585.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nThis PR improve the codegen of Filter by:\r\n\r\n1. filter out the rows early if it have null value in it that will cause the condition result in null or false. After this, we could simplify the condition, because the input are not nullable anymore.\r\n\r\n2. Split the condition as conjunctive predicates, then check them one by one.\r\n\r\nHere is a piece of generated code for Filter in TPCDS Q55:\r\n```java\r\n/* 109 */       /*** CONSUME: Filter ((((isnotnull(d_moy#149) && isnotnull(d_year#147)) && (d_moy#149 = 11)) && (d_year#147 = 1999)) && isnotnull(d_date_sk#141)) */\r\n/* 110 */       /* input[0, int] */\r\n/* 111 */       boolean project_isNull2 = rdd_row.isNullAt(0);\r\n/* 112 */       int project_value2 = project_isNull2 ? -1 : (rdd_row.getInt(0));\r\n/* 113 */       /* input[1, int] */\r\n/* 114 */       boolean project_isNull3 = rdd_row.isNullAt(1);\r\n/* 115 */       int project_value3 = project_isNull3 ? -1 : (rdd_row.getInt(1));\r\n/* 116 */       /* input[2, int] */\r\n/* 117 */       boolean project_isNull4 = rdd_row.isNullAt(2);\r\n/* 118 */       int project_value4 = project_isNull4 ? -1 : (rdd_row.getInt(2));\r\n/* 119 */\r\n/* 120 */       if (project_isNull3) continue;\r\n/* 121 */       if (project_isNull4) continue;\r\n/* 122 */       if (project_isNull2) continue;\r\n/* 123 */\r\n/* 124 */       /* (input[1, int] = 11) */\r\n/* 125 */       boolean filter_value6 = false;\r\n/* 126 */       filter_value6 = project_value3 == 11;\r\n/* 127 */       if (!filter_value6) continue;\r\n/* 128 */\r\n/* 129 */       /* (input[2, int] = 1999) */\r\n/* 130 */       boolean filter_value9 = false;\r\n/* 131 */       filter_value9 = project_value4 == 1999;\r\n/* 132 */       if (!filter_value9) continue;\r\n/* 133 */\r\n/* 134 */       filter_metricValue1.add(1);\r\n/* 135 */\r\n/* 136 */       /*** CONSUME: Project [d_date_sk#141] */\r\n/* 137 */\r\n/* 138 */       project_rowWriter1.write(0, project_value2);\r\n/* 139 */       append(project_result1.copy());\r\n```\r\n\r\n## How was this patch tested?\r\n\r\nExisting tests.\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/195114031","html_url":"https://github.com/apache/spark/pull/11585#issuecomment-195114031","issue_url":"https://api.github.com/repos/apache/spark/issues/11585","id":195114031,"user":{"login":"nongli","id":242468,"avatar_url":"https://avatars.githubusercontent.com/u/242468?v=3","gravatar_id":"","url":"https://api.github.com/users/nongli","html_url":"https://github.com/nongli","followers_url":"https://api.github.com/users/nongli/followers","following_url":"https://api.github.com/users/nongli/following{/other_user}","gists_url":"https://api.github.com/users/nongli/gists{/gist_id}","starred_url":"https://api.github.com/users/nongli/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nongli/subscriptions","organizations_url":"https://api.github.com/users/nongli/orgs","repos_url":"https://api.github.com/users/nongli/repos","events_url":"https://api.github.com/users/nongli/events{/privacy}","received_events_url":"https://api.github.com/users/nongli/received_events","type":"User","site_admin":false},"created_at":"2016-03-11T00:22:37Z","updated_at":"2016-03-11T00:22:37Z","body":"LGTM"}}