{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/11810","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/11810/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/11810/comments","events_url":"https://api.github.com/repos/apache/spark/issues/11810/events","html_url":"https://github.com/apache/spark/pull/11810","id":141765931,"number":11810,"title":"[SPARK-13996][SQL] Add more not null attributes for Filter codegen","user":{"login":"viirya","id":68855,"avatar_url":"https://avatars.githubusercontent.com/u/68855?v=3","gravatar_id":"","url":"https://api.github.com/users/viirya","html_url":"https://github.com/viirya","followers_url":"https://api.github.com/users/viirya/followers","following_url":"https://api.github.com/users/viirya/following{/other_user}","gists_url":"https://api.github.com/users/viirya/gists{/gist_id}","starred_url":"https://api.github.com/users/viirya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/viirya/subscriptions","organizations_url":"https://api.github.com/users/viirya/orgs","repos_url":"https://api.github.com/users/viirya/repos","events_url":"https://api.github.com/users/viirya/events{/privacy}","received_events_url":"https://api.github.com/users/viirya/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2016-03-18T03:42:36Z","updated_at":"2016-04-02T10:12:29Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/11810","html_url":"https://github.com/apache/spark/pull/11810","diff_url":"https://github.com/apache/spark/pull/11810.diff","patch_url":"https://github.com/apache/spark/pull/11810.patch"},"body":"## What changes were proposed in this pull request?\r\nJIRA: https://issues.apache.org/jira/browse/SPARK-13996\r\n\r\nFilter codegen finds the attributes not null by checking IsNotNull(a) expression with a condition if child.output.contains(a). However, the current approach to checking it is not comprehensive. We can improve it.\r\n\r\nE.g., for this plan:\r\n\r\n    val rdd = sqlContext.sparkContext.makeRDD(Seq(Row(1, \"1\"), Row(null, \"1\"), Row(2, \"2\")))\r\n    val schema = new StructType().add(\"k\", IntegerType).add(\"v\", StringType)\r\n    val smallDF = sqlContext.createDataFrame(rdd, schema)\r\n    val df = smallDF.filter(\"isnotnull(k + 1)\")\r\n\r\nThe code snippet generated without this patch:\r\n\r\n    /* 031 */   protected void processNext() throws java.io.IOException {\r\n    /* 032 */     /*** PRODUCE: Filter isnotnull((k#0 + 1)) */\r\n    /* 033 */\r\n    /* 034 */     /*** PRODUCE: INPUT */\r\n    /* 035 */\r\n    /* 036 */     while (!shouldStop() && inputadapter_input.hasNext()) {\r\n    /* 037 */       InternalRow inputadapter_row = (InternalRow) inputadapter_input.next();\r\n    /* 038 */       /*** CONSUME: Filter isnotnull((k#0 + 1)) */\r\n    /* 039 */       /* input[0, int] */\r\n    /* 040 */       boolean filter_isNull = inputadapter_row.isNullAt(0);\r\n    /* 041 */       int filter_value = filter_isNull ? -1 : (inputadapter_row.getInt(0));\r\n    /* 042 */\r\n    /* 043 */       /* isnotnull((input[0, int] + 1)) */\r\n    /* 044 */       /* (input[0, int] + 1) */\r\n    /* 045 */       boolean filter_isNull3 = true;\r\n    /* 046 */       int filter_value3 = -1;\r\n    /* 047 */   \r\n    /* 048 */       if (!filter_isNull) {\r\n    /* 049 */         filter_isNull3 = false; // resultCode could change nullability.\r\n    /* 050 */         filter_value3 = filter_value + 1;\r\n    /* 051 */     \r\n    /* 052 */       }\r\n    /* 053 */       if (!(!(filter_isNull3))) continue;\r\n    /* 054 */   \r\n    /* 055 */       filter_metricValue.add(1);\r\n\r\nWith this patch:\r\n\r\n    /* 031 */   protected void processNext() throws java.io.IOException {\r\n    /* 032 */     /*** PRODUCE: Filter isnotnull((k#0 + 1)) */\r\n    /* 033 */\r\n    /* 034 */     /*** PRODUCE: INPUT */\r\n    /* 035 */\r\n    /* 036 */     while (!shouldStop() && inputadapter_input.hasNext()) {\r\n    /* 037 */       InternalRow inputadapter_row = (InternalRow) inputadapter_input.next();\r\n    /* 038 */       /*** CONSUME: Filter isnotnull((k#0 + 1)) */\r\n    /* 039 */       /* input[0, int] */\r\n    /* 040 */       boolean filter_isNull = inputadapter_row.isNullAt(0);\r\n    /* 041 */       int filter_value = filter_isNull ? -1 : (inputadapter_row.getInt(0));\r\n    /* 042 */\r\n    /* 043 */       if (filter_isNull) continue;\r\n    /* 044 */\r\n    /* 045 */       filter_metricValue.add(1);\r\n\r\n\r\n## How was this patch tested?\r\n\r\nExisting tests.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/204688087","html_url":"https://github.com/apache/spark/pull/11810#issuecomment-204688087","issue_url":"https://api.github.com/repos/apache/spark/issues/11810","id":204688087,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2016-04-02T10:12:29Z","updated_at":"2016-04-02T10:12:29Z","body":"**[Test build #54771 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54771/consoleFull)** for PR 11810 at commit [`0c7afe7`](https://github.com/apache/spark/commit/0c7afe751e087b5de3b55ac40d189e10a3742776)."}}