{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/23320","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/23320/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/23320/comments","events_url":"https://api.github.com/repos/apache/spark/issues/23320/events","html_url":"https://github.com/apache/spark/pull/23320","id":391000477,"node_id":"MDExOlB1bGxSZXF1ZXN0MjM4NjM1ODgw","number":23320,"title":"[SPARK-26370][SQL] Fix resolution of higher-order function for the same identifier.","user":{"login":"ueshin","id":506656,"node_id":"MDQ6VXNlcjUwNjY1Ng==","avatar_url":"https://avatars1.githubusercontent.com/u/506656?v=4","gravatar_id":"","url":"https://api.github.com/users/ueshin","html_url":"https://github.com/ueshin","followers_url":"https://api.github.com/users/ueshin/followers","following_url":"https://api.github.com/users/ueshin/following{/other_user}","gists_url":"https://api.github.com/users/ueshin/gists{/gist_id}","starred_url":"https://api.github.com/users/ueshin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ueshin/subscriptions","organizations_url":"https://api.github.com/users/ueshin/orgs","repos_url":"https://api.github.com/users/ueshin/repos","events_url":"https://api.github.com/users/ueshin/events{/privacy}","received_events_url":"https://api.github.com/users/ueshin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-12-14T07:44:54Z","updated_at":"2018-12-14T12:02:19Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/23320","html_url":"https://github.com/apache/spark/pull/23320","diff_url":"https://github.com/apache/spark/pull/23320.diff","patch_url":"https://github.com/apache/spark/pull/23320.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nWhen using a higher-order function with the same variable name as the existing columns in `Filter` or something which uses `Analyzer.resolveExpressionBottomUp` during the resolution, e.g.,:\r\n\r\n```scala\r\nval df = Seq(\r\n  (Seq(1, 9, 8, 7), 1, 2),\r\n  (Seq(5, 9, 7), 2, 2),\r\n  (Seq.empty, 3, 2),\r\n  (null, 4, 2)\r\n).toDF(\"i\", \"x\", \"d\")\r\n\r\ncheckAnswer(df.filter(\"exists(i, x -> x % d == 0)\"),\r\n  Seq(Row(Seq(1, 9, 8, 7), 1, 2)))\r\ncheckAnswer(df.select(\"x\").filter(\"exists(i, x -> x % d == 0)\"),\r\n  Seq(Row(1)))\r\n```\r\n\r\nthe following exception happens:\r\n\r\n```\r\njava.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.BoundReference cannot be cast to org.apache.spark.sql.catalyst.expressions.NamedExpression\r\n  at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)\r\n  at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)\r\n  at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)\r\n  at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)\r\n  at scala.collection.TraversableLike.map(TraversableLike.scala:237)\r\n  at scala.collection.TraversableLike.map$(TraversableLike.scala:230)\r\n  at scala.collection.AbstractTraversable.map(Traversable.scala:108)\r\n  at org.apache.spark.sql.catalyst.expressions.HigherOrderFunction.$anonfun$functionsForEval$1(higherOrderFunctions.scala:147)\r\n  at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)\r\n  at scala.collection.immutable.List.foreach(List.scala:392)\r\n  at scala.collection.TraversableLike.map(TraversableLike.scala:237)\r\n  at scala.collection.TraversableLike.map$(TraversableLike.scala:230)\r\n  at scala.collection.immutable.List.map(List.scala:298)\r\n  at org.apache.spark.sql.catalyst.expressions.HigherOrderFunction.functionsForEval(higherOrderFunctions.scala:145)\r\n  at org.apache.spark.sql.catalyst.expressions.HigherOrderFunction.functionsForEval$(higherOrderFunctions.scala:145)\r\n  at org.apache.spark.sql.catalyst.expressions.ArrayExists.functionsForEval$lzycompute(higherOrderFunctions.scala:369)\r\n  at org.apache.spark.sql.catalyst.expressions.ArrayExists.functionsForEval(higherOrderFunctions.scala:369)\r\n  at org.apache.spark.sql.catalyst.expressions.SimpleHigherOrderFunction.functionForEval(higherOrderFunctions.scala:176)\r\n  at org.apache.spark.sql.catalyst.expressions.SimpleHigherOrderFunction.functionForEval$(higherOrderFunctions.scala:176)\r\n  at org.apache.spark.sql.catalyst.expressions.ArrayExists.functionForEval(higherOrderFunctions.scala:369)\r\n  at org.apache.spark.sql.catalyst.expressions.ArrayExists.nullSafeEval(higherOrderFunctions.scala:387)\r\n  at org.apache.spark.sql.catalyst.expressions.SimpleHigherOrderFunction.eval(higherOrderFunctions.scala:190)\r\n  at org.apache.spark.sql.catalyst.expressions.SimpleHigherOrderFunction.eval$(higherOrderFunctions.scala:185)\r\n  at org.apache.spark.sql.catalyst.expressions.ArrayExists.eval(higherOrderFunctions.scala:369)\r\n  at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificPredicate.eval(Unknown Source)\r\n  at org.apache.spark.sql.execution.FilterExec.$anonfun$doExecute$3(basicPhysicalOperators.scala:216)\r\n  at org.apache.spark.sql.execution.FilterExec.$anonfun$doExecute$3$adapted(basicPhysicalOperators.scala:215)\r\n\r\n...\r\n```\r\n\r\nbecause the `UnresolvedAttribute`s in `LambdaFunction` are unexpectedly resolved by the rule.\r\n\r\nThis pr modified to use a placeholder `UnresolvedNamedLambdaVariable` to prevent unexpected resolution.\r\n\r\n## How was this patch tested?\r\n\r\nAdded a test and modified some tests.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/447305059","html_url":"https://github.com/apache/spark/pull/23320#issuecomment-447305059","issue_url":"https://api.github.com/repos/apache/spark/issues/23320","id":447305059,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NzMwNTA1OQ==","user":{"login":"AmplabJenkins","id":4031291,"node_id":"MDQ6VXNlcjQwMzEyOTE=","avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2018-12-14T12:02:19Z","updated_at":"2018-12-14T12:02:19Z","author_association":"NONE","body":"Merged build finished. Test PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}