{"push_id":1445577674,"size":2,"distinct_size":2,"ref":"refs/heads/master","head":"6a5a7254dc37952505989e9e580a14543adb730c","before":"9bf8f3cd4f62f921c32fb50b8abf49576a80874f","commits":[{"sha":"7f3c778fd0ab0298d71e756990051e7c073f9151","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-18718][TESTS] Skip some test failures due to path length limitation and fix tests to pass on Windows\n\n## What changes were proposed in this pull request?\n\nThere are some tests failed on Windows due to the wrong format of path and the limitation of path length as below:\n\nThis PR proposes both to fix the failed tests by fixing the path for the tests below:\n\n- `InsertSuite`\n  ```\n  Exception encountered when attempting to run a suite with class name: org.apache.spark.sql.sources.InsertSuite *** ABORTED *** (12 seconds, 547 milliseconds)\n      org.apache.spark.sql.AnalysisException: Path does not exist: file:/C:projectsspark\target\tmpspark-177945ef-9128-42b4-8c07-de31f78bbbd6;\n      at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:382)\n      at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:370)\n      at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)\n  ```\n\n- `PathOptionSuite`\n  ```\n  - path option also exist for write path *** FAILED *** (1 second, 93 milliseconds)\n    \"C:[projectsspark\target\tmp]spark-5ab34a58-df8d-...\" did not equal \"C:[\\projects\\spark\\target\\tmp\\]spark-5ab34a58-df8d-...\" (PathOptionSuite.scala:93)\n    org.scalatest.exceptions.TestFailedException:\n        at org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:500)\n        at org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1555)\n    ...\n  ```\n\n- `UDFSuite`\n  ```\n  - SPARK-8005 input_file_name *** FAILED *** (2 seconds, 234 milliseconds)\n    \"file:///C:/projects/spark/target/tmp/spark-e4e5720a-2006-48f9-8b11-797bf59794bf/part-00001-26fb05e4-603d-471d-ae9d-b9549e0c7765.snappy.parquet\" did not contain \"C:\\projects\\spark\\target\\tmp\\spark-e4e5720a-2006-48f9-8b11-797bf59794bf\" (UDFSuite.scala:67)\n    org.scalatest.exceptions.TestFailedException:\n      at org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:500)\n      at org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1555)\n    ...\n  ```\n\nand to skip the tests belows which are being failed on Windows due to path length limitation.\n\n- `SparkLauncherSuite`\n  ```\n  Test org.apache.spark.launcher.SparkLauncherSuite.testChildProcLauncher failed: java.lang.AssertionError: expected:<0> but was:<1>, took 0.062 sec\n    at org.apache.spark.launcher.SparkLauncherSuite.testChildProcLauncher(SparkLauncherSuite.java:177)\n      ...\n  ```\n\n  The stderr from the process is `The filename or extension is too long` which is equivalent to the one below.\n\n- `BroadcastJoinSuite`\n  ```\n  04:09:40.882 ERROR org.apache.spark.deploy.worker.ExecutorRunner: Error running executor\n  java.io.IOException: Cannot run program \"C:\\Progra~1\\Java\\jdk1.8.0\\bin\\java\" (in directory \"C:\\projects\\spark\\work\\app-20161205040542-0000\\51658\"): CreateProcess error=206, The filename or extension is too long\n      at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)\n      at org.apache.spark.deploy.worker.ExecutorRunner.org$apache$spark$deploy$worker$ExecutorRunner$$fetchAndRunExecutor(ExecutorRunner.scala:167)\n      at org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:73)\n  Caused by: java.io.IOException: CreateProcess error=206, The filename or extension is too long\n      at java.lang.ProcessImpl.create(Native Method)\n      at java.lang.ProcessImpl.<init>(ProcessImpl.java:386)\n      at java.lang.ProcessImpl.start(ProcessImpl.java:137)\n      at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)\n      ... 2 more\n  04:09:40.929 ERROR org.apache.spark.deploy.worker.ExecutorRunner: Error running executor\n\n    (appearently infinite same error messages)\n\n  ...\n  ```\n\n## How was this patch tested?\n\nManually tested via AppVeyor.\n\n**Before**\n\n`InsertSuite`: https://ci.appveyor.com/project/spark-test/spark/build/148-InsertSuite-pr\n`PathOptionSuite`: https://ci.appveyor.com/project/spark-test/spark/build/139-PathOptionSuite-pr\n`UDFSuite`: https://ci.appveyor.com/project/spark-test/spark/build/143-UDFSuite-pr\n`SparkLauncherSuite`: https://ci.appveyor.com/project/spark-test/spark/build/141-SparkLauncherSuite-pr\n`BroadcastJoinSuite`: https://ci.appveyor.com/project/spark-test/spark/build/145-BroadcastJoinSuite-pr\n\n**After**\n\n`PathOptionSuite`: https://ci.appveyor.com/project/spark-test/spark/build/140-PathOptionSuite-pr\n`SparkLauncherSuite`: https://ci.appveyor.com/project/spark-test/spark/build/142-SparkLauncherSuite-pr\n`UDFSuite`: https://ci.appveyor.com/project/spark-test/spark/build/144-UDFSuite-pr\n`InsertSuite`: https://ci.appveyor.com/project/spark-test/spark/build/147-InsertSuite-pr\n`BroadcastJoinSuite`: https://ci.appveyor.com/project/spark-test/spark/build/149-BroadcastJoinSuite-pr\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #16147 from HyukjinKwon/fix-tests.","distinct":true,"url":"https://api.github.com/repos/dbtsai/spark/commits/7f3c778fd0ab0298d71e756990051e7c073f9151"},{"sha":"6a5a7254dc37952505989e9e580a14543adb730c","author":{"name":"Liang-Chi Hsieh","email":"9f98897f009ec1944a62ee7d389f9f5b1873efea@gmail.com"},"message":"[SPARK-18667][PYSPARK][SQL] Change the way to group row in BatchEvalPythonExec so input_file_name function can work with UDF in pyspark\n\n## What changes were proposed in this pull request?\n\n`input_file_name` doesn't return filename when working with UDF in PySpark. An example shows the problem:\n\n    from pyspark.sql.functions import *\n    from pyspark.sql.types import *\n\n    def filename(path):\n        return path\n\n    sourceFile = udf(filename, StringType())\n    spark.read.json(\"tmp.json\").select(sourceFile(input_file_name())).show()\n\n    +---------------------------+\n    |filename(input_file_name())|\n    +---------------------------+\n    |                           |\n    +---------------------------+\n\nThe cause of this issue is, we group rows in `BatchEvalPythonExec` for batching processing of PythonUDF. Currently we group rows first and then evaluate expressions on the rows. If the data is less than the required number of rows for a group, the iterator will be consumed to the end before the evaluation. However, once the iterator reaches the end, we will unset input filename. So the input_file_name expression can't return correct filename.\n\nThis patch fixes the approach to group the batch of rows. We evaluate the expression first and then group evaluated results to batch.\n\n## How was this patch tested?\n\nAdded unit test to PySpark.\n\nPlease review http://spark.apache.org/contributing.html before opening a pull request.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #16115 from viirya/fix-py-udf-input-filename.","distinct":true,"url":"https://api.github.com/repos/dbtsai/spark/commits/6a5a7254dc37952505989e9e580a14543adb730c"}]}
{"actor":{"display_login":"dbtsai-bot"}}