{"push_id":1470909544,"size":147,"distinct_size":147,"ref":"refs/heads/master","head":"a6f008227b1515a6dcaf5ea3b756069aef8d5412","before":"97255497d885f0f8ccfc808e868bc8aa5e4d1063","commits":[{"sha":"330fda8aa289e0142e174ed6f03b8fa28d08470f","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"Close stale pull requests.\n\nCloses #15689\nCloses #14640\nCloses #15917\nCloses #16188\nCloses #16206","distinct":true,"url":"https://api.github.com/repos/dautovri/spark/commits/330fda8aa289e0142e174ed6f03b8fa28d08470f"},{"sha":"b47b892e4579b7b06b4b2837ee4b614e517789f9","author":{"name":"Shixiong Zhu","email":"eb2f6da57ec54996cebae8c2a82b9fbfd527e14c@databricks.com"},"message":"[SPARK-18774][CORE][SQL] Ignore non-existing files when ignoreCorruptFiles is enabled\n\n## What changes were proposed in this pull request?\n\nWhen `ignoreCorruptFiles` is enabled, it's better to also ignore non-existing files.\n\n## How was this patch tested?\n\nJenkins\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #16203 from zsxwing/ignore-file-not-found.","distinct":true,"url":"https://api.github.com/repos/dautovri/spark/commits/b47b892e4579b7b06b4b2837ee4b614e517789f9"},{"sha":"9bf8f3cd4f62f921c32fb50b8abf49576a80874f","author":{"name":"Yanbo Liang","email":"b6929798b4e4fba75014af8265904a7818902ec7@gmail.com"},"message":"[SPARK-18325][SPARKR][ML] SparkR ML wrappers example code and user guide\n\n## What changes were proposed in this pull request?\n* Add all R examples for ML wrappers which were added during 2.1 release cycle.\n* Split the whole ```ml.R``` example file into individual example for each algorithm, which will be convenient for users to rerun them.\n* Add corresponding examples to ML user guide.\n* Update ML section of SparkR user guide.\n\nNote: MLlib Scala/Java/Python examples will be consistent, however, SparkR examples may different from them, since R users may use the algorithms in a different way, for example, using R ```formula``` to specify ```featuresCol``` and ```labelCol```.\n\n## How was this patch tested?\nRun all examples manually.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #16148 from yanboliang/spark-18325.","distinct":true,"url":"https://api.github.com/repos/dautovri/spark/commits/9bf8f3cd4f62f921c32fb50b8abf49576a80874f"},{"sha":"7f3c778fd0ab0298d71e756990051e7c073f9151","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-18718][TESTS] Skip some test failures due to path length limitation and fix tests to pass on Windows\n\n## What changes were proposed in this pull request?\n\nThere are some tests failed on Windows due to the wrong format of path and the limitation of path length as below:\n\nThis PR proposes both to fix the failed tests by fixing the path for the tests below:\n\n- `InsertSuite`\n  ```\n  Exception encountered when attempting to run a suite with class name: org.apache.spark.sql.sources.InsertSuite *** ABORTED *** (12 seconds, 547 milliseconds)\n      org.apache.spark.sql.AnalysisException: Path does not exist: file:/C:projectsspark\target\tmpspark-177945ef-9128-42b4-8c07-de31f78bbbd6;\n      at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:382)\n      at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:370)\n      at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)\n  ```\n\n- `PathOptionSuite`\n  ```\n  - path option also exist for write path *** FAILED *** (1 second, 93 milliseconds)\n    \"C:[projectsspark\target\tmp]spark-5ab34a58-df8d-...\" did not equal \"C:[\\projects\\spark\\target\\tmp\\]spark-5ab34a58-df8d-...\" (PathOptionSuite.scala:93)\n    org.scalatest.exceptions.TestFailedException:\n        at org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:500)\n        at org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1555)\n    ...\n  ```\n\n- `UDFSuite`\n  ```\n  - SPARK-8005 input_file_name *** FAILED *** (2 seconds, 234 milliseconds)\n    \"file:///C:/projects/spark/target/tmp/spark-e4e5720a-2006-48f9-8b11-797bf59794bf/part-00001-26fb05e4-603d-471d-ae9d-b9549e0c7765.snappy.parquet\" did not contain \"C:\\projects\\spark\\target\\tmp\\spark-e4e5720a-2006-48f9-8b11-797bf59794bf\" (UDFSuite.scala:67)\n    org.scalatest.exceptions.TestFailedException:\n      at org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:500)\n      at org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1555)\n    ...\n  ```\n\nand to skip the tests belows which are being failed on Windows due to path length limitation.\n\n- `SparkLauncherSuite`\n  ```\n  Test org.apache.spark.launcher.SparkLauncherSuite.testChildProcLauncher failed: java.lang.AssertionError: expected:<0> but was:<1>, took 0.062 sec\n    at org.apache.spark.launcher.SparkLauncherSuite.testChildProcLauncher(SparkLauncherSuite.java:177)\n      ...\n  ```\n\n  The stderr from the process is `The filename or extension is too long` which is equivalent to the one below.\n\n- `BroadcastJoinSuite`\n  ```\n  04:09:40.882 ERROR org.apache.spark.deploy.worker.ExecutorRunner: Error running executor\n  java.io.IOException: Cannot run program \"C:\\Progra~1\\Java\\jdk1.8.0\\bin\\java\" (in directory \"C:\\projects\\spark\\work\\app-20161205040542-0000\\51658\"): CreateProcess error=206, The filename or extension is too long\n      at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)\n      at org.apache.spark.deploy.worker.ExecutorRunner.org$apache$spark$deploy$worker$ExecutorRunner$$fetchAndRunExecutor(ExecutorRunner.scala:167)\n      at org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:73)\n  Caused by: java.io.IOException: CreateProcess error=206, The filename or extension is too long\n      at java.lang.ProcessImpl.create(Native Method)\n      at java.lang.ProcessImpl.<init>(ProcessImpl.java:386)\n      at java.lang.ProcessImpl.start(ProcessImpl.java:137)\n      at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)\n      ... 2 more\n  04:09:40.929 ERROR org.apache.spark.deploy.worker.ExecutorRunner: Error running executor\n\n    (appearently infinite same error messages)\n\n  ...\n  ```\n\n## How was this patch tested?\n\nManually tested via AppVeyor.\n\n**Before**\n\n`InsertSuite`: https://ci.appveyor.com/project/spark-test/spark/build/148-InsertSuite-pr\n`PathOptionSuite`: https://ci.appveyor.com/project/spark-test/spark/build/139-PathOptionSuite-pr\n`UDFSuite`: https://ci.appveyor.com/project/spark-test/spark/build/143-UDFSuite-pr\n`SparkLauncherSuite`: https://ci.appveyor.com/project/spark-test/spark/build/141-SparkLauncherSuite-pr\n`BroadcastJoinSuite`: https://ci.appveyor.com/project/spark-test/spark/build/145-BroadcastJoinSuite-pr\n\n**After**\n\n`PathOptionSuite`: https://ci.appveyor.com/project/spark-test/spark/build/140-PathOptionSuite-pr\n`SparkLauncherSuite`: https://ci.appveyor.com/project/spark-test/spark/build/142-SparkLauncherSuite-pr\n`UDFSuite`: https://ci.appveyor.com/project/spark-test/spark/build/144-UDFSuite-pr\n`InsertSuite`: https://ci.appveyor.com/project/spark-test/spark/build/147-InsertSuite-pr\n`BroadcastJoinSuite`: https://ci.appveyor.com/project/spark-test/spark/build/149-BroadcastJoinSuite-pr\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #16147 from HyukjinKwon/fix-tests.","distinct":true,"url":"https://api.github.com/repos/dautovri/spark/commits/7f3c778fd0ab0298d71e756990051e7c073f9151"},{"sha":"6a5a7254dc37952505989e9e580a14543adb730c","author":{"name":"Liang-Chi Hsieh","email":"9f98897f009ec1944a62ee7d389f9f5b1873efea@gmail.com"},"message":"[SPARK-18667][PYSPARK][SQL] Change the way to group row in BatchEvalPythonExec so input_file_name function can work with UDF in pyspark\n\n## What changes were proposed in this pull request?\n\n`input_file_name` doesn't return filename when working with UDF in PySpark. An example shows the problem:\n\n    from pyspark.sql.functions import *\n    from pyspark.sql.types import *\n\n    def filename(path):\n        return path\n\n    sourceFile = udf(filename, StringType())\n    spark.read.json(\"tmp.json\").select(sourceFile(input_file_name())).show()\n\n    +---------------------------+\n    |filename(input_file_name())|\n    +---------------------------+\n    |                           |\n    +---------------------------+\n\nThe cause of this issue is, we group rows in `BatchEvalPythonExec` for batching processing of PythonUDF. Currently we group rows first and then evaluate expressions on the rows. If the data is less than the required number of rows for a group, the iterator will be consumed to the end before the evaluation. However, once the iterator reaches the end, we will unset input filename. So the input_file_name expression can't return correct filename.\n\nThis patch fixes the approach to group the batch of rows. We evaluate the expression first and then group evaluated results to batch.\n\n## How was this patch tested?\n\nAdded unit test to PySpark.\n\nPlease review http://spark.apache.org/contributing.html before opening a pull request.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #16115 from viirya/fix-py-udf-input-filename.","distinct":true,"url":"https://api.github.com/repos/dautovri/spark/commits/6a5a7254dc37952505989e9e580a14543adb730c"},{"sha":"b44d1b8fcf00b238df434cf70ad09460b27adf07","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"[SPARK-18662][HOTFIX] Add new resource-managers directories to SparkLauncher.\n\nThese directories are added to the classpath of applications when testing or\nusing SPARK_PREPEND_CLASSES, otherwise updated classes are not seen. Also,\nadd the mesos directory which was missing.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #16202 from vanzin/SPARK-18662.","distinct":true,"url":"https://api.github.com/repos/dautovri/spark/commits/b44d1b8fcf00b238df434cf70ad09460b27adf07"},{"sha":"ed8869ebbf39783b16daba2e2498a2bc1889306f","author":{"name":"Ergin Seyfe","email":"551bf3c71d9da660a52ecd3224fc4c86679b46d0@fb.com"},"message":"[SPARK-8617][WEBUI] HistoryServer: Include in-progress files during cleanup\n\n## What changes were proposed in this pull request?\n- Removed the`attempt.completed ` filter so cleaner would include the orphan inprogress files.\n- Use loading time for inprogress files as lastUpdated. Keep using the modTime for completed files. First one will prevent deletion of inprogress job files. Second one will ensure that lastUpdated time won't change for completed jobs in an event of HistoryServer reboot.\n\n## How was this patch tested?\nAdded new unittests and via existing tests.\n\nAuthor: Ergin Seyfe <eseyfe@fb.com>\n\nCloses #16165 from seyfe/clear_old_inprogress_files.","distinct":true,"url":"https://api.github.com/repos/dautovri/spark/commits/ed8869ebbf39783b16daba2e2498a2bc1889306f"},{"sha":"3c68944b229aaaeeaee3efcbae3e3be9a2914855","author":{"name":"Andrew Ray","email":"2ffe820a8c24bad2ad24e5f1099708221aa8fd00@gmail.com"},"message":"[SPARK-16589] [PYTHON] Chained cartesian produces incorrect number of records\n\n## What changes were proposed in this pull request?\n\nFixes a bug in the python implementation of rdd cartesian product related to batching that showed up in repeated cartesian products with seemingly random results. The root cause being multiple iterators pulling from the same stream in the wrong order because of logic that ignored batching.\n\n`CartesianDeserializer` and `PairDeserializer` were changed to implement `_load_stream_without_unbatching` and borrow the one line implementation of `load_stream` from `BatchedSerializer`. The default implementation of `_load_stream_without_unbatching` was changed to give consistent results (always an iterable) so that it could be used without additional checks.\n\n`PairDeserializer` no longer extends `CartesianDeserializer` as it was not really proper. If wanted a new common super class could be added.\n\nBoth `CartesianDeserializer` and `PairDeserializer` now only extend `Serializer` (which has no `dump_stream` implementation) since they are only meant for *de*serialization.\n\n## How was this patch tested?\n\nAdditional unit tests (sourced from #14248) plus one for testing a cartesian with zip.\n\nAuthor: Andrew Ray <ray.andrew@gmail.com>\n\nCloses #16121 from aray/fix-cartesian.","distinct":true,"url":"https://api.github.com/repos/dautovri/spark/commits/3c68944b229aaaeeaee3efcbae3e3be9a2914855"},{"sha":"c3d3a9d0e85b834abef87069e4edd27db87fc607","author":{"name":"Felix Cheung","email":"00eb71019ad7acfd67cff51fce666aad32ce6eed@hotmail.com"},"message":"[SPARK-18590][SPARKR] build R source package when making distribution\n\n## What changes were proposed in this pull request?\n\nThis PR has 2 key changes. One, we are building source package (aka bundle package) for SparkR which could be released on CRAN. Two, we should include in the official Spark binary distributions SparkR installed from this source package instead (which would have help/vignettes rds needed for those to work when the SparkR package is loaded in R, whereas earlier approach with devtools does not)\n\nBut, because of various differences in how R performs different tasks, this PR is a fair bit more complicated. More details below.\n\nThis PR also includes a few minor fixes.\n\n### more details\n\nThese are the additional steps in make-distribution; please see [here](https://github.com/apache/spark/blob/master/R/CRAN_RELEASE.md) on what's going to a CRAN release, which is now run during make-distribution.sh.\n1. package needs to be installed because the first code block in vignettes is `library(SparkR)` without lib path\n2. `R CMD build` will build vignettes (this process runs Spark/SparkR code and captures outputs into pdf documentation)\n3. `R CMD check` on the source package will install package and build vignettes again (this time from source packaged) - this is a key step required to release R package on CRAN\n (will skip tests here but tests will need to pass for CRAN release process to success - ideally, during release signoff we should install from the R source package and run tests)\n4. `R CMD Install` on the source package (this is the only way to generate doc/vignettes rds files correctly, not in step # 1)\n (the output of this step is what we package into Spark dist and sparkr.zip)\n\nAlternatively,\n   R CMD build should already be installing the package in a temp directory though it might just be finding this location and set it to lib.loc parameter; another approach is perhaps we could try calling `R CMD INSTALL --build pkg` instead.\n But in any case, despite installing the package multiple times this is relatively fast.\nBuilding vignettes takes a while though.\n\n## How was this patch tested?\n\nManually, CI.\n\nAuthor: Felix Cheung <felixcheung_m@hotmail.com>\n\nCloses #16014 from felixcheung/rdist.","distinct":true,"url":"https://api.github.com/repos/dautovri/spark/commits/c3d3a9d0e85b834abef87069e4edd27db87fc607"},{"sha":"26432df9cc6ffe569583aa628c6ecd7050b38316","author":{"name":"Shixiong Zhu","email":"eb2f6da57ec54996cebae8c2a82b9fbfd527e14c@databricks.com"},"message":"[SPARK-18751][CORE] Fix deadlock when SparkContext.stop is called in Utils.tryOrStopSparkContext\n\n## What changes were proposed in this pull request?\n\nWhen `SparkContext.stop` is called in `Utils.tryOrStopSparkContext` (the following three places), it will cause deadlock because the `stop` method needs to wait for the thread running `stop` to exit.\n\n- ContextCleaner.keepCleaning\n- LiveListenerBus.listenerThread.run\n- TaskSchedulerImpl.start\n\nThis PR adds `SparkContext.stopInNewThread` and uses it to eliminate the potential deadlock. I also removed my changes in #15775 since they are not necessary now.\n\n## How was this patch tested?\n\nJenkins\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #16178 from zsxwing/fix-stop-deadlock.","distinct":true,"url":"https://api.github.com/repos/dautovri/spark/commits/26432df9cc6ffe569583aa628c6ecd7050b38316"},{"sha":"5f894d23a54ea99f75f8b722e111e5270f7f80cf","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-18760][SQL] Consistent format specification for FileFormats\n\n## What changes were proposed in this pull request?\nThis patch fixes the format specification in explain for file sources (Parquet and Text formats are the only two that are different from the rest):\n\nBefore:\n```\nscala> spark.read.text(\"test.text\").explain()\n== Physical Plan ==\n*FileScan text [value#15] Batched: false, Format: org.apache.spark.sql.execution.datasources.text.TextFileFormatxyz, Location: InMemoryFileIndex[file:/scratch/rxin/spark/test.text], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<value:string>\n```\n\nAfter:\n```\nscala> spark.read.text(\"test.text\").explain()\n== Physical Plan ==\n*FileScan text [value#15] Batched: false, Format: Text, Location: InMemoryFileIndex[file:/scratch/rxin/spark/test.text], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<value:string>\n```\n\nAlso closes #14680.\n\n## How was this patch tested?\nVerified in spark-shell.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #16187 from rxin/SPARK-18760.","distinct":true,"url":"https://api.github.com/repos/dautovri/spark/commits/5f894d23a54ea99f75f8b722e111e5270f7f80cf"},{"sha":"3261e25da39135f8e013b306953bc1fb7860065a","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"Close stale PRs.\n\nCloses #16191\nCloses #16198\nCloses #14561\nCloses #14223\nCloses #7739\nCloses #13026\nCloses #16217","distinct":true,"url":"https://api.github.com/repos/dautovri/spark/commits/3261e25da39135f8e013b306953bc1fb7860065a"},{"sha":"202fcd21ce01393fa6dfaa1c2126e18e9b85ee96","author":{"name":"Shivaram Venkataraman","email":"a88946ecf9c024d7b7eca4dac2be9c7e472cd2f5@cs.berkeley.edu"},"message":"[SPARK-18590][SPARKR] Change the R source build to Hadoop 2.6\n\nThis PR changes the SparkR source release tarball to be built using the Hadoop 2.6 profile. Previously it was using the without hadoop profile which leads to an error as discussed in https://github.com/apache/spark/pull/16014#issuecomment-265843991\n\nAuthor: Shivaram Venkataraman <shivaram@cs.berkeley.edu>\n\nCloses #16218 from shivaram/fix-sparkr-release-build.","distinct":true,"url":"https://api.github.com/repos/dautovri/spark/commits/202fcd21ce01393fa6dfaa1c2126e18e9b85ee96"},{"sha":"458fa3325e5f8c21c50e406ac8059d6236f93a9c","author":{"name":"Tathagata Das","email":"58b3b6773dfe925407947ae07f6b206d48277ab4@gmail.com"},"message":"[SPARK-18776][SS] Make Offset for FileStreamSource corrected formatted in json\n\n## What changes were proposed in this pull request?\n\n- Changed FileStreamSource to use new FileStreamSourceOffset rather than LongOffset. The field is named as `logOffset` to make it more clear that this is a offset in the file stream log.\n- Fixed bug in FileStreamSourceLog, the field endId in the FileStreamSourceLog.get(startId, endId) was not being used at all. No test caught it earlier. Only my updated tests caught it.\n\nOther minor changes\n- Dont use batchId in the FileStreamSource, as calling it batch id is extremely miss leading. With multiple sources, it may happen that a new batch has no new data from a file source. So offset of FileStreamSource != batchId after that batch.\n\n## How was this patch tested?\n\nUpdated unit test.\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #16205 from tdas/SPARK-18776.","distinct":true,"url":"https://api.github.com/repos/dautovri/spark/commits/458fa3325e5f8c21c50e406ac8059d6236f93a9c"},{"sha":"4ac8b20bf2f962d9b8b6b209468896758d49efe3","author":{"name":"Shivaram Venkataraman","email":"a88946ecf9c024d7b7eca4dac2be9c7e472cd2f5@cs.berkeley.edu"},"message":"[SPARKR][PYSPARK] Fix R source package name to match Spark version. Remove pip tar.gz from distribution\n\n## What changes were proposed in this pull request?\n\nFixes name of R source package so that the `cp` in release-build.sh works correctly.\n\nIssue discussed in https://github.com/apache/spark/pull/16014#issuecomment-265867125\n\nAuthor: Shivaram Venkataraman <shivaram@cs.berkeley.edu>\n\nCloses #16221 from shivaram/fix-sparkr-release-build-name.","distinct":true,"url":"https://api.github.com/repos/dautovri/spark/commits/4ac8b20bf2f962d9b8b6b209468896758d49efe3"},{"sha":"86a96034ccb47c5bba2cd739d793240afcfc25f6","author":{"name":"wm624@hotmail.com","email":"b6c6222622ee4c4512cfd0b24fc74f0642b6e399@hotmail.com"},"message":"[SPARK-18349][SPARKR] Update R API documentation on ml model summary\n\n## What changes were proposed in this pull request?\nIn this PR, the document of `summary` method is improved in the format:\n\nreturns summary information of the fitted model, which is a list. The list includes .......\n\nSince `summary` in R is mainly about the model, which is not the same as `summary` object on scala side, if there is one, the scala API doc is not pointed here.\n\nIn current document, some `return` have `.` and some don't have. `.` is added to missed ones.\n\nSince spark.logit `summary` has a big refactoring, this PR doesn't include this one. It will be changed when the `spark.logit` PR is merged.\n\n## How was this patch tested?\n\nManual build.\n\nAuthor: wm624@hotmail.com <wm624@hotmail.com>\n\nCloses #16150 from wangmiao1981/audit2.","distinct":true,"url":"https://api.github.com/repos/dautovri/spark/commits/86a96034ccb47c5bba2cd739d793240afcfc25f6"},{"sha":"9338aa4f89821c5640f7d007a67f9b947aa2bcd4","author":{"name":"Weiqing Yang","email":"c0c88f1153e01312d8f77a3074d15074c5d139c9@gmail.com"},"message":"[SPARK-18697][BUILD] Upgrade sbt plugins\n\n## What changes were proposed in this pull request?\n\nThis PR is to upgrade sbt plugins. The following sbt plugins will be upgraded:\n```\nsbteclipse-plugin: 4.0.0 -> 5.0.1\nsbt-mima-plugin: 0.1.11 -> 0.1.12\norg.ow2.asm/asm: 5.0.3 -> 5.1\norg.ow2.asm/asm-commons: 5.0.3 -> 5.1\n```\n## How was this patch tested?\nPass the Jenkins build.\n\nAuthor: Weiqing Yang <yangweiqing001@gmail.com>\n\nCloses #16223 from weiqingy/SPARK_18697.","distinct":true,"url":"https://api.github.com/repos/dautovri/spark/commits/9338aa4f89821c5640f7d007a67f9b947aa2bcd4"},{"sha":"934035ae7cb648fe61665d8efe0b7aa2bbe4ca47","author":{"name":"Shivaram Venkataraman","email":"a88946ecf9c024d7b7eca4dac2be9c7e472cd2f5@cs.berkeley.edu"},"message":"Copy the SparkR source package with LFTP\n\nThis PR adds a line in release-build.sh to copy the SparkR source archive using LFTP\n\nAuthor: Shivaram Venkataraman <shivaram@cs.berkeley.edu>\n\nCloses #16226 from shivaram/fix-sparkr-copy-build.","distinct":true,"url":"https://api.github.com/repos/dautovri/spark/commits/934035ae7cb648fe61665d8efe0b7aa2bbe4ca47"},{"sha":"c074c96dc57bf18b28fafdcac0c768d75c642cba","author":{"name":"Felix Cheung","email":"00eb71019ad7acfd67cff51fce666aad32ce6eed@hotmail.com"},"message":"Copy pyspark and SparkR packages to latest release dir too\n\n## What changes were proposed in this pull request?\n\nCopy pyspark and SparkR packages to latest release dir, as per comment [here](https://github.com/apache/spark/pull/16226#discussion_r91664822)\n\nAuthor: Felix Cheung <felixcheung_m@hotmail.com>\n\nCloses #16227 from felixcheung/pyrftp.","distinct":true,"url":"https://api.github.com/repos/dautovri/spark/commits/c074c96dc57bf18b28fafdcac0c768d75c642cba"},{"sha":"67587d961d5f94a8639c20cb80127c86bf79d5a8","author":{"name":"Zhan Zhang","email":"e965a14e36a6f6b1cb0a2060e7a12b97f12d2a62@fb.com"},"message":"[SPARK-18637][SQL] Stateful UDF should be considered as nondeterministic\n\n## What changes were proposed in this pull request?\n\nMake stateful udf as nondeterministic\n\n## How was this patch tested?\nAdd new test cases with both Stateful and Stateless UDF.\nWithout the patch, the test cases will throw exception:\n\n1 did not equal 10\nScalaTestFailureLocation: org.apache.spark.sql.hive.execution.HiveUDFSuite$$anonfun$21 at (HiveUDFSuite.scala:501)\norg.scalatest.exceptions.TestFailedException: 1 did not equal 10\n        at org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:500)\n        at org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1555)\n        ...\n\nAuthor: Zhan Zhang <zhanzhang@fb.com>\n\nCloses #16068 from zhzhan/state.","distinct":true,"url":"https://api.github.com/repos/dautovri/spark/commits/67587d961d5f94a8639c20cb80127c86bf79d5a8"}]}
{"actor":{"display_login":"dautovri"}}