{"push_id":1222993372,"size":119,"distinct_size":3,"ref":"refs/heads/branch-2.0-alti","head":"5c260ad0e03b5099a149834a993ca7bd3738149c","before":"f6b3e0901b9146a668786044690244958a437ed9","commits":[{"sha":"6e8fa86ebf30a9b850f4a66810d5d38d1f188b33","author":{"name":"Patrick Wendell","email":"dfc81a5a586cf6efa894b43abd4f9d9faf041679@gmail.com"},"message":"Preparing development version 2.0.1-SNAPSHOT","distinct":false,"url":"https://api.github.com/repos/Altiscale/spark/commits/6e8fa86ebf30a9b850f4a66810d5d38d1f188b33"},{"sha":"521fc7186a2637321f7a7cfac713537de73ae66f","author":{"name":"Tejas Patil","email":"6ad8f8971953bfaf2a65cdbf75ed01ef8f5bdc87@fb.com"},"message":"[SPARK-16339][CORE] ScriptTransform does not print stderr when outstream is lost\n\n## What changes were proposed in this pull request?\n\nCurrently, if due to some failure, the outstream gets destroyed or closed and later `outstream.close()` leads to IOException in such case. Due to this, the `stderrBuffer` does not get logged and there is no way for users to see why the job failed.\n\nThe change is to first display the stderr buffer and then try closing the outstream.\n\n## How was this patch tested?\n\nThe correct way to test this fix would be to grep the log to see if the `stderrBuffer` gets logged but I dont think having test cases which do that is a good idea.\n\n(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)\n\n…\n\nAuthor: Tejas Patil <tejasp@fb.com>\n\nCloses #13834 from tejasapatil/script_transform.\n\n(cherry picked from commit 5f342049cce9102fb62b4de2d8d8fa691c2e8ac4)\nSigned-off-by: Sean Owen <sowen@cloudera.com>","distinct":false,"url":"https://api.github.com/repos/Altiscale/spark/commits/521fc7186a2637321f7a7cfac713537de73ae66f"},{"sha":"25006c8bcc397c9f070cc5d685ffeb5b8fb0a341","author":{"name":"Yuhao Yang","email":"857769fa089c7426f7be1ac29063eaedcb246e23@intel.com"},"message":"[SPARK-16249][ML] Change visibility of Object ml.clustering.LDA to public for loading\n\n## What changes were proposed in this pull request?\njira: https://issues.apache.org/jira/browse/SPARK-16249\nChange visibility of Object ml.clustering.LDA to public for loading, thus users can invoke LDA.load(\"path\").\n\n## How was this patch tested?\n\nexisting ut and manually test for load ( saved with current code)\n\nAuthor: Yuhao Yang <yuhao.yang@intel.com>\nAuthor: Yuhao Yang <hhbyyh@gmail.com>\n\nCloses #13941 from hhbyyh/ldapublic.\n\n(cherry picked from commit 5497242c769b40338bfa57d64f2c64996dfa57e8)\nSigned-off-by: Yanbo Liang <ybliang8@gmail.com>","distinct":false,"url":"https://api.github.com/repos/Altiscale/spark/commits/25006c8bcc397c9f070cc5d685ffeb5b8fb0a341"},{"sha":"d5d2457e41661a3402a8258026856454222a2f54","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-15968][SQL] Nonempty partitioned metastore tables are not cached\n\nThis PR backports your fix (https://github.com/apache/spark/pull/13818) to branch 2.0.\n\nThis PR addresses [SPARK-15968](https://issues.apache.org/jira/browse/SPARK-15968).\n\n## What changes were proposed in this pull request?\n\nThe `getCached` method of [HiveMetastoreCatalog](https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala) computes `pathsInMetastore` from the metastore relation's catalog table. This only returns the table base path, which is incomplete/inaccurate for a nonempty partitioned table. As a result, cached lookups on nonempty partitioned tables always miss.\n\nRather than get `pathsInMetastore` from\n\n    metastoreRelation.catalogTable.storage.locationUri.toSeq\n\nI modified the `getCached` method to take a `pathsInMetastore` argument. Calls to this method pass in the paths computed from calls to the Hive metastore. This is how `getCached` was implemented in Spark 1.5:\n\nhttps://github.com/apache/spark/blob/e0c3212a9b42e3e704b070da4ac25b68c584427f/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala#L444.\n\nI also added a call in `InsertIntoHiveTable.scala` to invalidate the table from the SQL session catalog.\n\n## How was this patch tested?\n\nI've added a new unit test to `parquetSuites.scala`:\n\n    SPARK-15968: nonempty partitioned metastore Parquet table lookup should use cached relation\n\nNote that the only difference between this new test and the one above it in the file is that the new test populates its partitioned table with a single value, while the existing test leaves the table empty. This reveals a subtle, unexpected hole in test coverage present before this patch.\n\nNote I also modified a different but related unit test in `parquetSuites.scala`:\n\n    SPARK-15248: explicitly added partitions should be readable\n\nThis unit test asserts that Spark SQL should return data from a table partition which has been placed there outside a metastore query immediately after it is added. I changed the test so that, instead of adding the data as a parquet file saved in the partition's location, the data is added through a SQL `INSERT` query. I made this change because I could find no way to efficiently support partitioned table caching without failing that test.\n\nIn addition to my primary motivation, I can offer a few reasons I believe this is an acceptable weakening of that test. First, it still validates a fix for [SPARK-15248](https://issues.apache.org/jira/browse/SPARK-15248), the issue for which it was written. Second, the assertion made is stronger than that required for non-partitioned tables. If you write data to the storage location of a non-partitioned metastore table without using a proper SQL DML query, a subsequent call to show that data will not return it. I believe this is an intentional limitation put in place to make table caching feasible, but I'm only speculating.\n\nBuilding a large `HadoopFsRelation` requires `stat`-ing all of its data files. In our environment, where we have tables with 10's of thousands of partitions, the difference between using a cached relation versus a new one is a matter of seconds versus minutes. Caching partitioned table metadata vastly improves the usability of Spark SQL for these cases.\n\nAuthor: Reynold Xin <rxin@databricks.com>\nAuthor: Michael Allman <michael@videoamp.com>\n\nCloses #14064 from yhuai/spark-15968-branch-2.0.","distinct":false,"url":"https://api.github.com/repos/Altiscale/spark/commits/d5d2457e41661a3402a8258026856454222a2f54"},{"sha":"e956bd7750882ce259a278e9eac7f64b4fb76286","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-16229][SQL] Drop Empty Table After CREATE TABLE AS SELECT fails\n\n#### What changes were proposed in this pull request?\nIn `CREATE TABLE AS SELECT`, if the `SELECT` query failed, the table should not exist. For example,\n\n```SQL\nCREATE TABLE tab\nSTORED AS TEXTFILE\nSELECT 1 AS a, (SELECT a FROM (SELECT 1 AS a UNION ALL SELECT 2 AS a) t) AS b\n```\nThe above query failed as expected but an empty table `t` is created.\n\nThis PR is to drop the created table when hitting any non-fatal exception.\n\n#### How was this patch tested?\nAdded a test case to verify the behavior\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #13926 from gatorsmile/dropTableAfterException.\n\n(cherry picked from commit 21eadd1d8cbf029197e73ffca1cba54d5a890c01)\nSigned-off-by: Wenchen Fan <wenchen@databricks.com>","distinct":false,"url":"https://api.github.com/repos/Altiscale/spark/commits/e956bd7750882ce259a278e9eac7f64b4fb76286"},{"sha":"091cd5f265166512a450333946c62c3eb3440e79","author":{"name":"WeichenXu","email":"7a440c883b23bc43b7a6fdb423cbfd8e75175a83@outlook.com"},"message":"[DOC][SQL] update out-of-date code snippets using SQLContext in all documents.\n\n## What changes were proposed in this pull request?\n\nI search the whole documents directory using SQLContext, and update the following places:\n\n- docs/configuration.md, sparkR code snippets.\n- docs/streaming-programming-guide.md, several example code.\n\n## How was this patch tested?\n\nN/A\n\nAuthor: WeichenXu <WeichenXu123@outlook.com>\n\nCloses #14025 from WeichenXu123/WIP_SQLContext_update.\n\n(cherry picked from commit b1310425b30cbd711e4834d65a0accb3c5a8403a)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":false,"url":"https://api.github.com/repos/Altiscale/spark/commits/091cd5f265166512a450333946c62c3eb3440e79"},{"sha":"03f336d8921e1f22ee4d1f6fa8869163b1f29ea9","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[MINOR][PYSPARK][DOC] Fix wrongly formatted examples in PySpark documentation\n\n## What changes were proposed in this pull request?\n\nThis PR fixes wrongly formatted examples in PySpark documentation as below:\n\n- **`SparkSession`**\n\n  - **Before**\n\n    ![2016-07-06 11 34 41](https://cloud.githubusercontent.com/assets/6477701/16605847/ae939526-436d-11e6-8ab8-6ad578362425.png)\n\n  - **After**\n\n    ![2016-07-06 11 33 56](https://cloud.githubusercontent.com/assets/6477701/16605845/ace9ee78-436d-11e6-8923-b76d4fc3e7c3.png)\n\n- **`Builder`**\n\n  - **Before**\n    ![2016-07-06 11 34 44](https://cloud.githubusercontent.com/assets/6477701/16605844/aba60dbc-436d-11e6-990a-c87bc0281c6b.png)\n\n  - **After**\n    ![2016-07-06 1 26 37](https://cloud.githubusercontent.com/assets/6477701/16607562/586704c0-437d-11e6-9483-e0af93d8f74e.png)\n\nThis PR also fixes several similar instances across the documentation in `sql` PySpark module.\n\n## How was this patch tested?\n\nN/A\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #14063 from HyukjinKwon/minor-pyspark-builder.\n\n(cherry picked from commit 4e14199ff740ea186eb2cec2e5cf901b58c5f90e)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":false,"url":"https://api.github.com/repos/Altiscale/spark/commits/03f336d8921e1f22ee4d1f6fa8869163b1f29ea9"},{"sha":"2465f0728e95109ab851ab09b5badd697928ba2b","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-16371][SQL] Do not push down filters incorrectly when inner name and outer name are the same in Parquet\n\n## What changes were proposed in this pull request?\n\nCurrently, if there is a schema as below:\n\n```\nroot\n  |-- _1: struct (nullable = true)\n  |    |-- _1: integer (nullable = true)\n```\n\nand if we execute the codes below:\n\n```scala\ndf.filter(\"_1 IS NOT NULL\").count()\n```\n\nThis pushes down a filter although this filter is being applied to `StructType`.(If my understanding is correct, Spark does not pushes down filters for those).\n\nThe reason is, `ParquetFilters.getFieldMap` produces results below:\n\n```\n(_1,StructType(StructField(_1,IntegerType,true)))\n(_1,IntegerType)\n```\n\nand then it becomes a `Map`\n\n```\n(_1,IntegerType)\n```\n\nNow, because of ` ....lift(dataTypeOf(name)).map(_(name, value))`, this pushes down filters for `_1` which Parquet thinks is `IntegerType`. However, it is actually `StructType`.\n\nSo, Parquet filter2 produces incorrect results, for example, the codes below:\n\n```\ndf.filter(\"_1 IS NOT NULL\").count()\n```\n\nproduces always 0.\n\nThis PR prevents this by not finding nested fields.\n\n## How was this patch tested?\n\nUnit test in `ParquetFilterSuite`.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #14067 from HyukjinKwon/SPARK-16371.\n\n(cherry picked from commit 4f8ceed59367319300e4bfa5b957c387be81ffa3)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":false,"url":"https://api.github.com/repos/Altiscale/spark/commits/2465f0728e95109ab851ab09b5badd697928ba2b"},{"sha":"d7926da5e72ee2015e3ebe39a5fd0b322e9d1334","author":{"name":"tmnd1991","email":"445e5c80fecd10c2370c17d3256340eaba5b8c1c@studio.unibo.it"},"message":"[SPARK-15740][MLLIB] Word2VecSuite \"big model load / save\" caused OOM in maven jenkins builds\n\n## What changes were proposed in this pull request?\n\"test big model load / save\" in Word2VecSuite, lately resulted into OOM.\nTherefore we decided to make the partitioning adaptive (not based on spark default \"spark.kryoserializer.buffer.max\" conf) and then testing it using a small buffer size in order to trigger partitioning without allocating too much memory for the test.\n\n## How was this patch tested?\nIt was tested running the following unit test:\norg.apache.spark.mllib.feature.Word2VecSuite\n\nAuthor: tmnd1991 <antonio.murgia2@studio.unibo.it>\n\nCloses #13509 from tmnd1991/SPARK-15740.\n\n(cherry picked from commit 040f6f9f468f153e4c4db78c26ced0299245fb6f)\nSigned-off-by: Joseph K. Bradley <joseph@databricks.com>","distinct":false,"url":"https://api.github.com/repos/Altiscale/spark/commits/d7926da5e72ee2015e3ebe39a5fd0b322e9d1334"},{"sha":"88be66b933a7b1f0f71b1eb6c88bf01ecbf1923c","author":{"name":"Sean Owen","email":"4c59f357315334980589bde1049040bf6de6532c@cloudera.com"},"message":"[SPARK-16379][CORE][MESOS] Spark on mesos is broken due to race condition in Logging\n\n## What changes were proposed in this pull request?\n\nThe commit https://github.com/apache/spark/commit/044971eca0ff3c2ce62afa665dbd3072d52cbbec introduced a lazy val to simplify code in Logging. Simple enough, though one side effect is that accessing log now means grabbing the instance's lock. This in turn turned up a form of deadlock in the Mesos code. It was arguably a bit of a problem in how this code is structured, but, in any event the safest thing to do seems to be to revert the commit, and that's 90% of the change here; it's just not worth the risk of similar more subtle issues.\n\nWhat I didn't revert here was the removal of this odd override of log in the Mesos code. In retrospect it might have been put in place at some stage as a defense against this type of problem. After all the Logging code still involved a lock at initialization before the change in question.\n\nEven after the revert, it doesn't seem like it does anything, given how Logging works now, so I left it removed. However, I also removed the particular log message that ended up playing a part in this problem anyway, maybe being paranoid, to make sure this type of problem can't happen even with how the current locking works in logging initialization.\n\n## How was this patch tested?\n\nJenkins tests\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #14069 from srowen/SPARK-16379.\n\n(cherry picked from commit a8f89df3b391e7a3fa9f73d9ec730d6eaa95bb09)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":false,"url":"https://api.github.com/repos/Altiscale/spark/commits/88be66b933a7b1f0f71b1eb6c88bf01ecbf1923c"},{"sha":"2c2b8f121a213618ef47cb030d17b9bd323f0d9e","author":{"name":"Michael Gummelt","email":"997b1eeb9667009a1fe95939cfe9ef99c4f83395@mesosphere.io"},"message":"[MESOS] expand coarse-grained mode docs\n\n## What changes were proposed in this pull request?\n\ndocs\n\n## How was this patch tested?\n\nviewed the docs in github\n\nAuthor: Michael Gummelt <mgummelt@mesosphere.io>\n\nCloses #14059 from mgummelt/coarse-grained.\n\n(cherry picked from commit 9c041990cf4d0138d9104207b5c2e7a319b42615)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":false,"url":"https://api.github.com/repos/Altiscale/spark/commits/2c2b8f121a213618ef47cb030d17b9bd323f0d9e"},{"sha":"05ddc75179acc582c615da01b9c0e7e049a5ecf0","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-16371][SQL] Two follow-up tasks\n\n## What changes were proposed in this pull request?\nThis is a small follow-up for SPARK-16371:\n\n1. Hide removeMetadata from public API.\n2. Add JIRA ticket number to test case name.\n\n## How was this patch tested?\nUpdated a test comment.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #14074 from rxin/parquet-filter.\n\n(cherry picked from commit 8e3e4ed6c090d18675d49eec46b3ee572457db95)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":false,"url":"https://api.github.com/repos/Altiscale/spark/commits/05ddc75179acc582c615da01b9c0e7e049a5ecf0"},{"sha":"920162a1e0b43b558ba2242868a44cad06bef946","author":{"name":"cody koeninger","email":"7e9fd717623d5c1ae8b16d280e7174a66cf3cec7@koeninger.org"},"message":"[SPARK-16212][STREAMING][KAFKA] apply test tweaks from 0-10 to 0-8 as well\n\n## What changes were proposed in this pull request?\n\nBring the kafka-0-8 subproject up to date with some test modifications from development on 0-10.\n\nMain changes are\n- eliminating waits on concurrent queue in favor of an assert on received results,\n- atomics instead of volatile (although this probably doesn't matter)\n- increasing uniqueness of topic names\n\n## How was this patch tested?\n\nUnit tests\n\nAuthor: cody koeninger <cody@koeninger.org>\n\nCloses #14073 from koeninger/kafka-0-8-test-direct-cleanup.\n\n(cherry picked from commit b8ebf63c1e1fa1ab53ea760fa293051c08ce5f59)\nSigned-off-by: Tathagata Das <tathagata.das1565@gmail.com>","distinct":false,"url":"https://api.github.com/repos/Altiscale/spark/commits/920162a1e0b43b558ba2242868a44cad06bef946"},{"sha":"d63428af6d1c7c0a0533567a0a7ccb5817a65de3","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-16368][SQL] Fix Strange Errors When Creating View With Unmatched Column Num\n\n#### What changes were proposed in this pull request?\nWhen creating a view, a common user error is the number of columns produced by the `SELECT` clause does not match the number of column names specified by `CREATE VIEW`.\n\nFor example, given Table `t1` only has 3 columns\n```SQL\ncreate view v1(col2, col4, col3, col5) as select * from t1\n```\nCurrently, Spark SQL reports the following error:\n```\nrequirement failed\njava.lang.IllegalArgumentException: requirement failed\n\tat scala.Predef$.require(Predef.scala:212)\n\tat org.apache.spark.sql.execution.command.CreateViewCommand.run(views.scala:90)\n```\n\nThis error message is very confusing. This PR is to detect the error and issue a meaningful error message.\n\n#### How was this patch tested?\nAdded test cases\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #14047 from gatorsmile/viewMismatchedColumns.\n\n(cherry picked from commit ab05db0b48f395543cd7d91e2ad9dd760516868b)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":false,"url":"https://api.github.com/repos/Altiscale/spark/commits/d63428af6d1c7c0a0533567a0a7ccb5817a65de3"},{"sha":"24933355c7211bbf6bc5bebfad91ed783d6b6a51","author":{"name":"Xusen Yin","email":"b3af76889e6bb255e66b77722a2ef0a4409760be@gmail.com"},"message":"[SPARK-16372][MLLIB] Retag RDD to tallSkinnyQR of RowMatrix\n\n## What changes were proposed in this pull request?\n\nThe following Java code because of type erasing:\n\n```Java\nJavaRDD<Vector> rows = jsc.parallelize(...);\nRowMatrix mat = new RowMatrix(rows.rdd());\nQRDecomposition<RowMatrix, Matrix> result = mat.tallSkinnyQR(true);\n```\n\nWe should use retag to restore the type to prevent the following exception:\n\n```Java\njava.lang.ClassCastException: [Ljava.lang.Object; cannot be cast to [Lorg.apache.spark.mllib.linalg.Vector;\n```\n\n## How was this patch tested?\n\nJava unit test\n\nAuthor: Xusen Yin <yinxusen@gmail.com>\n\nCloses #14051 from yinxusen/SPARK-16372.\n\n(cherry picked from commit 4c6f00d09c016dfc1d2de6e694dff219c9027fa0)\nSigned-off-by: Sean Owen <sowen@cloudera.com>","distinct":false,"url":"https://api.github.com/repos/Altiscale/spark/commits/24933355c7211bbf6bc5bebfad91ed783d6b6a51"},{"sha":"cbfd94eacf46b61011f1bd8d30f0c134cab37b09","author":{"name":"Liwei Lin","email":"878182135bcb2086b9b1c72277449d6e06bbf76f@gmail.com"},"message":"[SPARK-16350][SQL] Fix support for incremental planning in wirteStream.foreach()\n\n## What changes were proposed in this pull request?\n\nThere are cases where `complete` output mode does not output updated aggregated value; for details please refer to [SPARK-16350](https://issues.apache.org/jira/browse/SPARK-16350).\n\nThe cause is that, as we do `data.as[T].foreachPartition { iter => ... }` in `ForeachSink.addBatch()`, `foreachPartition()` does not support incremental planning for now.\n\nThis patches makes `foreachPartition()` support incremental planning in `ForeachSink`, by making a special version of `Dataset` with its `rdd()` method supporting incremental planning.\n\n## How was this patch tested?\n\nAdded a unit test which failed before the change\n\nAuthor: Liwei Lin <lwlin7@gmail.com>\n\nCloses #14030 from lw-lin/fix-foreach-complete.\n\n(cherry picked from commit 0f7175def985a7f1e37198680f893e749612ab76)\nSigned-off-by: Shixiong Zhu <shixiong@databricks.com>","distinct":false,"url":"https://api.github.com/repos/Altiscale/spark/commits/cbfd94eacf46b61011f1bd8d30f0c134cab37b09"},{"sha":"30cb3f1d3a1d413568d586e6b8df56f74f05d80e","author":{"name":"Daoyuan Wang","email":"ca9858c9dc0e8e1b4a4756052ed9da2bddb5bfdf@intel.com"},"message":"[SPARK-16415][SQL] fix catalog string error\n\n## What changes were proposed in this pull request?\n\nIn #13537 we truncate `simpleString` if it is a long `StructType`. But sometimes we need `catalogString` to reconstruct `TypeInfo`, for example in description of [SPARK-16415 ](https://issues.apache.org/jira/browse/SPARK-16415). So we need to keep the implementation of `catalogString` not affected by our truncate.\n\n## How was this patch tested?\n\nadded a test case.\n\nAuthor: Daoyuan Wang <daoyuan.wang@intel.com>\n\nCloses #14089 from adrian-wang/catalogstring.\n\n(cherry picked from commit 28710b42b0d18a55bd64d597558649537259b127)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":false,"url":"https://api.github.com/repos/Altiscale/spark/commits/30cb3f1d3a1d413568d586e6b8df56f74f05d80e"},{"sha":"5828da41cb2d815708191bd9a5cf3bd82795aa41","author":{"name":"Felix Cheung","email":"00eb71019ad7acfd67cff51fce666aad32ce6eed@hotmail.com"},"message":"[SPARK-16310][SPARKR] R na.string-like default for csv source\n\n## What changes were proposed in this pull request?\n\nApply default \"NA\" as null string for R, like R read.csv na.string parameter.\n\nhttps://stat.ethz.ch/R-manual/R-devel/library/utils/html/read.table.html\nna.strings = \"NA\"\n\nAn user passing a csv file with NA value should get the same behavior with SparkR read.df(... source = \"csv\")\n\n(couldn't open JIRA, will do that later)\n\n## How was this patch tested?\n\nunit tests\n\nshivaram\n\nAuthor: Felix Cheung <felixcheung_m@hotmail.com>\n\nCloses #13984 from felixcheung/rcsvnastring.\n\n(cherry picked from commit f4767bcc7a9d1bdd301f054776aa45e7c9f344a7)\nSigned-off-by: Shivaram Venkataraman <shivaram@cs.berkeley.edu>","distinct":false,"url":"https://api.github.com/repos/Altiscale/spark/commits/5828da41cb2d815708191bd9a5cf3bd82795aa41"},{"sha":"73c764a047f795c85909c7a7ea4324f286d2aafa","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-16425][R] `describe()` should not fail with non-numeric columns\n\n## What changes were proposed in this pull request?\n\nThis PR prevents ERRORs when `summary(df)` is called for `SparkDataFrame` with not-numeric columns. This failure happens only in `SparkR`.\n\n**Before**\n```r\n> df <- createDataFrame(faithful)\n> df <- withColumn(df, \"boolean\", df$waiting==79)\n> summary(df)\n16/07/07 14:15:16 ERROR RBackendHandler: describe on 34 failed\nError in invokeJava(isStatic = FALSE, objId$id, methodName, ...) :\n  org.apache.spark.sql.AnalysisException: cannot resolve 'avg(`boolean`)' due to data type mismatch: function average requires numeric types, not BooleanType;\n```\n\n**After**\n```r\n> df <- createDataFrame(faithful)\n> df <- withColumn(df, \"boolean\", df$waiting==79)\n> summary(df)\nSparkDataFrame[summary:string, eruptions:string, waiting:string]\n```\n\n## How was this patch tested?\n\nPass the Jenkins with a updated testcase.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #14096 from dongjoon-hyun/SPARK-16425.\n\n(cherry picked from commit 6aa7d09f4e126f42e41085dec169c813379ed354)\nSigned-off-by: Shivaram Venkataraman <shivaram@cs.berkeley.edu>","distinct":false,"url":"https://api.github.com/repos/Altiscale/spark/commits/73c764a047f795c85909c7a7ea4324f286d2aafa"},{"sha":"88603bd4f9a665ad02df40ed8a0dd78b65c9d152","author":{"name":"petermaxlee","email":"ac6c8937256e5ff211210b88da2643bdd0703390@gmail.com"},"message":"[SPARK-16276][SQL] Implement elt SQL function\n\nThis patch implements the elt function, as it is implemented in Hive.\n\nAdded expression unit test in StringExpressionsSuite and end-to-end test in StringFunctionsSuite.\n\nAuthor: petermaxlee <petermaxlee@gmail.com>\n\nCloses #13966 from petermaxlee/SPARK-16276.\n\n(cherry picked from commit 85f2303ecadd9bf6d9694a2743dda075654c5ccf)\nSigned-off-by: Reynold Xin <rxin@databricks.com>","distinct":false,"url":"https://api.github.com/repos/Altiscale/spark/commits/88603bd4f9a665ad02df40ed8a0dd78b65c9d152"}]}
{"actor":{"display_login":"alee-altiscale"}}