{"push_id":1380435540,"size":25,"distinct_size":25,"ref":"refs/heads/master","head":"85c5424d466f4a5765c825e0e2ab30da97611285","before":"dd85eb5448c8f2672260b57e94c0da0eaac12616","commits":[{"sha":"623fc7fc67735cfafdb7f527bd3df210987943c6","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[MINOR][DOC] Remove spaces following slashs\n\n## What changes were proposed in this pull request?\n\nThis PR merges multiple lines enumerating items in order to remove the redundant spaces following slashes in [Structured Streaming Programming Guide in 2.0.2-rc1](http://people.apache.org/~pwendell/spark-releases/spark-2.0.2-rc1-docs/structured-streaming-programming-guide.html).\n- Before: `Scala/ Java/ Python`\n- After: `Scala/Java/Python`\n## How was this patch tested?\n\nManual by the followings because this is documentation update.\n\n```\ncd docs\nSKIP_API=1 jekyll build\n```\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #15686 from dongjoon-hyun/minor_doc_space.","distinct":true,"url":"https://api.github.com/repos/lw-lin/spark/commits/623fc7fc67735cfafdb7f527bd3df210987943c6"},{"sha":"cb80edc26349e2e358d27fe2ae8e5d6959b77fab","author":{"name":"wangzhenhua","email":"93943f521ced4ea3f21d64a737435e7a7e171ac6@huawei.com"},"message":"[SPARK-18111][SQL] Wrong ApproximatePercentile answer when multiple records have the minimum value\n\n## What changes were proposed in this pull request?\n\nWhen multiple records have the minimum value, the answer of ApproximatePercentile is wrong.\n## How was this patch tested?\n\nadd a test case\n\nAuthor: wangzhenhua <wangzhenhua@huawei.com>\n\nCloses #15641 from wzhfy/percentile.","distinct":true,"url":"https://api.github.com/repos/lw-lin/spark/commits/cb80edc26349e2e358d27fe2ae8e5d6959b77fab"},{"sha":"e34b4e12673fb76c92f661d7c03527410857a0f8","author":{"name":"Charles Allen","email":"74964f4380c66ef5e4e7b490d8cdccec939315ab@allen-net.com"},"message":"[SPARK-15994][MESOS] Allow enabling Mesos fetch cache in coarse executor backend\n\nMesos 0.23.0 introduces a Fetch Cache feature http://mesos.apache.org/documentation/latest/fetcher/ which allows caching of resources specified in command URIs.\n\nThis patch:\n- Updates the Mesos shaded protobuf dependency to 0.23.0\n- Allows setting `spark.mesos.fetcherCache.enable` to enable the fetch cache for all specified URIs. (URIs must be specified for the setting to have any affect)\n- Updates documentation for Mesos configuration with the new setting.\n\nThis patch does NOT:\n- Allow for per-URI caching configuration. The cache setting is global to ALL URIs for the command.\n\nAuthor: Charles Allen <charles@allen-net.com>\n\nCloses #13713 from drcrallen/SPARK15994.","distinct":true,"url":"https://api.github.com/repos/lw-lin/spark/commits/e34b4e12673fb76c92f661d7c03527410857a0f8"},{"sha":"ec6f479bb1d14c9eb45e0418353007be0416e4c5","author":{"name":"Sandeep Singh","email":"69043e619d79bcf4ae068e5b131d66c33e73727d@techaddict.me"},"message":"[SPARK-16881][MESOS] Migrate Mesos configs to use ConfigEntry\n\n## What changes were proposed in this pull request?\n\nMigrate Mesos configs to use ConfigEntry\n## How was this patch tested?\n\nJenkins Tests\n\nAuthor: Sandeep Singh <sandeep@techaddict.me>\n\nCloses #15654 from techaddict/SPARK-16881.","distinct":true,"url":"https://api.github.com/repos/lw-lin/spark/commits/ec6f479bb1d14c9eb45e0418353007be0416e4c5"},{"sha":"9b377aa49f14af31f54164378d60e0fdea2142e5","author":{"name":"Wang Lei","email":"2ae6f49ce33307a941f16d81e58cb429f0aaf78d@kongming-inc.com"},"message":"[SPARK-18114][MESOS] Fix mesos cluster scheduler generage command option error\n\n## What changes were proposed in this pull request?\n\nEnclose --conf option value with \"\" to support multi value configs like spark.driver.extraJavaOptions, without \"\", driver will fail to start.\n## How was this patch tested?\n\nJenkins Tests.\n\nTest in our production environment, also unit tests, It is a very small change.\n\nAuthor: Wang Lei <lei.wang@kongming-inc.com>\n\nCloses #15643 from LeightonWong/messos-cluster.","distinct":true,"url":"https://api.github.com/repos/lw-lin/spark/commits/9b377aa49f14af31f54164378d60e0fdea2142e5"},{"sha":"f7c145d8ce14b23019099c509d5a2b6dfb1fe62c","author":{"name":"Herman van Hovell","email":"38f32192f78c1df35671ebb666a8ae224cdccefb@databricks.com"},"message":"[SPARK-17996][SQL] Fix unqualified catalog.getFunction(...)\n\n## What changes were proposed in this pull request?\n\nCurrently an unqualified `getFunction(..)`call returns a wrong result; the returned function is shown as temporary function without a database. For example:\n\n```\nscala> sql(\"create function fn1 as 'org.apache.hadoop.hive.ql.udf.generic.GenericUDFAbs'\")\nres0: org.apache.spark.sql.DataFrame = []\n\nscala> spark.catalog.getFunction(\"fn1\")\nres1: org.apache.spark.sql.catalog.Function = Function[name='fn1', className='org.apache.hadoop.hive.ql.udf.generic.GenericUDFAbs', isTemporary='true']\n```\n\nThis PR fixes this by adding database information to ExpressionInfo (which is used to store the function information).\n## How was this patch tested?\n\nAdded more thorough tests to `CatalogSuite`.\n\nAuthor: Herman van Hovell <hvanhovell@databricks.com>\n\nCloses #15542 from hvanhovell/SPARK-17996.","distinct":true,"url":"https://api.github.com/repos/lw-lin/spark/commits/f7c145d8ce14b23019099c509d5a2b6dfb1fe62c"},{"sha":"5441a6269e00e3903ae6c1ea8deb4ddf3d2e9975","author":{"name":"eyal farago","email":"74d0e2451c1814707187b75c7c67acf59bad9214"},"message":"[SPARK-16839][SQL] redundant aliases after cleanupAliases\n\n## What changes were proposed in this pull request?\n\nSimplify struct creation, especially the aspect of `CleanupAliases` which missed some aliases when handling trees created by `CreateStruct`.\n\nThis PR includes:\n\n1. A failing test (create struct with nested aliases, some of the aliases survive `CleanupAliases`).\n2. A fix that transforms `CreateStruct` into a `CreateNamedStruct` constructor, effectively eliminating `CreateStruct` from all expression trees.\n3. A `NamePlaceHolder` used by `CreateStruct` when column names cannot be extracted from unresolved `NamedExpression`.\n4. A new Analyzer rule that resolves `NamePlaceHolder` into a string literal once the `NamedExpression` is resolved.\n5. `CleanupAliases` code was simplified as it no longer has to deal with `CreateStruct`'s top level columns.\n\n## How was this patch tested?\n\nrunning all tests-suits in package org.apache.spark.sql, especially including the analysis suite, making sure added test initially fails, after applying suggested fix rerun the entire analysis package successfully.\n\nmodified few tests that expected `CreateStruct` which is now transformed into `CreateNamedStruct`.\n\nCredit goes to hvanhovell for assisting with this PR.\n\nAuthor: eyal farago <eyal farago>\nAuthor: eyal farago <eyal.farago@gmail.com>\nAuthor: Herman van Hovell <hvanhovell@databricks.com>\nAuthor: Eyal Farago <eyal.farago@actimize.com>\nAuthor: Hyukjin Kwon <gurwls223@gmail.com>\nAuthor: eyalfa <eyal.farago@gmail.com>\n\nCloses #14444 from eyalfa/SPARK-16839_redundant_aliases_after_cleanupAliases.","distinct":true,"url":"https://api.github.com/repos/lw-lin/spark/commits/5441a6269e00e3903ae6c1ea8deb4ddf3d2e9975"},{"sha":"0cba535af3c65618f342fa2d7db9647f5e6f6f1b","author":{"name":"Herman van Hovell","email":"38f32192f78c1df35671ebb666a8ae224cdccefb@databricks.com"},"message":"Revert \"[SPARK-16839][SQL] redundant aliases after cleanupAliases\"\n\nThis reverts commit 5441a6269e00e3903ae6c1ea8deb4ddf3d2e9975.","distinct":true,"url":"https://api.github.com/repos/lw-lin/spark/commits/0cba535af3c65618f342fa2d7db9647f5e6f6f1b"},{"sha":"8ac09108fcf3fb62a812333a5b386b566a9d98ec","author":{"name":"Zheng RuiFeng","email":"688fbc7eb0ddb0dce0ccb704b29547c7b3cab719@foxmail.com"},"message":"[SPARK-17848][ML] Move LabelCol datatype cast into Predictor.fit\n\n## What changes were proposed in this pull request?\n\n1, move cast to `Predictor`\n2, and then, remove unnecessary cast\n## How was this patch tested?\n\nexisting tests\n\nAuthor: Zheng RuiFeng <ruifengz@foxmail.com>\n\nCloses #15414 from zhengruifeng/move_cast.","distinct":true,"url":"https://api.github.com/repos/lw-lin/spark/commits/8ac09108fcf3fb62a812333a5b386b566a9d98ec"},{"sha":"8cdf143f4b1ca5c6bc0256808e6f42d9ef299cbd","author":{"name":"Liwei Lin","email":"878182135bcb2086b9b1c72277449d6e06bbf76f@gmail.com"},"message":"[SPARK-18103][FOLLOW-UP][SQL][MINOR] Rename `MetadataLogFileCatalog` to `MetadataLogFileIndex`\n\n## What changes were proposed in this pull request?\n\nThis is a follow-up to https://github.com/apache/spark/pull/15634.\n\n## How was this patch tested?\n\nN/A\n\nAuthor: Liwei Lin <lwlin7@gmail.com>\n\nCloses #15712 from lw-lin/18103.","distinct":true,"url":"https://api.github.com/repos/lw-lin/spark/commits/8cdf143f4b1ca5c6bc0256808e6f42d9ef299cbd"},{"sha":"8a538c97b556f80f67c80519af0ce879557050d5","author":{"name":"Ergin Seyfe","email":"551bf3c71d9da660a52ecd3224fc4c86679b46d0@fb.com"},"message":"[SPARK-18189][SQL] Fix serialization issue in KeyValueGroupedDataset\n\n## What changes were proposed in this pull request?\nLikewise [DataSet.scala](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L156) KeyValueGroupedDataset should mark the queryExecution as transient.\n\nAs mentioned in the Jira ticket, without transient we saw serialization issues like\n\n```\nCaused by: java.io.NotSerializableException: org.apache.spark.sql.execution.QueryExecution\nSerialization stack:\n        - object not serializable (class: org.apache.spark.sql.execution.QueryExecution, value: ==\n```\n\n## How was this patch tested?\n\nRun the query which is specified in the Jira ticket before and after:\n```\nval a = spark.createDataFrame(sc.parallelize(Seq((1,2),(3,4)))).as[(Int,Int)]\nval grouped = a.groupByKey(\n{x:(Int,Int)=>x._1}\n)\nval mappedGroups = grouped.mapGroups((k,x)=>\n{(k,1)}\n)\nval yyy = sc.broadcast(1)\nval last = mappedGroups.rdd.map(xx=>\n{ val simpley = yyy.value 1 }\n)\n```\n\nAuthor: Ergin Seyfe <eseyfe@fb.com>\n\nCloses #15706 from seyfe/keyvaluegrouped_serialization.","distinct":true,"url":"https://api.github.com/repos/lw-lin/spark/commits/8a538c97b556f80f67c80519af0ce879557050d5"},{"sha":"d0272b436512b71f04313e109d3d21a6e9deefca","author":{"name":"jiangxingbo","email":"5b9d05a325a3f85d4c824395342a210596784248@gmail.com"},"message":"[SPARK-18148][SQL] Misleading Error Message for Aggregation Without Window/GroupBy\n\n## What changes were proposed in this pull request?\n\nAggregation Without Window/GroupBy expressions will fail in `checkAnalysis`, the error message is a bit misleading, we should generate a more specific error message for this case.\n\nFor example,\n\n```\nspark.read.load(\"/some-data\")\n  .withColumn(\"date_dt\", to_date($\"date\"))\n  .withColumn(\"year\", year($\"date_dt\"))\n  .withColumn(\"week\", weekofyear($\"date_dt\"))\n  .withColumn(\"user_count\", count($\"userId\"))\n  .withColumn(\"daily_max_in_week\", max($\"user_count\").over(weeklyWindow))\n)\n```\n\ncreates the following output:\n\n```\norg.apache.spark.sql.AnalysisException: expression '`randomColumn`' is neither present in the group by, nor is it an aggregate function. Add to group by or wrap in first() (or first_value) if you don't care which value you get.;\n```\n\nIn the error message above, `randomColumn` doesn't appear in the query(acturally it's added by function `withColumn`), so the message is not enough for the user to address the problem.\n## How was this patch tested?\n\nManually test\n\nBefore:\n\n```\nscala> spark.sql(\"select col, count(col) from tbl\")\norg.apache.spark.sql.AnalysisException: expression 'tbl.`col`' is neither present in the group by, nor is it an aggregate function. Add to group by or wrap in first() (or first_value) if you don't care which value you get.;;\n```\n\nAfter:\n\n```\nscala> spark.sql(\"select col, count(col) from tbl\")\norg.apache.spark.sql.AnalysisException: grouping expressions sequence is empty, and 'tbl.`col`' is not an aggregate function. Wrap '(count(col#231L) AS count(col)#239L)' in windowing function(s) or wrap 'tbl.`col`' in first() (or first_value) if you don't care which value you get.;;\n```\n\nAlso add new test sqls in `group-by.sql`.\n\nAuthor: jiangxingbo <jiangxb1987@gmail.com>\n\nCloses #15672 from jiangxb1987/groupBy-empty.","distinct":true,"url":"https://api.github.com/repos/lw-lin/spark/commits/d0272b436512b71f04313e109d3d21a6e9deefca"},{"sha":"cfac17ee1cec414663b957228e469869eb7673c1","author":{"name":"Eric Liang","email":"8825d93f7d4ea34a2c506e18a7ac7514d6509e7d@databricks.com"},"message":"[SPARK-18167] Disable flaky SQLQuerySuite test\n\nWe now know it's a persistent environmental issue that is causing this test to sometimes fail. One hypothesis is that some configuration is leaked from another suite, and depending on suite ordering this can cause this test to fail.\n\nI am planning on mining the jenkins logs to try to narrow down which suite could be causing this. For now, disable the test.\n\nAuthor: Eric Liang <ekl@databricks.com>\n\nCloses #15720 from ericl/disable-flaky-test.","distinct":true,"url":"https://api.github.com/repos/lw-lin/spark/commits/cfac17ee1cec414663b957228e469869eb7673c1"},{"sha":"01dd0083011741c2bbe5ae1d2a25f2c9a1302b76","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-17764][SQL] Add `to_json` supporting to convert nested struct column to JSON string\n\n## What changes were proposed in this pull request?\n\nThis PR proposes to add `to_json` function in contrast with `from_json` in Scala, Java and Python.\n\nIt'd be useful if we can convert a same column from/to json. Also, some datasources do not support nested types. If we are forced to save a dataframe into those data sources, we might be able to work around by this function.\n\nThe usage is as below:\n\n``` scala\nval df = Seq(Tuple1(Tuple1(1))).toDF(\"a\")\ndf.select(to_json($\"a\").as(\"json\")).show()\n```\n\n``` bash\n+--------+\n|    json|\n+--------+\n|{\"_1\":1}|\n+--------+\n```\n## How was this patch tested?\n\nUnit tests in `JsonFunctionsSuite` and `JsonExpressionsSuite`.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #15354 from HyukjinKwon/SPARK-17764.","distinct":true,"url":"https://api.github.com/repos/lw-lin/spark/commits/01dd0083011741c2bbe5ae1d2a25f2c9a1302b76"},{"sha":"6e6298154aba63831a292117797798131a646869","author":{"name":"Josh Rosen","email":"1fa93d1db1fb7c1c51f2467ef55c960ec1b64919@databricks.com"},"message":"[SPARK-17350][SQL] Disable default use of KryoSerializer in Thrift Server\n\nIn SPARK-4761 / #3621 (December 2014) we enabled Kryo serialization by default in the Spark Thrift Server. However, I don't think that the original rationale for doing this still holds now that most Spark SQL serialization is now performed via encoders and our UnsafeRow format.\n\nIn addition, the use of Kryo as the default serializer can introduce performance problems because the creation of new KryoSerializer instances is expensive and we haven't performed instance-reuse optimizations in several code paths (including DirectTaskResult deserialization).\n\nGiven all of this, I propose to revert back to using JavaSerializer as the default serializer in the Thrift Server.\n\n/cc liancheng\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #14906 from JoshRosen/disable-kryo-in-thriftserver.","distinct":true,"url":"https://api.github.com/repos/lw-lin/spark/commits/6e6298154aba63831a292117797798131a646869"},{"sha":"b929537b6eb0f8f34497c3dbceea8045bf5dffdb","author":{"name":"Josh Rosen","email":"1fa93d1db1fb7c1c51f2467ef55c960ec1b64919@databricks.com"},"message":"[SPARK-18182] Expose ReplayListenerBus.read() overload which takes string iterator\n\nThe `ReplayListenerBus.read()` method is used when implementing a custom `ApplicationHistoryProvider`. The current interface only exposes a `read()` method which takes an `InputStream` and performs stream-to-lines conversion itself, but it would also be useful to expose an overloaded method which accepts an iterator of strings, thereby enabling events to be provided from non-`InputStream` sources.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #15698 from JoshRosen/replay-listener-bus-interface.","distinct":true,"url":"https://api.github.com/repos/lw-lin/spark/commits/b929537b6eb0f8f34497c3dbceea8045bf5dffdb"},{"sha":"91c33a0ca5c8287f710076ed7681e5aa13ca068f","author":{"name":"Joseph K. Bradley","email":"9b6e24a661c030e39d84e80f958c2d82d5dc3d65@databricks.com"},"message":"[SPARK-18088][ML] Various ChiSqSelector cleanups\n\n## What changes were proposed in this pull request?\n- Renamed kbest to numTopFeatures\n- Renamed alpha to fpr\n- Added missing Since annotations\n- Doc cleanups\n## How was this patch tested?\n\nAdded new standardized unit tests for spark.ml.\nImproved existing unit test coverage a bit.\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #15647 from jkbradley/chisqselector-follow-ups.","distinct":true,"url":"https://api.github.com/repos/lw-lin/spark/commits/91c33a0ca5c8287f710076ed7681e5aa13ca068f"},{"sha":"77a98162d1ec28247053b8b3ad4af28baa950797","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-18025] Use commit protocol API in structured streaming\n\n## What changes were proposed in this pull request?\nThis patch adds a new commit protocol implementation ManifestFileCommitProtocol that follows the existing streaming flow, and uses it in FileStreamSink to consolidate the write path in structured streaming with the batch mode write path.\n\nThis deletes a lot of code, and would make it trivial to support other functionalities that are currently available in batch but not in streaming, including all file formats and bucketing.\n\n## How was this patch tested?\nShould be covered by existing tests.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #15710 from rxin/SPARK-18025.","distinct":true,"url":"https://api.github.com/repos/lw-lin/spark/commits/77a98162d1ec28247053b8b3ad4af28baa950797"},{"sha":"ad4832a9faf2c0c869bbcad9d71afe1cecbd3ec8","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-18216][SQL] Make Column.expr public\n\n## What changes were proposed in this pull request?\nColumn.expr is private[sql], but it's an actually really useful field to have for debugging. We should open it up, similar to how we use QueryExecution.\n\n## How was this patch tested?\nN/A - this is a simple visibility change.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #15724 from rxin/SPARK-18216.","distinct":true,"url":"https://api.github.com/repos/lw-lin/spark/commits/ad4832a9faf2c0c869bbcad9d71afe1cecbd3ec8"},{"sha":"1ecfafa0869cb3a3e367bda8be252a69874dc4de","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-17838][SPARKR] Check named arguments for options and use formatted R friendly message from JVM exception message\n\n## What changes were proposed in this pull request?\n\nThis PR proposes to\n- improve the R-friendly error messages rather than raw JVM exception one.\n\n  As `read.json`, `read.text`, `read.orc`, `read.parquet` and `read.jdbc` are executed in the same  path with `read.df`, and `write.json`, `write.text`, `write.orc`, `write.parquet` and `write.jdbc` shares the same path with `write.df`, it seems it is safe to call `handledCallJMethod` to handle\n  JVM messages.\n-  prevent `zero-length variable name` and prints the ignored options as an warning message.\n\n**Before**\n\n``` r\n> read.json(\"path\", a = 1, 2, 3, \"a\")\nError in env[[name]] <- value :\n  zero-length variable name\n```\n\n``` r\n> read.json(\"arbitrary_path\")\nError in invokeJava(isStatic = FALSE, objId$id, methodName, ...) :\n  org.apache.spark.sql.AnalysisException: Path does not exist: file:/...;\n  at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$12.apply(DataSource.scala:398)\n  ...\n\n> read.orc(\"arbitrary_path\")\nError in invokeJava(isStatic = FALSE, objId$id, methodName, ...) :\n  org.apache.spark.sql.AnalysisException: Path does not exist: file:/...;\n  at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$12.apply(DataSource.scala:398)\n  ...\n\n> read.text(\"arbitrary_path\")\nError in invokeJava(isStatic = FALSE, objId$id, methodName, ...) :\n  org.apache.spark.sql.AnalysisException: Path does not exist: file:/...;\n  at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$12.apply(DataSource.scala:398)\n  ...\n\n> read.parquet(\"arbitrary_path\")\nError in invokeJava(isStatic = FALSE, objId$id, methodName, ...) :\n  org.apache.spark.sql.AnalysisException: Path does not exist: file:/...;\n  at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$12.apply(DataSource.scala:398)\n  ...\n```\n\n``` r\n> write.json(df, \"existing_path\")\nError in invokeJava(isStatic = FALSE, objId$id, methodName, ...) :\n  org.apache.spark.sql.AnalysisException: path file:/... already exists.;\n  at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:68)\n\n> write.orc(df, \"existing_path\")\nError in invokeJava(isStatic = FALSE, objId$id, methodName, ...) :\n  org.apache.spark.sql.AnalysisException: path file:/... already exists.;\n  at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:68)\n\n> write.text(df, \"existing_path\")\nError in invokeJava(isStatic = FALSE, objId$id, methodName, ...) :\n  org.apache.spark.sql.AnalysisException: path file:/... already exists.;\n  at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:68)\n\n> write.parquet(df, \"existing_path\")\nError in invokeJava(isStatic = FALSE, objId$id, methodName, ...) :\n  org.apache.spark.sql.AnalysisException: path file:/... already exists.;\n  at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:68)\n```\n\n**After**\n\n``` r\nread.json(\"arbitrary_path\", a = 1, 2, 3, \"a\")\nUnnamed arguments ignored: 2, 3, a.\n```\n\n``` r\n> read.json(\"arbitrary_path\")\nError in json : analysis error - Path does not exist: file:/...\n\n> read.orc(\"arbitrary_path\")\nError in orc : analysis error - Path does not exist: file:/...\n\n> read.text(\"arbitrary_path\")\nError in text : analysis error - Path does not exist: file:/...\n\n> read.parquet(\"arbitrary_path\")\nError in parquet : analysis error - Path does not exist: file:/...\n```\n\n``` r\n> write.json(df, \"existing_path\")\nError in json : analysis error - path file:/... already exists.;\n\n> write.orc(df, \"existing_path\")\nError in orc : analysis error - path file:/... already exists.;\n\n> write.text(df, \"existing_path\")\nError in text : analysis error - path file:/... already exists.;\n\n> write.parquet(df, \"existing_path\")\nError in parquet : analysis error - path file:/... already exists.;\n```\n## How was this patch tested?\n\nUnit tests in `test_utils.R` and `test_sparkSQL.R`.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #15608 from HyukjinKwon/SPARK-17838.","distinct":true,"url":"https://api.github.com/repos/lw-lin/spark/commits/1ecfafa0869cb3a3e367bda8be252a69874dc4de"}]}
{"actor":{"display_login":"lw-lin"}}