{"push_id":1423240301,"size":41,"distinct_size":26,"ref":"refs/heads/jdbc","head":"1b0caea20bd233ffda5113c11234d8fd57f6faa3","before":"5c5b3cab4ec77484467e960319abbddb49313952","commits":[{"sha":"e811fbf9ed131bccbc46f3c5701c4ff317222fd9","author":{"name":"sethah","email":"c41626c9d885a60b8db6c40a26d851740b23beef@gmail.com"},"message":"[SPARK-18282][ML][PYSPARK] Add python clustering summaries for GMM and BKM\n\n## What changes were proposed in this pull request?\n\nAdd model summary APIs for `GaussianMixtureModel` and `BisectingKMeansModel` in pyspark.\n\n## How was this patch tested?\n\nUnit tests.\n\nAuthor: sethah <seth.hendrickson16@gmail.com>\n\nCloses #15777 from sethah/pyspark_cluster_summaries.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/e811fbf9ed131bccbc46f3c5701c4ff317222fd9"},{"sha":"9f262ae163b6dca6526665b3ad12b3b2ea8fb873","author":{"name":"Takuya UESHIN","email":"9d9445011f9a909d02246cb8499bd5adcacb7b63@happy-camper.st"},"message":"[SPARK-18398][SQL] Fix nullabilities of MapObjects and ExternalMapToCatalyst.\n\n## What changes were proposed in this pull request?\n\nThe nullabilities of `MapObject` can be made more strict by relying on `inputObject.nullable` and `lambdaFunction.nullable`.\n\nAlso `ExternalMapToCatalyst.dataType` can be made more strict by relying on `valueConverter.nullable`.\n\n## How was this patch tested?\n\nExisting tests.\n\nAuthor: Takuya UESHIN <ueshin@happy-camper.st>\n\nCloses #15840 from ueshin/issues/SPARK-18398.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/9f262ae163b6dca6526665b3ad12b3b2ea8fb873"},{"sha":"07beb5d21c6803e80733149f1560c71cd3cacc86","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-18413][SQL] Add `maxConnections` JDBCOption\n\n## What changes were proposed in this pull request?\n\nThis PR adds a new JDBCOption `maxConnections` which means the maximum number of simultaneous JDBC connections allowed. This option applies only to writing with coalesce operation if needed. It defaults to the number of partitions of RDD. Previously, SQL users cannot cannot control this while Scala/Java/Python users can use `coalesce` (or `repartition`) API.\n\n**Reported Scenario**\n\nFor the following cases, the number of connections becomes 200 and database cannot handle all of them.\n\n```sql\nCREATE OR REPLACE TEMPORARY VIEW resultview\nUSING org.apache.spark.sql.jdbc\nOPTIONS (\n  url \"jdbc:oracle:thin:10.129.10.111:1521:BKDB\",\n  dbtable \"result\",\n  user \"HIVE\",\n  password \"HIVE\"\n);\n-- set spark.sql.shuffle.partitions=200\nINSERT OVERWRITE TABLE resultview SELECT g, count(1) AS COUNT FROM tnet.DT_LIVE_INFO GROUP BY g\n```\n\n## How was this patch tested?\n\nManual. Do the followings and see Spark UI.\n\n**Step 1 (MySQL)**\n```\nCREATE TABLE t1 (a INT);\nCREATE TABLE data (a INT);\nINSERT INTO data VALUES (1);\nINSERT INTO data VALUES (2);\nINSERT INTO data VALUES (3);\n```\n\n**Step 2 (Spark)**\n```scala\nSPARK_HOME=$PWD bin/spark-shell --driver-memory 4G --driver-class-path mysql-connector-java-5.1.40-bin.jar\nscala> sql(\"SET spark.sql.shuffle.partitions=3\")\nscala> sql(\"CREATE OR REPLACE TEMPORARY VIEW data USING org.apache.spark.sql.jdbc OPTIONS (url 'jdbc:mysql://localhost:3306/t', dbtable 'data', user 'root', password '')\")\nscala> sql(\"CREATE OR REPLACE TEMPORARY VIEW t1 USING org.apache.spark.sql.jdbc OPTIONS (url 'jdbc:mysql://localhost:3306/t', dbtable 't1', user 'root', password '', maxConnections '1')\")\nscala> sql(\"INSERT OVERWRITE TABLE t1 SELECT a FROM data GROUP BY a\")\nscala> sql(\"CREATE OR REPLACE TEMPORARY VIEW t1 USING org.apache.spark.sql.jdbc OPTIONS (url 'jdbc:mysql://localhost:3306/t', dbtable 't1', user 'root', password '', maxConnections '2')\")\nscala> sql(\"INSERT OVERWRITE TABLE t1 SELECT a FROM data GROUP BY a\")\nscala> sql(\"CREATE OR REPLACE TEMPORARY VIEW t1 USING org.apache.spark.sql.jdbc OPTIONS (url 'jdbc:mysql://localhost:3306/t', dbtable 't1', user 'root', password '', maxConnections '3')\")\nscala> sql(\"INSERT OVERWRITE TABLE t1 SELECT a FROM data GROUP BY a\")\nscala> sql(\"CREATE OR REPLACE TEMPORARY VIEW t1 USING org.apache.spark.sql.jdbc OPTIONS (url 'jdbc:mysql://localhost:3306/t', dbtable 't1', user 'root', password '', maxConnections '4')\")\nscala> sql(\"INSERT OVERWRITE TABLE t1 SELECT a FROM data GROUP BY a\")\n```\n\n![maxconnections](https://cloud.githubusercontent.com/assets/9700541/20287987/ed8409c2-aa84-11e6-8aab-ae28e63fe54d.png)\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #15868 from dongjoon-hyun/SPARK-18413.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/07beb5d21c6803e80733149f1560c71cd3cacc86"},{"sha":"70176871ae10509f1a727a96e96b3da7762605b1","author":{"name":"Gabriel Huang","email":"aaa49f1c44f417fca0453fdbc39ee9b47d0e7191@gmail.com"},"message":"[SPARK-18361][PYSPARK] Expose RDD localCheckpoint in PySpark\n\n## What changes were proposed in this pull request?\n\nExpose RDD's localCheckpoint() and associated functions in PySpark.\n\n## How was this patch tested?\n\nI added a UnitTest in python/pyspark/tests.py which passes.\n\nI certify that this is my original work, and I license it to the project under the project's open source license.\n\nGabriel HUANG\nDeveloper at Cardabel (http://cardabel.com/)\n\nAuthor: Gabriel Huang <gabi.xiaohuang@gmail.com>\n\nCloses #15811 from gabrielhuang/pyspark-localcheckpoint.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/70176871ae10509f1a727a96e96b3da7762605b1"},{"sha":"ddd02f50bb7458410d65427321efc75da5e65224","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-18517][SQL] DROP TABLE IF EXISTS should not warn for non-existing tables\n\n## What changes were proposed in this pull request?\n\nCurrently, `DROP TABLE IF EXISTS` shows warning for non-existing tables. However, it had better be quiet for this case by definition of the command.\n\n**BEFORE**\n```scala\nscala> sql(\"DROP TABLE IF EXISTS nonexist\")\n16/11/20 20:48:26 WARN DropTableCommand: org.apache.spark.sql.catalyst.analysis.NoSuchTableException: Table or view 'nonexist' not found in database 'default';\n```\n\n**AFTER**\n```scala\nscala> sql(\"DROP TABLE IF EXISTS nonexist\")\nres0: org.apache.spark.sql.DataFrame = []\n```\n\n## How was this patch tested?\n\nManual because this is related to the warning messages instead of exceptions.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #15953 from dongjoon-hyun/SPARK-18517.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/ddd02f50bb7458410d65427321efc75da5e65224"},{"sha":"a2d464770cd183daa7d727bf377bde9c21e29e6a","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-17765][SQL] Support for writing out user-defined type in ORC datasource\n\n## What changes were proposed in this pull request?\n\nThis PR adds the support for `UserDefinedType` when writing out instead of throwing `ClassCastException` in ORC data source.\n\nIn more details, `OrcStruct` is being created based on string from`DataType.catalogString`. For user-defined type, it seems it returns `sqlType.simpleString` for `catalogString` by default[1]. However, during type-dispatching to match the output with the schema, it tries to cast to, for example, `StructType`[2].\n\nSo, running the codes below (`MyDenseVector` was borrowed[3]) :\n\n``` scala\nval data = Seq((1, new UDT.MyDenseVector(Array(0.25, 2.25, 4.25))))\nval udtDF = data.toDF(\"id\", \"vectors\")\nudtDF.write.orc(\"/tmp/test.orc\")\n```\n\nends up throwing an exception as below:\n\n```\njava.lang.ClassCastException: org.apache.spark.sql.UDT$MyDenseVectorUDT cannot be cast to org.apache.spark.sql.types.ArrayType\n    at org.apache.spark.sql.hive.HiveInspectors$class.wrapperFor(HiveInspectors.scala:381)\n    at org.apache.spark.sql.hive.orc.OrcSerializer.wrapperFor(OrcFileFormat.scala:164)\n...\n```\n\nSo, this PR uses `UserDefinedType.sqlType` during finding the correct converter when writing out in ORC data source.\n\n[1]https://github.com/apache/spark/blob/dfdcab00c7b6200c22883baa3ebc5818be09556f/sql/catalyst/src/main/scala/org/apache/spark/sql/types/UserDefinedType.scala#L95\n[2]https://github.com/apache/spark/blob/d2dc8c4a162834818190ffd82894522c524ca3e5/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala#L326\n[3]https://github.com/apache/spark/blob/2bfed1a0c5be7d0718fd574a4dad90f4f6b44be7/sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala#L38-L70\n## How was this patch tested?\n\nUnit tests in `OrcQuerySuite`.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #15361 from HyukjinKwon/SPARK-17765.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/a2d464770cd183daa7d727bf377bde9c21e29e6a"},{"sha":"97a8239a625df455d2c439f3628a529d6d9413ca","author":{"name":"Burak Yavuz","email":"39bd9266538731f16447e7d968c0ba25275ede1e@gmail.com"},"message":"[SPARK-18493] Add missing python APIs: withWatermark and checkpoint to dataframe\n\n## What changes were proposed in this pull request?\n\nThis PR adds two of the newly added methods of `Dataset`s to Python:\n`withWatermark` and `checkpoint`\n\n## How was this patch tested?\n\nDoc tests\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #15921 from brkyvz/py-watermark.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/97a8239a625df455d2c439f3628a529d6d9413ca"},{"sha":"ebeb0830a3a4837c7354a0eee667b9f5fad389c5","author":{"name":"Liwei Lin","email":"878182135bcb2086b9b1c72277449d6e06bbf76f@gmail.com"},"message":"[SPARK-18425][STRUCTURED STREAMING][TESTS] Test `CompactibleFileStreamLog` directly\n\n## What changes were proposed in this pull request?\n\nRight now we are testing the most of `CompactibleFileStreamLog` in `FileStreamSinkLogSuite` (because `FileStreamSinkLog` once was the only subclass of `CompactibleFileStreamLog`, but now it's not the case any more).\n\nLet's refactor the tests so that `CompactibleFileStreamLog` is directly tested, making future changes (like https://github.com/apache/spark/pull/15828, https://github.com/apache/spark/pull/15827) to `CompactibleFileStreamLog` much easier to test and much easier to review.\n\n## How was this patch tested?\n\nthe PR itself is about tests\n\nAuthor: Liwei Lin <lwlin7@gmail.com>\n\nCloses #15870 from lw-lin/test-compact-1113.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/ebeb0830a3a4837c7354a0eee667b9f5fad389c5"},{"sha":"acb97157796231fef74aba985825b05b607b9279","author":{"name":"Yanbo Liang","email":"b6929798b4e4fba75014af8265904a7818902ec7@gmail.com"},"message":"[SPARK-18444][SPARKR] SparkR running in yarn-cluster mode should not download Spark package.\n\n## What changes were proposed in this pull request?\nWhen running SparkR job in yarn-cluster mode, it will download Spark package from apache website which is not necessary.\n```\n./bin/spark-submit --master yarn-cluster ./examples/src/main/r/dataframe.R\n```\nThe following is output:\n```\nAttaching package: ‘SparkR’\n\nThe following objects are masked from ‘package:stats’:\n\n    cov, filter, lag, na.omit, predict, sd, var, window\n\nThe following objects are masked from ‘package:base’:\n\n    as.data.frame, colnames, colnames<-, drop, endsWith, intersect,\n    rank, rbind, sample, startsWith, subset, summary, transform, union\n\nSpark not found in SPARK_HOME:\nSpark not found in the cache directory. Installation will start.\nMirrorUrl not provided.\nLooking for preferred site from apache website...\n......\n```\nThere's no ```SPARK_HOME``` in yarn-cluster mode since the R process is in a remote host of the yarn cluster rather than in the client host. The JVM comes up first and the R process then connects to it. So in such cases we should never have to download Spark as Spark is already running.\n\n## How was this patch tested?\nOffline test.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #15888 from yanboliang/spark-18444.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/acb97157796231fef74aba985825b05b607b9279"},{"sha":"4922f9cdcac8b7c10320ac1fb701997fffa45d46","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-18514][DOCS] Fix the markdown for `Note:`/`NOTE:`/`Note that` across R API documentation\n\n## What changes were proposed in this pull request?\n\nIt seems in R, there are\n\n- `Note:`\n- `NOTE:`\n- `Note that`\n\nThis PR proposes to fix those to `Note:` to be consistent.\n\n**Before**\n\n![2016-11-21 11 30 07](https://cloud.githubusercontent.com/assets/6477701/20468848/2f27b0fa-afde-11e6-89e3-993701269dbe.png)\n\n**After**\n\n![2016-11-21 11 29 44](https://cloud.githubusercontent.com/assets/6477701/20468851/39469664-afde-11e6-9929-ad80be7fc405.png)\n\n## How was this patch tested?\n\nThe notes were found via\n\n```bash\ngrep -r \"NOTE: \" .\ngrep -r \"Note that \" .\n```\n\nAnd then fixed one by one comparing with API documentation.\n\nAfter that, manually tested via `sh create-docs.sh` under `./R`.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #15952 from HyukjinKwon/SPARK-18514.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/4922f9cdcac8b7c10320ac1fb701997fffa45d46"},{"sha":"933a6548d423cf17448207a99299cf36fc1a95f6","author":{"name":"hyukjinkwon","email":"4c114bf3676ebcdb1616a11ca478d9181f4025da@gmail.com"},"message":"[SPARK-18447][DOCS] Fix the markdown for `Note:`/`NOTE:`/`Note that` across Python API documentation\n\n## What changes were proposed in this pull request?\n\nIt seems in Python, there are\n\n- `Note:`\n- `NOTE:`\n- `Note that`\n- `.. note::`\n\nThis PR proposes to fix those to `.. note::` to be consistent.\n\n**Before**\n\n<img width=\"567\" alt=\"2016-11-21 1 18 49\" src=\"https://cloud.githubusercontent.com/assets/6477701/20464305/85144c86-af88-11e6-8ee9-90f584dd856c.png\">\n\n<img width=\"617\" alt=\"2016-11-21 12 42 43\" src=\"https://cloud.githubusercontent.com/assets/6477701/20464263/27be5022-af88-11e6-8577-4bbca7cdf36c.png\">\n\n**After**\n\n<img width=\"554\" alt=\"2016-11-21 1 18 42\" src=\"https://cloud.githubusercontent.com/assets/6477701/20464306/8fe48932-af88-11e6-83e1-fc3cbf74407d.png\">\n\n<img width=\"628\" alt=\"2016-11-21 12 42 51\" src=\"https://cloud.githubusercontent.com/assets/6477701/20464264/2d3e156e-af88-11e6-93f3-cab8d8d02983.png\">\n\n## How was this patch tested?\n\nThe notes were found via\n\n```bash\ngrep -r \"Note: \" .\ngrep -r \"NOTE: \" .\ngrep -r \"Note that \" .\n```\n\nAnd then fixed one by one comparing with API documentation.\n\nAfter that, manually tested via `make html` under `./python/docs`.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #15947 from HyukjinKwon/SPARK-18447.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/933a6548d423cf17448207a99299cf36fc1a95f6"},{"sha":"bb152cdfbb8d02130c71d2326ae81939725c2cf0","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-18519][SQL] map type can not be used in EqualTo\n\n## What changes were proposed in this pull request?\n\nTechnically map type is not orderable, but can be used in equality comparison. However, due to the limitation of the current implementation, map type can't be used in equality comparison so that it can't be join key or grouping key.\n\nThis PR makes this limitation explicit, to avoid wrong result.\n\n## How was this patch tested?\n\nupdated tests.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #15956 from cloud-fan/map-type.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/bb152cdfbb8d02130c71d2326ae81939725c2cf0"},{"sha":"45ea46b7b397f023b4da878eb11e21b08d931115","author":{"name":"Nattavut Sutyanyong","email":"baa31781995a0fba3cdaaf459d34022aca55c280@gmail.com"},"message":"[SPARK-18504][SQL] Scalar subquery with extra group by columns returning incorrect result\n\n## What changes were proposed in this pull request?\n\nThis PR blocks an incorrect result scenario in scalar subquery where there are GROUP BY column(s)\nthat are not part of the correlated predicate(s).\n\nExample:\n// Incorrect result\nSeq(1).toDF(\"c1\").createOrReplaceTempView(\"t1\")\nSeq((1,1),(1,2)).toDF(\"c1\",\"c2\").createOrReplaceTempView(\"t2\")\nsql(\"select (select sum(-1) from t2 where t1.c1=t2.c1 group by t2.c2) from t1\").show\n\n// How can selecting a scalar subquery from a 1-row table return 2 rows?\n\n## How was this patch tested?\nsql/test, catalyst/test\nnew test case covering the reported problem is added to SubquerySuite.scala\n\nAuthor: Nattavut Sutyanyong <nsy.can@gmail.com>\n\nCloses #15936 from nsyca/scalarSubqueryIncorrect-1.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/45ea46b7b397f023b4da878eb11e21b08d931115"},{"sha":"702cd403fc8e5ce8281fe8828197ead46bdb8832","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-18507][SQL] HiveExternalCatalog.listPartitions should only call getTable once\n\n## What changes were proposed in this pull request?\n\nHiveExternalCatalog.listPartitions should only call `getTable` once, instead of calling it for every partitions.\n\n## How was this patch tested?\n\nN/A\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #15978 from cloud-fan/perf.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/702cd403fc8e5ce8281fe8828197ead46bdb8832"},{"sha":"bdc8153e8689262708c7fade5c065bd7fc8a84fc","author":{"name":"Burak Yavuz","email":"39bd9266538731f16447e7d968c0ba25275ede1e@gmail.com"},"message":"[SPARK-18465] Add 'IF EXISTS' clause to 'UNCACHE' to not throw exceptions when table doesn't exist\n\n## What changes were proposed in this pull request?\n\nWhile this behavior is debatable, consider the following use case:\n```sql\nUNCACHE TABLE foo;\nCACHE TABLE foo AS\nSELECT * FROM bar\n```\nThe command above fails the first time you run it. But I want to run the command above over and over again, and I don't want to change my code just for the first run of it.\nThe issue is that subsequent `CACHE TABLE` commands do not overwrite the existing table.\n\nNow we can do:\n```sql\nUNCACHE TABLE IF EXISTS foo;\nCACHE TABLE foo AS\nSELECT * FROM bar\n```\n\n## How was this patch tested?\n\nUnit tests\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #15896 from brkyvz/uncache.","distinct":false,"url":"https://api.github.com/repos/gatorsmile/spark/commits/bdc8153e8689262708c7fade5c065bd7fc8a84fc"},{"sha":"2fd101b2f0028e005fbb0bdd29e59af37aa637da","author":{"name":"Shixiong Zhu","email":"eb2f6da57ec54996cebae8c2a82b9fbfd527e14c@databricks.com"},"message":"[SPARK-18373][SPARK-18529][SS][KAFKA] Make failOnDataLoss=false work with Spark jobs\n\n## What changes were proposed in this pull request?\n\nThis PR adds `CachedKafkaConsumer.getAndIgnoreLostData` to handle corner cases of `failOnDataLoss=false`.\n\nIt also resolves [SPARK-18529](https://issues.apache.org/jira/browse/SPARK-18529) after refactoring codes: Timeout will throw a TimeoutException.\n\n## How was this patch tested?\n\nBecause I cannot find any way to manually control the Kafka server to clean up logs, it's impossible to write unit tests for each corner case. Therefore, I just created `test(\"stress test for failOnDataLoss=false\")` which should cover most of corner cases.\n\nI also modified some existing tests to test for both `failOnDataLoss=false` and `failOnDataLoss=true` to make sure it doesn't break existing logic.\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #15820 from zsxwing/failOnDataLoss.","distinct":true,"url":"https://api.github.com/repos/gatorsmile/spark/commits/2fd101b2f0028e005fbb0bdd29e59af37aa637da"},{"sha":"9c42d4a76ca8046fcca2e20067f2aa461977e65a","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-16803][SQL] SaveAsTable does not work when target table is a Hive serde table\n\n### What changes were proposed in this pull request?\n\nIn Spark 2.0, `SaveAsTable` does not work when the target table is a Hive serde table, but Spark 1.6 works.\n\n**Spark 1.6**\n\n``` Scala\nscala> sql(\"create table sample.sample stored as SEQUENCEFILE as select 1 as key, 'abc' as value\")\nres2: org.apache.spark.sql.DataFrame = []\n\nscala> val df = sql(\"select key, value as value from sample.sample\")\ndf: org.apache.spark.sql.DataFrame = [key: int, value: string]\n\nscala> df.write.mode(\"append\").saveAsTable(\"sample.sample\")\n\nscala> sql(\"select * from sample.sample\").show()\n+---+-----+\n|key|value|\n+---+-----+\n|  1|  abc|\n|  1|  abc|\n+---+-----+\n```\n\n**Spark 2.0**\n\n``` Scala\nscala> df.write.mode(\"append\").saveAsTable(\"sample.sample\")\norg.apache.spark.sql.AnalysisException: Saving data in MetastoreRelation sample, sample\n is not supported.;\n```\n\nSo far, we do not plan to support it in Spark 2.1 due to the risk. Spark 1.6 works because it internally uses insertInto. But, if we change it back it will break the semantic of saveAsTable (this method uses by-name resolution instead of using by-position resolution used by insertInto). More extra changes are needed to support `hive` as a `format` in DataFrameWriter.\n\nInstead, users should use insertInto API. This PR corrects the error messages. Users can understand how to bypass it before we support it in a separate PR.\n### How was this patch tested?\n\nTest cases are added\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #15926 from gatorsmile/saveAsTableFix5.","distinct":true,"url":"https://api.github.com/repos/gatorsmile/spark/commits/9c42d4a76ca8046fcca2e20067f2aa461977e65a"},{"sha":"39a1d30636857715247c82d551b200e1c331ad69","author":{"name":"Dilip Biswal","email":"6ab79ed79ae7ed4bf1dfe07d1c3e4c433e1f76a0@us.ibm.com"},"message":"[SPARK-18533] Raise correct error upon specification of schema for datasource tables created using CTAS\n\n## What changes were proposed in this pull request?\nFixes the inconsistency of error raised between data source and hive serde\ntables when schema is specified in CTAS scenario. In the process the grammar for\ncreate table (datasource) is simplified.\n\n**before:**\n``` SQL\nspark-sql> create table t2 (c1 int, c2 int) using parquet as select * from t1;\nError in query:\nmismatched input 'as' expecting {<EOF>, '.', 'OPTIONS', 'CLUSTERED', 'PARTITIONED'}(line 1, pos 64)\n\n== SQL ==\ncreate table t2 (c1 int, c2 int) using parquet as select * from t1\n----------------------------------------------------------------^^^\n```\n\n**After:**\n```SQL\nspark-sql> create table t2 (c1 int, c2 int) using parquet as select * from t1\n         > ;\nError in query:\nOperation not allowed: Schema may not be specified in a Create Table As Select (CTAS) statement(line 1, pos 0)\n\n== SQL ==\ncreate table t2 (c1 int, c2 int) using parquet as select * from t1\n^^^\n```\n## How was this patch tested?\nAdded a new test in CreateTableAsSelectSuite\n\nAuthor: Dilip Biswal <dbiswal@us.ibm.com>\n\nCloses #15968 from dilipbiswal/ctas.","distinct":true,"url":"https://api.github.com/repos/gatorsmile/spark/commits/39a1d30636857715247c82d551b200e1c331ad69"},{"sha":"d0212eb0f22473ee5482fe98dafc24e16ffcfc63","author":{"name":"Shixiong Zhu","email":"eb2f6da57ec54996cebae8c2a82b9fbfd527e14c@databricks.com"},"message":"[SPARK-18530][SS][KAFKA] Change Kafka timestamp column type to TimestampType\n\n## What changes were proposed in this pull request?\n\nChanged Kafka timestamp column type to TimestampType.\n\n## How was this patch tested?\n\n`test(\"Kafka column types\")`.\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #15969 from zsxwing/SPARK-18530.","distinct":true,"url":"https://api.github.com/repos/gatorsmile/spark/commits/d0212eb0f22473ee5482fe98dafc24e16ffcfc63"},{"sha":"982b82e32e0fc7d30c5d557944a79eb3e6d2da59","author":{"name":"Yanbo Liang","email":"b6929798b4e4fba75014af8265904a7818902ec7@gmail.com"},"message":"[SPARK-18501][ML][SPARKR] Fix spark.glm errors when fitting on collinear data\n\n## What changes were proposed in this pull request?\n* Fix SparkR ```spark.glm``` errors when fitting on collinear data, since ```standard error of coefficients, t value and p value``` are not available in this condition.\n* Scala/Python GLM summary should throw exception if users get ```standard error of coefficients, t value and p value``` but the underlying WLS was solved by local \"l-bfgs\".\n\n## How was this patch tested?\nAdd unit tests.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #15930 from yanboliang/spark-18501.","distinct":true,"url":"https://api.github.com/repos/gatorsmile/spark/commits/982b82e32e0fc7d30c5d557944a79eb3e6d2da59"}]}
{"actor":{"display_login":"gatorsmile"}}