{"push_id":1121809237,"size":30,"distinct_size":30,"ref":"refs/heads/master","head":"22947cd0213856442025baf653be588c6c707e36","before":"59e6c5560d13def686091391aabe024ecb43174b","commits":[{"sha":"ef43a5fe51614eecce2d144cc13b33004a47533a","author":{"email":"ad9280c159074d9ec90899b584f520606e83d10e@techaddict.me","name":"Sandeep Singh"},"message":"[SPARK-15414][MLLIB] Make the mllib,ml linalg type conversion APIs public\n\n## What changes were proposed in this pull request?\nOpen up APIs for converting between new, old linear algebra types (in spark.mllib.linalg):\n`Sparse`/`Dense` X `Vector`/`Matrices` `.asML` and `.fromML`\n\n## How was this patch tested?\nExisting Tests\n\nAuthor: Sandeep Singh <sandeep@techaddict.me>\n\nCloses #13202 from techaddict/SPARK-15414.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/ef43a5fe51614eecce2d144cc13b33004a47533a"},{"sha":"dcf407de676ce99462a62f17cd98c9df56f057df","author":{"email":"2676c1e0be9d9df57419d05258d46b7142312de2@hortonworks.com","name":"jerryshao"},"message":"[SPARK-15375][SQL][STREAMING] Add ConsoleSink to structure streaming\n\n## What changes were proposed in this pull request?\n\nAdd ConsoleSink to structure streaming, user could use it to display dataframes on the console (useful for debugging and demostrating), similar to the functionality of `DStream#print`, to use it:\n\n```\n    val query = result.write\n      .format(\"console\")\n      .trigger(ProcessingTime(\"2 seconds\"))\n      .startStream()\n```\n\n## How was this patch tested?\n\nlocal verified.\n\nNot sure it is suitable to add into structure streaming, please review and help to comment, thanks a lot.\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #13162 from jerryshao/SPARK-15375.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/dcf407de676ce99462a62f17cd98c9df56f057df"},{"sha":"f8107c7846c9fcabbe2579867574305c7f2028e7","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-15341][DOC][ML] Add documentation for \"model.write\" to clarify \"summary\" was not saved\n\n## What changes were proposed in this pull request?\nCurrently in ```model.write```, we don't save ```summary```(if applicable). We should add documentation to clarify it.\nWe fixed the incorrect link ```[[MLWriter]]``` to ```[[org.apache.spark.ml.util.MLWriter]]``` BTW.\n\n## How was this patch tested?\nDocumentation update, no unit test.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #13131 from yanboliang/spark-15341.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/f8107c7846c9fcabbe2579867574305c7f2028e7"},{"sha":"664367781786df7ec52e39950dccd5a09681602c","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[MINOR][ML][PYSPARK] ml.evaluation Scala and Python API sync\n\n## What changes were proposed in this pull request?\n```ml.evaluation``` Scala and Python API sync.\n\n## How was this patch tested?\nOnly API docs change, no new tests.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #13195 from yanboliang/evaluation-doc.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/664367781786df7ec52e39950dccd5a09681602c"},{"sha":"16ba71aba4e68bbb892d4ceb38d6d1d135d63fd3","author":{"email":"b59cd5ce8cb5ae44d26fc6984be1f84cf74771a7@databricks.com","name":"Shixiong Zhu"},"message":"[SPARK-15416][SQL] Display a better message for not finding classes removed in Spark 2.0\n\n## What changes were proposed in this pull request?\n\nIf finding `NoClassDefFoundError` or `ClassNotFoundException`, check if the class name is removed in Spark 2.0. If so, the user must be using an incompatible library and we can provide a better message.\n\n## How was this patch tested?\n\n1. Run `bin/pyspark --packages com.databricks:spark-avro_2.10:2.0.1`\n2. type `sqlContext.read.format(\"com.databricks.spark.avro\").load(\"src/test/resources/episodes.avro\")`.\n\nIt will show `java.lang.ClassNotFoundException: org.apache.spark.sql.sources.HadoopFsRelationProvider is removed in Spark 2.0. Please check if your library is compatible with Spark 2.0`\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #13201 from zsxwing/better-message.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/16ba71aba4e68bbb892d4ceb38d6d1d135d63fd3"},{"sha":"01cf649c4f96f64fb4bd09e0e1811cabcc5ead2e","author":{"email":"ad9280c159074d9ec90899b584f520606e83d10e@techaddict.me","name":"Sandeep Singh"},"message":"[SPARK-15296][MLLIB] Refactor All Java Tests that use SparkSession\n\n## What changes were proposed in this pull request?\nRefactor All Java Tests that use SparkSession, to extend SharedSparkSesion\n\n## How was this patch tested?\nExisting Tests\n\nAuthor: Sandeep Singh <sandeep@techaddict.me>\n\nCloses #13101 from techaddict/SPARK-15296.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/01cf649c4f96f64fb4bd09e0e1811cabcc5ead2e"},{"sha":"66ec2494938e2f9f4403285713d3cba7a1495cf1","author":{"email":"9616cf314f60cbadfd3d5d51110130ad7923defb@databricks.com","name":"Xiangrui Meng"},"message":"Closes #11915\nCloses #8648\nCloses #13089","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/66ec2494938e2f9f4403285713d3cba7a1495cf1"},{"sha":"d5c47f8ff8c09ff017e896835db044661ee60909","author":{"email":"8772288d764ccf315b59fc495a905dbe6cfa6478@gmail.com","name":"Sumedh Mungee"},"message":"[SPARK-15321] Fix bug where Array[Timestamp] cannot be encoded/decoded correctly\n\n## What changes were proposed in this pull request?\n\nFix `MapObjects.itemAccessorMethod` to handle `TimestampType`. Without this fix, `Array[Timestamp]` cannot be properly encoded or decoded. To reproduce this, in `ExpressionEncoderSuite`, if you add the following test case:\n\n`encodeDecodeTest(Array(Timestamp.valueOf(\"2016-01-29 10:00:00\")), \"array of timestamp\")\n`\n... you will see that (without this fix) it fails with the following output:\n\n```\n- encode/decode for array of timestamp: [Ljava.sql.Timestamp;fd9ebde *** FAILED ***\n  Exception thrown while decoding\n  Converted: [0,1000000010,800000001,52a7ccdc36800]\n  Schema: value#61615\n  root\n  -- value: array (nullable = true)\n      |-- element: timestamp (containsNull = true)\n  Encoder:\n  class[value[0]: array<timestamp>] (ExpressionEncoderSuite.scala:312)\n```\n\n## How was this patch tested?\n\nExisting tests\n\nAuthor: Sumedh Mungee <smungee@gmail.com>\n\nCloses #13108 from smungee/fix-itemAccessorMethod.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/d5c47f8ff8c09ff017e896835db044661ee60909"},{"sha":"17591d90e6873f30a042112f56a1686726ccbd60","author":{"email":"58f74537bdbc2dd7be823680c4a7e30cfa52e668@us.ibm.com","name":"Kevin Yu"},"message":"[SPARK-11827][SQL] Adding java.math.BigInteger support in Java type inference for POJOs and Java collections\n\nHello : Can you help check this PR? I am adding support for the java.math.BigInteger for java bean code path. I saw internally spark is converting the BigInteger to BigDecimal in ColumnType.scala and CatalystRowConverter.scala. I use the similar way and convert the BigInteger to the BigDecimal. .\n\nAuthor: Kevin Yu <qyu@us.ibm.com>\n\nCloses #10125 from kevinyu98/working_on_spark-11827.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/17591d90e6873f30a042112f56a1686726ccbd60"},{"sha":"f2ee0ed4b7ecb2855cc4928a9613a07d45446f4e","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-15075][SPARK-15345][SQL] Clean up SparkSession builder and propagate config options to existing sessions if specified\n\n## What changes were proposed in this pull request?\nCurrently SparkSession.Builder use SQLContext.getOrCreate. It should probably the the other way around, i.e. all the core logic goes in SparkSession, and SQLContext just calls that. This patch does that.\n\nThis patch also makes sure config options specified in the builder are propagated to the existing (and of course the new) SparkSession.\n\n## How was this patch tested?\nUpdated tests to reflect the change, and also introduced a new SparkSessionBuilderSuite that should cover all the branches.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #13200 from rxin/SPARK-15075.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/f2ee0ed4b7ecb2855cc4928a9613a07d45446f4e"},{"sha":"3ba34d435c1f61435c2dddc28650cd111e7c1f33","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-14990][SQL] Fix checkForSameTypeInputExpr (ignore nullability)\n\n## What changes were proposed in this pull request?\nThis patch fixes a bug in TypeUtils.checkForSameTypeInputExpr. Previously the code was testing on strict equality, which does not taking nullability into account.\n\nThis is based on https://github.com/apache/spark/pull/12768. This patch fixed a bug there (with empty expression) and added a test case.\n\n## How was this patch tested?\nAdded a new test suite and test case.\n\nCloses #12768.\n\nAuthor: Reynold Xin <rxin@databricks.com>\nAuthor: Oleg Danilov <oleg.danilov@wandisco.com>\n\nCloses #13208 from rxin/SPARK-14990.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/3ba34d435c1f61435c2dddc28650cd111e7c1f33"},{"sha":"e384c7fbb94cef3c18e8fa8d06159b76b88b5167","author":{"email":"992ae4d28cc00099147263f35b077a3f30432d87@wandisco.com","name":"Oleg Danilov"},"message":"[SPARK-14261][SQL] Memory leak in Spark Thrift Server\n\nFixed memory leak (HiveConf in the CommandProcessorFactory)\n\nAuthor: Oleg Danilov <oleg.danilov@wandisco.com>\n\nCloses #12932 from dosoft/SPARK-14261.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/e384c7fbb94cef3c18e8fa8d06159b76b88b5167"},{"sha":"d5e1c5acde95158db38448526c8afad4a6d21dc2","author":{"email":"f070f8f0c50caeda5841963fc7b57e73e59dad56@happy-camper.st","name":"Takuya UESHIN"},"message":"[SPARK-15313][SQL] EmbedSerializerInFilter rule should keep exprIds of output of surrounded SerializeFromObject.\n\n## What changes were proposed in this pull request?\n\nThe following code:\n\n```\nval ds = Seq((\"a\", 1), (\"b\", 2), (\"c\", 3)).toDS()\nds.filter(_._1 == \"b\").select(expr(\"_1\").as[String]).foreach(println(_))\n```\n\nthrows an Exception:\n\n```\norg.apache.spark.sql.catalyst.errors.package$TreeNodeException: Binding attribute, tree: _1#420\n at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:50)\n at org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1.applyOrElse(BoundAttribute.scala:88)\n at org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1.applyOrElse(BoundAttribute.scala:87)\n\n...\n Cause: java.lang.RuntimeException: Couldn't find _1#420 in [_1#416,_2#417]\n at scala.sys.package$.error(package.scala:27)\n at org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1$$anonfun$applyOrElse$1.apply(BoundAttribute.scala:94)\n at org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1$$anonfun$applyOrElse$1.apply(BoundAttribute.scala:88)\n at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:49)\n at org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1.applyOrElse(BoundAttribute.scala:88)\n at org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1.applyOrElse(BoundAttribute.scala:87)\n...\n```\n\nThis is because `EmbedSerializerInFilter` rule drops the `exprId`s of output of surrounded `SerializeFromObject`.\n\nThe analyzed and optimized plans of the above example are as follows:\n\n```\n== Analyzed Logical Plan ==\n_1: string\nProject [_1#420]\n+- SerializeFromObject [staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, input[0, scala.Tuple2]._1, true) AS _1#420,input[0, scala.Tuple2]._2 AS _2#421]\n   +- Filter <function1>.apply\n      +- DeserializeToObject newInstance(class scala.Tuple2), obj#419: scala.Tuple2\n         +- LocalRelation [_1#416,_2#417], [[0,1800000001,1,61],[0,1800000001,2,62],[0,1800000001,3,63]]\n\n== Optimized Logical Plan ==\n!Project [_1#420]\n+- Filter <function1>.apply\n   +- LocalRelation [_1#416,_2#417], [[0,1800000001,1,61],[0,1800000001,2,62],[0,1800000001,3,63]]\n```\n\nThis PR fixes `EmbedSerializerInFilter` rule to keep `exprId`s of output of surrounded `SerializeFromObject`.\n\nThe plans after this patch are as follows:\n\n```\n== Analyzed Logical Plan ==\n_1: string\nProject [_1#420]\n+- SerializeFromObject [staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, input[0, scala.Tuple2]._1, true) AS _1#420,input[0, scala.Tuple2]._2 AS _2#421]\n   +- Filter <function1>.apply\n      +- DeserializeToObject newInstance(class scala.Tuple2), obj#419: scala.Tuple2\n         +- LocalRelation [_1#416,_2#417], [[0,1800000001,1,61],[0,1800000001,2,62],[0,1800000001,3,63]]\n\n== Optimized Logical Plan ==\nProject [_1#416]\n+- Filter <function1>.apply\n   +- LocalRelation [_1#416,_2#417], [[0,1800000001,1,61],[0,1800000001,2,62],[0,1800000001,3,63]]\n```\n\n## How was this patch tested?\n\nExisting tests and I added a test to check if `filter and then select` works.\n\nAuthor: Takuya UESHIN <ueshin@happy-camper.st>\n\nCloses #13096 from ueshin/issues/SPARK-15313.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/d5e1c5acde95158db38448526c8afad4a6d21dc2"},{"sha":"09a00510c4759ff87abb0b2fdf1630ddf36ca12c","author":{"email":"0da04896db04dd2393029a4b0c77df7858a20dea@gmail.com","name":"Lianhui Wang"},"message":"[SPARK-15335][SQL] Implement TRUNCATE TABLE Command\n\n## What changes were proposed in this pull request?\n\nLike TRUNCATE TABLE Command in Hive, TRUNCATE TABLE is also supported by Hive. See the link: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL\nBelow is the related Hive JIRA: https://issues.apache.org/jira/browse/HIVE-446\nThis PR is to implement such a command for truncate table excluded column truncation(HIVE-4005).\n\n## How was this patch tested?\nAdded a test case.\n\nAuthor: Lianhui Wang <lianhuiwang09@gmail.com>\n\nCloses #13170 from lianhuiwang/truncate.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/09a00510c4759ff87abb0b2fdf1630ddf36ca12c"},{"sha":"4c7a6b385c79f4de07a89495afce4f8e73b06086","author":{"email":"d82d494ad12feccfb00e29073898c0c0951e931f@hotmail.com","name":"wm624@hotmail.com"},"message":"[SPARK-15363][ML][EXAMPLE] Example code shouldn't use VectorImplicits._, asML/fromML\n\n## What changes were proposed in this pull request?\n\n(Please fill in changes proposed in this fix)\nIn this DataFrame example, we use VectorImplicits._, which is private API.\n\nSince Vectors object has public API, we use Vectors.fromML instead of implicts.\n\n## How was this patch tested?\n\n(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)\n\nManually run the example.\n\nAuthor: wm624@hotmail.com <wm624@hotmail.com>\n\nCloses #13213 from wangmiao1981/ml.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/4c7a6b385c79f4de07a89495afce4f8e73b06086"},{"sha":"47a2940da97caa55bbb8bb8ec1d51c9f6d5041c6","author":{"email":"21ef94aec54fae1a36abbd011347596ae66d9959@foxmail.com","name":"Zheng RuiFeng"},"message":"[SPARK-15398][ML] Update the warning message to recommend ML usage\n\n## What changes were proposed in this pull request?\nMLlib are not recommended to use, and some methods are even deprecated.\nUpdate the warning message to recommend ML usage.\n```\n  def showWarning() {\n    System.err.println(\n      \"\"\"WARN: This is a naive implementation of Logistic Regression and is given as an example!\n        |Please use either org.apache.spark.mllib.classification.LogisticRegressionWithSGD or\n        |org.apache.spark.mllib.classification.LogisticRegressionWithLBFGS\n        |for more conventional use.\n      \"\"\".stripMargin)\n  }\n```\nTo\n```\n  def showWarning() {\n    System.err.println(\n      \"\"\"WARN: This is a naive implementation of Logistic Regression and is given as an example!\n        |Please use org.apache.spark.ml.classification.LogisticRegression\n        |for more conventional use.\n      \"\"\".stripMargin)\n  }\n```\n\n## How was this patch tested?\nlocal build\n\nAuthor: Zheng RuiFeng <ruifengz@foxmail.com>\n\nCloses #13190 from zhengruifeng/update_recd.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/47a2940da97caa55bbb8bb8ec1d51c9f6d5041c6"},{"sha":"5e203505f1a092e5849ebd01d9ff9e4fc6cdc34a","author":{"email":"8b686eb7d2bd6ecbcaccba98aff1768585dde44b@gmail.com","name":"sethah"},"message":"[SPARK-15394][ML][DOCS] User guide typos and grammar audit\n\n## What changes were proposed in this pull request?\n\nCorrect some typos and incorrectly worded sentences.\n\n## How was this patch tested?\n\nDoc changes only.\n\nNote that many of these changes were identified by whomfire01\n\nAuthor: sethah <seth.hendrickson16@gmail.com>\n\nCloses #13180 from sethah/ml_guide_audit.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/5e203505f1a092e5849ebd01d9ff9e4fc6cdc34a"},{"sha":"c94b34ebbf4c6ce353c899c571beb34e8db98917","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-15339][ML] ML 2.0 QA: Scala APIs and code audit for regression\n\n## What changes were proposed in this pull request?\n* ```GeneralizedLinearRegression``` API docs enhancement.\n* The default value of ```GeneralizedLinearRegression``` ```linkPredictionCol``` is not set rather than empty. This will consistent with other similar params such as ```weightCol```\n* Make some methods more private.\n* Fix a minor bug of LinearRegression.\n* Fix some other issues.\n\n## How was this patch tested?\nExisting tests.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #13129 from yanboliang/spark-15339.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/c94b34ebbf4c6ce353c899c571beb34e8db98917"},{"sha":"39fd469078271aa12f3163606000e06e382d35dc","author":{"email":"905db6c57c9ba2de5402239d136139da9b5d0058@gmail.com","name":"gatorsmile"},"message":"[SPARK-15367][SQL] Add refreshTable back\n\n#### What changes were proposed in this pull request?\n`refreshTable` was a method in `HiveContext`. It was deleted accidentally while we were migrating the APIs. This PR is to add it back to `HiveContext`.\n\nIn addition, in `SparkSession`, we put it under the catalog namespace (`SparkSession.catalog.refreshTable`).\n\n#### How was this patch tested?\nChanged the existing test cases to use the function `refreshTable`. Also added a test case for refreshTable in `hivecontext-compatibility`\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #13156 from gatorsmile/refreshTable.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/39fd469078271aa12f3163606000e06e382d35dc"},{"sha":"257375019266ab9e3c320e33026318cc31f58ada","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-15421][SQL] Validate DDL property values\n\n## What changes were proposed in this pull request?\n\nWhen we parse DDLs involving table or database properties, we need to validate the values.\nE.g. if we alter a database's property without providing a value:\n```\nALTER DATABASE my_db SET DBPROPERTIES('some_key')\n```\nThen we'll ignore it with Hive, but override the property with the in-memory catalog. Inconsistencies like these arise because we don't validate the property values.\n\nIn such cases, we should throw exceptions instead.\n\n## How was this patch tested?\n\n`DDLCommandSuite`\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #13205 from andrewor14/ddl-prop-values.","distinct":true,"url":"https://api.github.com/repos/bomeng/spark/commits/257375019266ab9e3c320e33026318cc31f58ada"}]}
{"actor":{"display_login":"bomeng"}}