{"push_id":925802562,"size":551,"distinct_size":293,"ref":"refs/heads/SPARK-11871","head":"d0e517fb557ddb285e7bbdb7944ac5d0af65a02b","before":"5b852cdfc391a029f4b4f5aa1824b0550e03a5d2","commits":[{"sha":"880128f37e1bc0b9d98d1786670be62a06c648f2","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-4134][CORE] Lower severity of some executor loss logs.\n\nDon't log ERROR messages when executors are explicitly killed or when\nthe exit reason is not yet known.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #9780 from vanzin/SPARK-11789.","distinct":false,"url":"https://api.github.com/repos/yinxusen/spark/commits/880128f37e1bc0b9d98d1786670be62a06c648f2"},{"sha":"b2cecb80ece59a1c086d4ae7aeebef445a4e7299","author":{"email":"4330a85689689e23b81e88bdf5853b25c33529eb@gmail.com","name":"Tathagata Das"},"message":"[SPARK-11845][STREAMING][TEST] Added unit test to verify TrackStateRDD is correctly checkpointed\n\nTo make sure that all lineage is correctly truncated for TrackStateRDD when checkpointed.\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #9831 from tdas/SPARK-11845.","distinct":false,"url":"https://api.github.com/repos/yinxusen/spark/commits/b2cecb80ece59a1c086d4ae7aeebef445a4e7299"},{"sha":"ee21407747fb00db2f26d1119446ccbb20c19232","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-11864][SQL] Improve performance of max/min\n\nThis PR has the following optimization:\n\n1) The greatest/least already does the null-check, so the `If` and `IsNull` are not necessary.\n\n2) In greatest/least, it should initialize the result using the first child (removing one block).\n\n3) For primitive types, the generated greater expression is too complicated (`a > b ? 1 : (a < b) ? -1 : 0) > 0`), should be as simple as `a > b`\n\nCombine these optimization, this could improve the performance of `ss_max` query by 30%.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #9846 from davies/improve_max.","distinct":false,"url":"https://api.github.com/repos/yinxusen/spark/commits/ee21407747fb00db2f26d1119446ccbb20c19232"},{"sha":"7ee7d5a3c4ff77d2cee2afce36ff41f6302e6315","author":{"email":"ee7669731148a67ed053406d4e2088f2bc2c2de0@us.ibm.com","name":"Dilip Biswal"},"message":"[SPARK-11544][SQL][TEST-HADOOP1.0] sqlContext doesn't use PathFilter\n\nApply the user supplied pathfilter while retrieving the files from fs.\n\nAuthor: Dilip Biswal <dbiswal@us.ibm.com>\n\nCloses #9830 from dilipbiswal/spark-11544.","distinct":false,"url":"https://api.github.com/repos/yinxusen/spark/commits/7ee7d5a3c4ff77d2cee2afce36ff41f6302e6315"},{"sha":"4114ce20fbe820f111e55e891ae3889b0e6e0006","author":{"email":"7853928d03425cf3be35be2fa0c5b99641435d26@gmail.com","name":"Xusen Yin"},"message":"[SPARK-11846] Add save/load for AFTSurvivalRegression and IsotonicRegression\n\nhttps://issues.apache.org/jira/browse/SPARK-11846\n\nmengxr\n\nAuthor: Xusen Yin <yinxusen@gmail.com>\n\nCloses #9836 from yinxusen/SPARK-11846.","distinct":false,"url":"https://api.github.com/repos/yinxusen/spark/commits/4114ce20fbe820f111e55e891ae3889b0e6e0006"},{"sha":"3b7f056da87a23f3a96f0311b3a947a9b698f38b","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-11829][ML] Add read/write to estimators under ml.feature (II)\n\nAdd read/write support to the following estimators under spark.ml:\n* ChiSqSelector\n* PCA\n* VectorIndexer\n* Word2Vec\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #9838 from yanboliang/spark-11829.","distinct":false,"url":"https://api.github.com/repos/yinxusen/spark/commits/3b7f056da87a23f3a96f0311b3a947a9b698f38b"},{"sha":"7216f405454f6f3557b5b1f72df8f393605faf60","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-11875][ML][PYSPARK] Update doc for PySpark HasCheckpointInterval\n\n* Update doc for PySpark ```HasCheckpointInterval``` that users can understand how to disable checkpoint.\n* Update doc for PySpark ```cacheNodeIds``` of ```DecisionTreeParams``` to notify the relationship between ```cacheNodeIds``` and ```checkpointInterval```.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #9856 from yanboliang/spark-11875.","distinct":false,"url":"https://api.github.com/repos/yinxusen/spark/commits/7216f405454f6f3557b5b1f72df8f393605faf60"},{"sha":"0fff8eb3e476165461658d4e16682ec64269fdfe","author":{"email":"461476587780aa9fa5611ea6dc3912c146a91760@databricks.com","name":"Joseph K. Bradley"},"message":"[SPARK-11869][ML] Clean up TempDirectory properly in ML tests\n\nNeed to remove parent directory (```className```) rather than just tempDir (```className/random_name```)\n\nI tested this with IDFSuite, which has 2 read/write tests, and it fixes the problem.\n\nCC: mengxr  Can you confirm this is fine?  I believe it is since the same ```random_name``` is used for all tests in a suite; we basically have an extra unneeded level of nesting.\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #9851 from jkbradley/tempdir-cleanup.","distinct":false,"url":"https://api.github.com/repos/yinxusen/spark/commits/0fff8eb3e476165461658d4e16682ec64269fdfe"},{"sha":"3e1d120cedb4bd9e1595e95d4d531cf61da6684d","author":{"email":"7853928d03425cf3be35be2fa0c5b99641435d26@gmail.com","name":"Xusen Yin"},"message":"[SPARK-11867] Add save/load for kmeans and naive bayes\n\nhttps://issues.apache.org/jira/browse/SPARK-11867\n\nAuthor: Xusen Yin <yinxusen@gmail.com>\n\nCloses #9849 from yinxusen/SPARK-11867.","distinct":false,"url":"https://api.github.com/repos/yinxusen/spark/commits/3e1d120cedb4bd9e1595e95d4d531cf61da6684d"},{"sha":"a66142decee48bf5689fb7f4f33646d7bb1ac08d","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-11877] Prevent agg. fallback conf. from leaking across test suites\n\nThis patch fixes an issue where the `spark.sql.TungstenAggregate.testFallbackStartsAt` SQLConf setting was not properly reset / cleared at the end of `TungstenAggregationQueryWithControlledFallbackSuite`. This ended up causing test failures in HiveCompatibilitySuite in Maven builds by causing spilling to occur way too frequently.\n\nThis configuration leak was inadvertently introduced during test cleanup in #9618.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #9857 from JoshRosen/clear-fallback-prop-in-test-teardown.","distinct":false,"url":"https://api.github.com/repos/yinxusen/spark/commits/a66142decee48bf5689fb7f4f33646d7bb1ac08d"},{"sha":"9ace2e5c8d7fbd360a93bc5fc4eace64a697b44f","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-11852][ML] StandardScaler minor refactor\n\n```withStd``` and ```withMean``` should be params of ```StandardScaler``` and ```StandardScalerModel```.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #9839 from yanboliang/standardScaler-refactor.","distinct":false,"url":"https://api.github.com/repos/yinxusen/spark/commits/9ace2e5c8d7fbd360a93bc5fc4eace64a697b44f"},{"sha":"e359d5dcf5bd300213054ebeae9fe75c4f7eb9e7","author":{"email":"66332c93c294b372ab1bdd5acb9b89a538e2d2f2@gmail.com","name":"Yuhao Yang"},"message":"[SPARK-11689][ML] Add user guide and example code for LDA under spark.ml\n\njira: https://issues.apache.org/jira/browse/SPARK-11689\n\nAdd simple user guide for LDA under spark.ml and example code under examples/. Use include_example to include example code in the user guide markdown. Check SPARK-11606 for instructions.\n\nAuthor: Yuhao Yang <hhbyyh@gmail.com>\n\nCloses #9722 from hhbyyh/ldaMLExample.","distinct":false,"url":"https://api.github.com/repos/yinxusen/spark/commits/e359d5dcf5bd300213054ebeae9fe75c4f7eb9e7"},{"sha":"bef361c589c0a38740232fd8d0a45841e4fc969a","author":{"email":"905db6c57c9ba2de5402239d136139da9b5d0058@gmail.com","name":"gatorsmile"},"message":"[SPARK-11876][SQL] Support printSchema in DataSet API\n\nDataSet APIs look great! However, I am lost when doing multiple level joins.  For example,\n```\nval ds1 = Seq((\"a\", 1), (\"b\", 2)).toDS().as(\"a\")\nval ds2 = Seq((\"a\", 1), (\"b\", 2)).toDS().as(\"b\")\nval ds3 = Seq((\"a\", 1), (\"b\", 2)).toDS().as(\"c\")\n\nds1.joinWith(ds2, $\"a._2\" === $\"b._2\").as(\"ab\").joinWith(ds3, $\"ab._1._2\" === $\"c._2\").printSchema()\n```\n\nThe printed schema is like\n```\nroot\n |-- _1: struct (nullable = true)\n |    |-- _1: struct (nullable = true)\n |    |    |-- _1: string (nullable = true)\n |    |    |-- _2: integer (nullable = true)\n |    |-- _2: struct (nullable = true)\n |    |    |-- _1: string (nullable = true)\n |    |    |-- _2: integer (nullable = true)\n |-- _2: struct (nullable = true)\n |    |-- _1: string (nullable = true)\n |    |-- _2: integer (nullable = true)\n```\n\nPersonally, I think we need the printSchema function. Sometimes, I do not know how to specify the column, especially when their data types are mixed. For example, if I want to write the following select for the above multi-level join, I have to know the schema:\n```\nnewDS.select(expr(\"_1._2._2 + 1\").as[Int]).collect()\n```\n\nmarmbrus rxin cloud-fan  Do you have the same feeling?\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #9855 from gatorsmile/printSchemaDataSet.","distinct":false,"url":"https://api.github.com/repos/yinxusen/spark/commits/bef361c589c0a38740232fd8d0a45841e4fc969a"},{"sha":"60bfb113325c71491f8dcf98b6036b0caa2144fe","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@appier.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-11817][SQL] Truncating the fractional seconds to prevent inserting a NULL\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-11817\n\nInstead of return None, we should truncate the fractional seconds to prevent inserting NULL.\n\nAuthor: Liang-Chi Hsieh <viirya@appier.com>\n\nCloses #9834 from viirya/truncate-fractional-sec.","distinct":false,"url":"https://api.github.com/repos/yinxusen/spark/commits/60bfb113325c71491f8dcf98b6036b0caa2144fe"},{"sha":"3b9d2a347f9c796b90852173d84189834e499e25","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-11819][SQL] nice error message for missing encoder\n\nbefore this PR, when users try to get an encoder for an un-supported class, they will only get a very simple error message like `Encoder for type xxx is not supported`.\n\nAfter this PR, the error message become more friendly, for example:\n```\nNo Encoder found for abc.xyz.NonEncodable\n- array element class: \"abc.xyz.NonEncodable\"\n- field (class: \"scala.Array\", name: \"arrayField\")\n- root class: \"abc.xyz.AnotherClass\"\n```\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #9810 from cloud-fan/error-message.","distinct":false,"url":"https://api.github.com/repos/yinxusen/spark/commits/3b9d2a347f9c796b90852173d84189834e499e25"},{"sha":"652def318e47890bd0a0977dc982cc07f99fb06a","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-11650] Reduce RPC timeouts to speed up slow AkkaUtilsSuite test\n\nThis patch reduces some RPC timeouts in order to speed up the slow \"AkkaUtilsSuite.remote fetch ssl on - untrusted server\", which used to take two minutes to run.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #9869 from JoshRosen/SPARK-11650.","distinct":false,"url":"https://api.github.com/repos/yinxusen/spark/commits/652def318e47890bd0a0977dc982cc07f99fb06a"},{"sha":"9ed4ad4265cf9d3135307eb62dae6de0b220fc21","author":{"email":"99a5e5f8f5911755b88e0b536d46aafa102bed41@databricks.com","name":"Nong Li"},"message":"[SPARK-11724][SQL] Change casting between int and timestamp to consistently treat int in seconds.\n\nHive has since changed this behavior as well. https://issues.apache.org/jira/browse/HIVE-3454\n\nAuthor: Nong Li <nong@databricks.com>\nAuthor: Nong Li <nongli@gmail.com>\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #9685 from nongli/spark-11724.","distinct":false,"url":"https://api.github.com/repos/yinxusen/spark/commits/9ed4ad4265cf9d3135307eb62dae6de0b220fc21"},{"sha":"be7a2cfd978143f6f265eca63e9e24f755bc9f22","author":{"email":"b59cd5ce8cb5ae44d26fc6984be1f84cf74771a7@databricks.com","name":"Shixiong Zhu"},"message":"[SPARK-11870][STREAMING][PYSPARK] Rethrow the exceptions in TransformFunction and TransformFunctionSerializer\n\nTransformFunction and TransformFunctionSerializer don't rethrow the exception, so when any exception happens, it just return None. This will cause some weird NPE and confuse people.\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #9847 from zsxwing/pyspark-streaming-exception.","distinct":false,"url":"https://api.github.com/repos/yinxusen/spark/commits/be7a2cfd978143f6f265eca63e9e24f755bc9f22"},{"sha":"89fd9bd06160fa89dedbf685bfe159ffe4a06ec6","author":{"email":"e5dd8a3899598b2e85930c5ff06b11c053ef654f@databricks.com","name":"Josh Rosen"},"message":"[SPARK-11887] Close PersistenceEngine at the end of PersistenceEngineSuite tests\n\nIn PersistenceEngineSuite, we do not call `close()` on the PersistenceEngine at the end of the test. For the ZooKeeperPersistenceEngine, this causes us to leak a ZooKeeper client, causing the logs of unrelated tests to be periodically spammed with connection error messages from that client:\n\n```\n15/11/20 05:13:35.789 pool-1-thread-1-ScalaTest-running-PersistenceEngineSuite-SendThread(localhost:15741) INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:15741. Will not attempt to authenticate using SASL (unknown error)\n15/11/20 05:13:35.790 pool-1-thread-1-ScalaTest-running-PersistenceEngineSuite-SendThread(localhost:15741) WARN ClientCnxn: Session 0x15124ff48dd0000 for server null, unexpected error, closing socket connection and attempting reconnect\njava.net.ConnectException: Connection refused\n\tat sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)\n\tat sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)\n\tat org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)\n\tat org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)\n```\n\nThis patch fixes this by using a `finally` block.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #9864 from JoshRosen/close-zookeeper-client-in-tests.","distinct":false,"url":"https://api.github.com/repos/yinxusen/spark/commits/89fd9bd06160fa89dedbf685bfe159ffe4a06ec6"},{"sha":"03ba56d78f50747710d01c27d409ba2be42ae557","author":{"email":"b29f906215354b2d63cd7fb7b9919be54e0af7cd@apache.org","name":"Jean-Baptiste Onofré"},"message":"[SPARK-11716][SQL] UDFRegistration just drops the input type when re-creating the UserDefinedFunction\n\nhttps://issues.apache.org/jira/browse/SPARK-11716\n\nThis is one is #9739 and a regression test. When commit it, please make sure the author is jbonofre.\n\nYou can find the original PR at https://github.com/apache/spark/pull/9739\n\ncloses #9739\n\nAuthor: Jean-Baptiste Onofré <jbonofre@apache.org>\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #9868 from yhuai/SPARK-11716.","distinct":false,"url":"https://api.github.com/repos/yinxusen/spark/commits/03ba56d78f50747710d01c27d409ba2be42ae557"}]}