{"push_id":1019152942,"size":35,"distinct_size":35,"ref":"refs/heads/master","head":"6a1c864ab6ee3e869a16ffdbaf6fead21c7aac6d","before":"6cfe341ee89baa952929e91d33b9ecbca73a3ea0","commits":[{"sha":"1c6cf1a5639bf5111324e44d93a8c6462958750a","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-12570][ML][DOC] DecisionTreeRegressor: provide variance of prediction: user guide update\n\nUpdate user guide doc for ```DecisionTreeRegressor``` providing variance of prediction.\n\ncc jkbradley\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #10594 from yanboliang/spark-12570.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/1c6cf1a5639bf5111324e44d93a8c6462958750a"},{"sha":"78015a8b7cc316343e302eeed6fe30af9f2961e8","author":{"email":"3270cab15a4ae4af5d46c40db43f5b35eac294d2@gmail.com","name":"RJ Nowling"},"message":"[SPARK-12450][MLLIB] Un-persist broadcasted variables in KMeans\n\nSPARK-12450 . Un-persist broadcasted variables in KMeans.\n\nAuthor: RJ Nowling <rnowling@gmail.com>\n\nCloses #10415 from rnowling/spark-12450.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/78015a8b7cc316343e302eeed6fe30af9f2961e8"},{"sha":"ff89975543b153d0d235c0cac615d45b34aa8fe7","author":{"email":"760e7dab2836853c63805033e514668301fa9c47@seatgeek.com","name":"BrianLondon"},"message":"[SPARK-12453][STREAMING] Remove explicit dependency on aws-java-sdk\n\nSuccessfully ran kinesis demo on a live, aws hosted kinesis stream against master and 1.6 branches.  For reasons I don't entirely understand it required a manual merge to 1.5 which I did as shown here: https://github.com/BrianLondon/spark/commit/075c22e89bc99d5e99be21f40e0d72154a1e23a2\n\nThe demo ran successfully on the 1.5 branch as well.\n\nAccording to `mvn dependency:tree` it is still pulling a fairly old version of the aws-java-sdk (1.9.37), but this appears to have fixed the kinesis regression in 1.5.2.\n\nAuthor: BrianLondon <brian@seatgeek.com>\n\nCloses #10492 from BrianLondon/remove-only.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/ff89975543b153d0d235c0cac615d45b34aa8fe7"},{"sha":"1537e55604cafafa49a8b7f3ce915f9745392bc0","author":{"email":"a8225d365811cb0449b395490c087d0ddd3c6c6f@gmail.com","name":"Kai Jiang"},"message":"[SPARK-12041][ML][PYSPARK] Add columnSimilarities to IndexedRowMatrix\n\nAdd `columnSimilarities` to IndexedRowMatrix for PySpark spark.mllib.linalg.\n\nAuthor: Kai Jiang <jiangkai@gmail.com>\n\nCloses #10158 from vectorijk/spark-12041.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/1537e55604cafafa49a8b7f3ce915f9745392bc0"},{"sha":"df8bd97520fc67dad95141c5a8cf2e0d5332e693","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-3873][SQL] Import ordering fixes.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #10573 from vanzin/SPARK-3873-sql.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/df8bd97520fc67dad95141c5a8cf2e0d5332e693"},{"sha":"0d42292f6a2dbe626e8f6a50e6c61dd79533f235","author":{"email":"d5493c25651f198266b02bc9287959eaadab29fc@gmail.com","name":"sureshthalamati"},"message":"[SPARK-12504][SQL] Masking credentials in the sql plan explain output for JDBC data sources.\n\nThis fix masks JDBC  credentials in the explain output.  URL patterns to specify credential seems to be vary between different databases. Added a new method to dialect to mask the credentials according to the database specific URL pattern.\n\nWhile adding tests I noticed explain output includes array variable for partitions ([Lorg.apache.spark.Partition;3ff74546,).  Modified the code to include the first, and last partition information.\n\nAuthor: sureshthalamati <suresh.thalamati@gmail.com>\n\nCloses #10452 from sureshthalamati/mask_jdbc_credentials_spark-12504.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/0d42292f6a2dbe626e8f6a50e6c61dd79533f235"},{"sha":"70fe6ce52f26904aa53bd20409db69b52bccf315","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-12659] fix NPE in UnsafeExternalSorter (used by cartesian product)\n\nCartesian product use UnsafeExternalSorter without comparator to do spilling, it will NPE if spilling happens.\n\nThis bug also hitted by #10605\n\ncc JoshRosen\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #10606 from davies/fix_spilling.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/70fe6ce52f26904aa53bd20409db69b52bccf315"},{"sha":"7a375bb87a8df56d9dde0c484e725e5c497a9876","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-3873][CORE] Import ordering fixes.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #10578 from vanzin/SPARK-3873-core.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/7a375bb87a8df56d9dde0c484e725e5c497a9876"},{"sha":"b3ba1be3b77e42120145252b2730a56f1d55fd21","author":{"email":"8283fd24ce58e0bb0d6bf349e9064fef42c4d84e@cloudera.com","name":"Marcelo Vanzin"},"message":"[SPARK-3873][TESTS] Import ordering fixes.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #10582 from vanzin/SPARK-3873-tests.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/b3ba1be3b77e42120145252b2730a56f1d55fd21"},{"sha":"d1fea41363c175a67b97cb7b3fe89f9043708739","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-12393][SPARKR] Add read.text and write.text for SparkR\n\nAdd ```read.text``` and ```write.text``` for SparkR.\ncc sun-rui felixcheung shivaram\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #10348 from yanboliang/spark-12393.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/d1fea41363c175a67b97cb7b3fe89f9043708739"},{"sha":"b2467b381096804b862990d9ecda554f67e07ee1","author":{"email":"d24118b4297e1ecf0c990d6935f5f9e6e4fd7701@gmail.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-12578][SQL] Distinct should not be silently ignored when used in an aggregate function with OVER clause\n\nJIRA: https://issues.apache.org/jira/browse/SPARK-12578\n\nSlightly update to Hive parser. We should keep the distinct keyword when used in an aggregate function with OVER clause. So the CheckAnalysis will detect it and throw exception later.\n\nAuthor: Liang-Chi Hsieh <viirya@gmail.com>\n\nCloses #10557 from viirya/keep-distinct-hivesql.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/b2467b381096804b862990d9ecda554f67e07ee1"},{"sha":"5d871ea43efdde59e05896a50d57021388412d30","author":{"email":"a71868187df990d49943173493ce6d1621f222ba@gmail.com","name":"QiangCai"},"message":"[SPARK-12340][SQL] fix Int overflow in the SparkPlan.executeTake, RDD.take and AsyncRDDActions.takeAsync\n\nI have closed pull request https://github.com/apache/spark/pull/10487. And I create this pull request to resolve the problem.\n\nspark jira\nhttps://issues.apache.org/jira/browse/SPARK-12340\n\nAuthor: QiangCai <david.caiq@gmail.com>\n\nCloses #10562 from QiangCai/bugfix.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/5d871ea43efdde59e05896a50d57021388412d30"},{"sha":"94c202c7d2c613d00a0a56f870a0cddbff599be3","author":{"email":"7d45abe39fd242ee2f6575da28f13d66568418b3@oss.nttdata.co.jp","name":"Kousuke Saruta"},"message":"[SPARK-12665][CORE][GRAPHX] Remove Vector, VectorSuite and GraphKryoRegistrator which are deprecated and no longer used\n\nWhole code of Vector.scala, VectorSuite.scala and GraphKryoRegistrator.scala  are no longer used so it's time to remove them in Spark 2.0.\n\nAuthor: Kousuke Saruta <sarutak@oss.nttdata.co.jp>\n\nCloses #10613 from sarutak/SPARK-12665.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/94c202c7d2c613d00a0a56f870a0cddbff599be3"},{"sha":"9061e777fdcd5767718808e325e8953d484aa761","author":{"email":"808d31295a27ded55339ed818ba0b1912ca15edb@guavus.com","name":"Yash Datta"},"message":"[SPARK-11878][SQL] Eliminate distribute by in case group by is present with exactly the same grouping expressi\n\nFor queries like :\nselect <> from table group by a distribute by a\nwe can eliminate distribute by ; since group by will anyways do a hash partitioning\nAlso applicable when user uses Dataframe API\n\nAuthor: Yash Datta <Yash.Datta@guavus.com>\n\nCloses #9858 from saucam/eliminatedistribute.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/9061e777fdcd5767718808e325e8953d484aa761"},{"sha":"3b29004d2439c03a7d9bfdf7c2edd757d3d8c240","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@us.ibm.com","name":"Holden Karau"},"message":"[SPARK-7675][ML][PYSPARK] sparkml params type conversion\n\nFrom JIRA:\nCurrently, PySpark wrappers for spark.ml Scala classes are brittle when accepting Param types. E.g., Normalizer's \"p\" param cannot be set to \"2\" (an integer); it must be set to \"2.0\" (a float). Fixing this is not trivial since there does not appear to be a natural place to insert the conversion before Python wrappers call Java's Params setter method.\n\nA possible fix will be to include a method \"_checkType\" to PySpark's Param class which checks the type, prints an error if needed, and converts types when relevant (e.g., int to float, or scipy matrix to array). The Java wrapper method which copies params to Scala can call this method when available.\n\nThis fix instead checks the types at set time since I think failing sooner is better, but I can switch it around to check at copy time if that would be better. So far this only converts int to float and other conversions (like scipymatrix to array) are left for the future.\n\nAuthor: Holden Karau <holden@us.ibm.com>\n\nCloses #9581 from holdenk/SPARK-7675-PySpark-sparkml-Params-type-conversion.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/3b29004d2439c03a7d9bfdf7c2edd757d3d8c240"},{"sha":"007da1a9dc3bb912da841cc0f5832a4fa28e6d9d","author":{"email":"957baac97c26135d676a6ed5e6e0814b0658c8e8@gmail.com","name":"Joshi"},"message":"[SPARK-11531][ML] SparseVector error Msg\n\nPySpark SparseVector should have \"Found duplicate indices\" error message\n\nAuthor: Joshi <rekhajoshm@gmail.com>\nAuthor: Rekha Joshi <rekhajoshm@gmail.com>\n\nCloses #9525 from rekhajoshm/SPARK-11531.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/007da1a9dc3bb912da841cc0f5832a4fa28e6d9d"},{"sha":"95eb65163391b9e910277a948b72efccf6136e0c","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-11945][ML][PYSPARK] Add computeCost to KMeansModel for PySpark spark.ml\n\nAdd ```computeCost``` to ```KMeansModel``` as evaluator for PySpark spark.ml.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #9931 from yanboliang/SPARK-11945.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/95eb65163391b9e910277a948b72efccf6136e0c"},{"sha":"3aa3488225af12a77da3ba807906bc6a461ef11c","author":{"email":"452ae006f36be6c70342f33f64bb6f6eb552689c@gmail.com","name":"Yanbo Liang"},"message":"[SPARK-11815][ML][PYSPARK] PySpark DecisionTreeClassifier & DecisionTreeRegressor should support setSeed\n\nPySpark ```DecisionTreeClassifier``` & ```DecisionTreeRegressor``` should support ```setSeed``` like what we do at Scala side.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #9807 from yanboliang/spark-11815.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/3aa3488225af12a77da3ba807906bc6a461ef11c"},{"sha":"ea489f14f11b2fdfb44c86634d2e2c2167b6ea18","author":{"email":"679c4ded150c6b9a8052807f108ce836961fed8d@questtec.nl","name":"Herman van Hovell"},"message":"[SPARK-12573][SPARK-12574][SQL] Move SQL Parser from Hive to Catalyst\n\nThis PR moves a major part of the new SQL parser to Catalyst. This is a prelude to start using this parser for all of our SQL parsing. The following key changes have been made:\n\nThe ANTLR Parser & Supporting classes have been moved to the Catalyst project. They are now part of the ```org.apache.spark.sql.catalyst.parser``` package. These classes contained quite a bit of code that was originally from the Hive project, I have added aknowledgements whenever this applied. All Hive dependencies have been factored out. I have also taken this chance to clean-up the ```ASTNode``` class, and to improve the error handling.\n\nThe HiveQl object that provides the functionality to convert an AST into a LogicalPlan has been refactored into three different classes, one for every SQL sub-project:\n- ```CatalystQl```: This implements Query and Expression parsing functionality.\n- ```SparkQl```: This is a subclass of CatalystQL and provides SQL/Core only functionality such as Explain and Describe.\n- ```HiveQl```: This is a subclass of ```SparkQl``` and this adds Hive-only functionality to the parser such as Analyze, Drop, Views, CTAS & Transforms. This class still depends on Hive.\n\ncc rxin\n\nAuthor: Herman van Hovell <hvanhovell@questtec.nl>\n\nCloses #10583 from hvanhovell/SPARK-12575.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/ea489f14f11b2fdfb44c86634d2e2c2167b6ea18"},{"sha":"fcd013cf70e7890aa25a8fe3cb6c8b36bf0e1f04","author":{"email":"ffad4dc97bb49db436733c212788f8b59c50dd2b@gmail.com","name":"zero323"},"message":"[SPARK-12006][ML][PYTHON] Fix GMM failure if initialModel is not None\n\nIf initial model passed to GMM is not empty it causes `net.razorvine.pickle.PickleException`. It can be fixed by converting `initialModel.weights` to `list`.\n\nAuthor: zero323 <matthew.szymkiewicz@gmail.com>\n\nCloses #9986 from zero323/SPARK-12006.","distinct":true,"url":"https://api.github.com/repos/WangTaoTheTonic/spark/commits/fcd013cf70e7890aa25a8fe3cb6c8b36bf0e1f04"}]}