{"push_id":1094751941,"size":107,"distinct_size":38,"ref":"refs/heads/spark-14971","head":"54643443b3a19869a34ecff3c236c2f7c3eb6006","before":"a9ffcb7156f993f7f101d96b4f3d39913c6e6844","commits":[{"sha":"ae4e3def5eacb8e383a3535e6c685897fd1aaf4c","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-14961] Build HashedRelation larger than 1G\n\n## What changes were proposed in this pull request?\n\nCurrently, LongToUnsafeRowMap use byte array as the underlying page, which can't be larger 1G.\n\nThis PR improves LongToUnsafeRowMap  to scale up to 8G bytes by using array of Long instead of array of byte.\n\n## How was this patch tested?\n\nManually ran a test to confirm that both UnsafeHashedRelation and LongHashedRelation could build a map that larger than 2G.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #12740 from davies/larger_broadcast.","distinct":false,"url":"https://api.github.com/repos/yanboliang/spark/commits/ae4e3def5eacb8e383a3535e6c685897fd1aaf4c"},{"sha":"be317d4a90b3ca906fefeb438f89a09b1c7da5a8","author":{"email":"6c68cabc7124e43c9b4d50f86510600f69e74c99@odersky.com","name":"Jakob Odersky"},"message":"[SPARK-10001][CORE] Don't short-circuit actions in signal handlers\n\n## What changes were proposed in this pull request?\nThe current signal handlers have a subtle bug that stops evaluating registered actions as soon as one of them returns true, this is because `forall` is short-circuited.\nThis PR adds a strict mapping stage before evaluating returned result.\nThere are no known occurrences of the bug and this is a preemptive fix.\n\n## How was this patch tested?\nAs with the original introduction of signal handlers, this was tested manually (unit testing with signals is not straightforward).\n\nAuthor: Jakob Odersky <jakob@odersky.com>\n\nCloses #12745 from jodersky/SPARK-10001-hotfix.","distinct":false,"url":"https://api.github.com/repos/yanboliang/spark/commits/be317d4a90b3ca906fefeb438f89a09b1c7da5a8"},{"sha":"bf5496dbdac75ea69081c95a92a29771e635ea98","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-14654][CORE] New accumulator API\n\n## What changes were proposed in this pull request?\n\nThis PR introduces a new accumulator API  which is much simpler than before:\n\n1. the type hierarchy is simplified, now we only have an `Accumulator` class\n2. Combine `initialValue` and `zeroValue` concepts into just one concept: `zeroValue`\n3. there in only one `register` method, the accumulator registration and cleanup registration are combined.\n4. the `id`,`name` and `countFailedValues` are combined into an `AccumulatorMetadata`, and is provided during registration.\n\n`SQLMetric` is a good example to show the simplicity of this new API.\n\nWhat we break:\n\n1. no `setValue` anymore. In the new API, the intermedia type can be different from the result type, it's very hard to implement a general `setValue`\n2. accumulator can't be serialized before registered.\n\nProblems need to be addressed in follow-ups:\n\n1. with this new API, `AccumulatorInfo` doesn't make a lot of sense, the partial output is not partial updates, we need to expose the intermediate value.\n2. `ExceptionFailure` should not carry the accumulator updates. Why do users care about accumulator updates for failed cases? It looks like we only use this feature to update the internal metrics, how about we sending a heartbeat to update internal metrics after the failure event?\n3. the public event `SparkListenerTaskEnd` carries a `TaskMetrics`. Ideally this `TaskMetrics` don't need to carry external accumulators, as the only method of `TaskMetrics` that can access external accumulators is `private[spark]`. However, `SQLListener` use it to retrieve sql metrics.\n\n## How was this patch tested?\n\nexisting tests\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #12612 from cloud-fan/acc.","distinct":false,"url":"https://api.github.com/repos/yanboliang/spark/commits/bf5496dbdac75ea69081c95a92a29771e635ea98"},{"sha":"7c6937a8859ebd3c971116dea54ef380c1636999","author":{"email":"b226223d7355fe9fd300b8643a53cd1893f12f1f@tw.ibm.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-14487][SQL] User Defined Type registration without SQLUserDefinedType annotation\n\n## What changes were proposed in this pull request?\n\nCurrently we use `SQLUserDefinedType` annotation to register UDTs for user classes. However, by doing this, we add Spark dependency to user classes.\n\nFor some user classes, it is unnecessary to add such dependency that will increase deployment difficulty.\n\nWe should provide alternative approach to register UDTs for user classes without `SQLUserDefinedType` annotation.\n\n## How was this patch tested?\n\n`UserDefinedTypeSuite`\n\nAuthor: Liang-Chi Hsieh <simonh@tw.ibm.com>\n\nCloses #12259 from viirya/improve-sql-usertype.","distinct":false,"url":"https://api.github.com/repos/yanboliang/spark/commits/7c6937a8859ebd3c971116dea54ef380c1636999"},{"sha":"23256be0d0846d4eb188a4d1cae6e3f261248153","author":{"email":"12e2ed2eb2580f464b25083ea1c45d9c37504d48@fb.com","name":"Ergin Seyfe"},"message":"[SPARK-14576][WEB UI] Spark console should display Web UI url\n\n## What changes were proposed in this pull request?\nThis is a proposal to print the Spark Driver UI link when spark-shell is launched.\n\n## How was this patch tested?\nLaunched spark-shell in local mode and cluster mode. Spark-shell console output included following line:\n\"Spark context Web UI available at <Spark web url>\"\n\nAuthor: Ergin Seyfe <eseyfe@fb.com>\n\nCloses #12341 from seyfe/spark_console_display_webui_link.","distinct":false,"url":"https://api.github.com/repos/yanboliang/spark/commits/23256be0d0846d4eb188a4d1cae6e3f261248153"},{"sha":"9e785079b6ed4ea691c3c14c762a7f73fb6254bf","author":{"email":"3f27eab2afd49386f0ac811afb0bc245aa92c0fa@intel.com","name":"Sun Rui"},"message":"[SPARK-12235][SPARKR] Enhance mutate() to support replace existing columns.\n\nMake the behavior of mutate more consistent with that in dplyr, besides support for replacing existing columns.\n1. Throw error message when there are duplicated column names in the DataFrame being mutated.\n2. when there are duplicated column names in specified columns by arguments, the last column of the same name takes effect.\n\nAuthor: Sun Rui <rui.sun@intel.com>\n\nCloses #10220 from sun-rui/SPARK-12235.","distinct":false,"url":"https://api.github.com/repos/yanboliang/spark/commits/9e785079b6ed4ea691c3c14c762a7f73fb6254bf"},{"sha":"8b44bd52fa40c0fc7d34798c3654e31533fd3008","author":{"email":"2676c1e0be9d9df57419d05258d46b7142312de2@hortonworks.com","name":"jerryshao"},"message":"[SPARK-6735][YARN] Add window based executor failure tracking mechanism for long running service\n\nThis work is based on twinkle-sachdeva 's proposal. In parallel to such mechanism for AM failures, here add similar mechanism for executor failure tracking, this is useful for long running Spark service to mitigate the executor failure problems.\n\nPlease help to review, tgravescs sryza and vanzin\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #10241 from jerryshao/SPARK-6735.","distinct":false,"url":"https://api.github.com/repos/yanboliang/spark/commits/8b44bd52fa40c0fc7d34798c3654e31533fd3008"},{"sha":"bed0b002023441f8c4bd132e9d209222943b20f7","author":{"email":"291c18f3fb7528c712d9098b0e50a515ea0b91d5@cloudera.com","name":"Sean Owen"},"message":"[SPARK-14882][DOCS] Clarify that Spark can be cross-built for other Scala versions\n\n## What changes were proposed in this pull request?\n\nAdd simple clarification that Spark can be cross-built for other Scala versions.\n\n## How was this patch tested?\n\nAutomated doc build\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #12757 from srowen/SPARK-14882.","distinct":false,"url":"https://api.github.com/repos/yanboliang/spark/commits/bed0b002023441f8c4bd132e9d209222943b20f7"},{"sha":"5743352a28fffbfbaca2201208ce7a1d7893f813","author":{"email":"623dda1cf42194a966de8b161dee064823eb61d3@126.com","name":"Xin Ren"},"message":"[SPARK-14935][CORE] DistributedSuite \"local-cluster format\" shouldn't actually launch clusters\n\nhttps://issues.apache.org/jira/browse/SPARK-14935\n\nIn DistributedSuite, the \"local-cluster format\" test actually launches a bunch of clusters, but this doesn't seem necessary for what should just be a unit test of a regex. We should clean up the code so that this is testable without actually launching a cluster, which should buy us about 20 seconds per build.\n\nPassed unit test on my local machine\n\nAuthor: Xin Ren <iamshrek@126.com>\n\nCloses #12744 from keypointt/SPARK-14935.","distinct":false,"url":"https://api.github.com/repos/yanboliang/spark/commits/5743352a28fffbfbaca2201208ce7a1d7893f813"},{"sha":"89addd40abdacd65cc03ac8aa5f9cf3dd4a4c19b","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-14945][PYTHON] SparkSession Python API\n\n## What changes were proposed in this pull request?\n\n```\nWelcome to\n      ____              __\n     / __/__  ___ _____/ /__\n    _\\ \\/ _ \\/ _ `/ __/  '_/\n   /__ / .__/\\_,_/_/ /_/\\_\\   version 2.0.0-SNAPSHOT\n      /_/\n\nUsing Python version 2.7.5 (default, Mar  9 2014 22:15:05)\nSparkSession available as 'spark'.\n>>> spark\n<pyspark.sql.session.SparkSession object at 0x101f3bfd0>\n>>> spark.sql(\"SHOW TABLES\").show()\n...\n+---------+-----------+\n|tableName|isTemporary|\n+---------+-----------+\n|      src|      false|\n+---------+-----------+\n\n>>> spark.range(1, 10, 2).show()\n+---+\n| id|\n+---+\n|  1|\n|  3|\n|  5|\n|  7|\n|  9|\n+---+\n```\n**Note**: This API is NOT complete in its current state. In particular, for now I left out the `conf` and `catalog` APIs, which were added later in Scala. These will be added later before 2.0.\n\n## How was this patch tested?\n\nPython tests.\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #12746 from andrewor14/python-spark-session.","distinct":false,"url":"https://api.github.com/repos/yanboliang/spark/commits/89addd40abdacd65cc03ac8aa5f9cf3dd4a4c19b"},{"sha":"12c360c057f09d13a31c458ad277640b5f6de394","author":{"email":"7cd42a84d9127f55f191494a86692bd6df9b4226@thinkbiganalytics.com","name":"Gregory Hart"},"message":"[SPARK-14965][SQL] Indicate an exception is thrown for a missing struct field\n\n## What changes were proposed in this pull request?\n\nFix to ScalaDoc for StructType.\n\n## How was this patch tested?\n\nBuilt locally.\n\nAuthor: Gregory Hart <greg.hart@thinkbiganalytics.com>\n\nCloses #12758 from freastro/hotfix/SPARK-14965.","distinct":false,"url":"https://api.github.com/repos/yanboliang/spark/commits/12c360c057f09d13a31c458ad277640b5f6de394"},{"sha":"5ee72454df21ef4668c855134627d0cdf5d35132","author":{"email":"461476587780aa9fa5611ea6dc3912c146a91760@databricks.com","name":"Joseph K. Bradley"},"message":"[SPARK-14852][ML] refactored GLM summary into training, non-training summaries\n\n## What changes were proposed in this pull request?\n\nThis splits GeneralizedLinearRegressionSummary into 2 summary types:\n* GeneralizedLinearRegressionSummary, which does not store info from fitting (diagInvAtWA)\n* GeneralizedLinearRegressionTrainingSummary, which is a subclass of GeneralizedLinearRegressionSummary and stores info from fitting\n\nThis also add a method evaluate() which can produce a GeneralizedLinearRegressionSummary on a new dataset.\n\nThe summary no longer provides the model itself as a public val.\n\nAlso:\n* Fixes bug where GeneralizedLinearRegressionTrainingSummary was created with model, not summaryModel.\n* Adds hasSummary method.\n* Renames findSummaryModelAndPredictionCol -> getSummaryModel and simplifies that method.\n* In summary, extract values from model immediately in case user later changes those (e.g., predictionCol).\n* Pardon the style fixes; that is IntelliJ being obnoxious.\n\n## How was this patch tested?\n\nExisting unit tests + updated test for evaluate and hasSummary\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #12624 from jkbradley/model-summary-api.","distinct":false,"url":"https://api.github.com/repos/yanboliang/spark/commits/5ee72454df21ef4668c855134627d0cdf5d35132"},{"sha":"d5ab42ceb94feb3645578bf459b104fb0b0d5682","author":{"email":"66332c93c294b372ab1bdd5acb9b89a538e2d2f2@gmail.com","name":"Yuhao Yang"},"message":"[SPARK-14916][MLLIB] A more friendly tostring for FreqItemset in mllib.fpm\n\n## What changes were proposed in this pull request?\n\njira: https://issues.apache.org/jira/browse/SPARK-14916\nFreqItemset as the result of FPGrowth should have a more friendly toString(), to help users and developers.\nsample:\n{a, b}: 5\n{x, y, z}: 4\n\n## How was this patch tested?\n\nexisting unit tests.\n\nAuthor: Yuhao Yang <hhbyyh@gmail.com>\n\nCloses #12698 from hhbyyh/freqtos.","distinct":false,"url":"https://api.github.com/repos/yanboliang/spark/commits/d5ab42ceb94feb3645578bf459b104fb0b0d5682"},{"sha":"0ee5419b6ce535c714718d0d33b80eedd4b0a5fd","author":{"email":"4330a85689689e23b81e88bdf5853b25c33529eb@gmail.com","name":"Tathagata Das"},"message":"[SPARK-14970][SQL] Prevent DataSource from enumerates all files in a directory if there is user specified schema\n\n## What changes were proposed in this pull request?\nThe FileCatalog object gets created even if the user specifies schema, which means files in the directory is enumerated even thought its not necessary. For large directories this is very slow. User would want to specify schema in such scenarios of large dirs, and this defeats the purpose quite a bit.\n\n## How was this patch tested?\nHard to test this with unit test.\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #12748 from tdas/SPARK-14970.","distinct":false,"url":"https://api.github.com/repos/yanboliang/spark/commits/0ee5419b6ce535c714718d0d33b80eedd4b0a5fd"},{"sha":"d584a2b8ac57eff3bf230c760e5bda205c6ea747","author":{"email":"a8225d365811cb0449b395490c087d0ddd3c6c6f@gmail.com","name":"Kai Jiang"},"message":"[SPARK-12810][PYSPARK] PySpark CrossValidatorModel should support avgMetrics\n\n## What changes were proposed in this pull request?\nsupport avgMetrics in CrossValidatorModel with Python\n## How was this patch tested?\nDoctest and `test_save_load` in `pyspark/ml/test.py`\n[JIRA](https://issues.apache.org/jira/browse/SPARK-12810)\n\nAuthor: Kai Jiang <jiangkai@gmail.com>\n\nCloses #12464 from vectorijk/spark-12810.","distinct":false,"url":"https://api.github.com/repos/yanboliang/spark/commits/d584a2b8ac57eff3bf230c760e5bda205c6ea747"},{"sha":"78c8aaf849aadbb065730959e7c1b70bb58d69c9","author":{"email":"affa8bc277be808c7f93de9249c2a75ded11876a@gmail.com","name":"Burak Yavuz"},"message":"[SPARK-14555] Second cut of Python API for Structured Streaming\n\n## What changes were proposed in this pull request?\n\nThis PR adds Python APIs for:\n - `ContinuousQueryManager`\n - `ContinuousQueryException`\n\nThe `ContinuousQueryException` is a very basic wrapper, it doesn't provide the functionality that the Scala side provides, but it follows the same pattern for `AnalysisException`.\n\nFor `ContinuousQueryManager`, all APIs are provided except for registering listeners.\n\nThis PR also attempts to fix test flakiness by stopping all active streams just before tests.\n\n## How was this patch tested?\n\nPython Doc tests and unit tests\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #12673 from brkyvz/pyspark-cqm.","distinct":false,"url":"https://api.github.com/repos/yanboliang/spark/commits/78c8aaf849aadbb065730959e7c1b70bb58d69c9"},{"sha":"dae538a4d7c36191c1feb02ba87ffc624ab960dc","author":{"email":"bc9570497728884c7f6686fa0fc06a8d320e124b@in.ibm.com","name":"Pravin Gadakh"},"message":"[SPARK-14613][ML] Add @Since into the matrix and vector classes in spark-mllib-local\n\n## What changes were proposed in this pull request?\n\nThis PR adds `since` tag into the matrix and vector classes in spark-mllib-local.\n\n## How was this patch tested?\n\nScala-style checks passed.\n\nAuthor: Pravin Gadakh <prgadakh@in.ibm.com>\n\nCloses #12416 from pravingadakh/SPARK-14613.","distinct":false,"url":"https://api.github.com/repos/yanboliang/spark/commits/dae538a4d7c36191c1feb02ba87ffc624ab960dc"},{"sha":"4f4721a21cc9acc2b6f685bbfc8757d29563a775","author":{"email":"461476587780aa9fa5611ea6dc3912c146a91760@databricks.com","name":"Joseph K. Bradley"},"message":"[SPARK-14862][ML] Updated Classifiers to not require labelCol metadata\n\n## What changes were proposed in this pull request?\n\nUpdated Classifier, DecisionTreeClassifier, RandomForestClassifier, GBTClassifier to not require input column metadata.\n* They first check for metadata.\n* If numClasses is not specified in metadata, they identify the largest label value (up to a limit).\n\nThis functionality is implemented in a new Classifier.getNumClasses method.\n\nAlso\n* Updated Classifier.extractLabeledPoints to (a) check label values and (b) include a second version which takes a numClasses value for validity checking.\n\n## How was this patch tested?\n\n* Unit tests in ClassifierSuite for helper methods\n* Unit tests for DecisionTreeClassifier, RandomForestClassifier, GBTClassifier with toy datasets lacking label metadata\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #12663 from jkbradley/trees-no-metadata.","distinct":false,"url":"https://api.github.com/repos/yanboliang/spark/commits/4f4721a21cc9acc2b6f685bbfc8757d29563a775"},{"sha":"2398e3d69c9a675d651c192107953de8e6c2aecd","author":{"email":"2676c1e0be9d9df57419d05258d46b7142312de2@hortonworks.com","name":"jerryshao"},"message":"[SPARK-14836][YARN] Zip all the jars before uploading to distributed cache\n\n## What changes were proposed in this pull request?\n\n<copy form JIRA>\n\nCurrently if neither `spark.yarn.jars` nor `spark.yarn.archive` is set (by default), Spark on yarn code will upload all the jars in the folder separately into distributed cache, this is quite time consuming, and very verbose, instead of upload jars separately into distributed cache, here changes to zip all the jars first, and then put into distributed cache.\n\nThis will significantly improve the speed of starting time.\n\n## How was this patch tested?\n\nUnit test and local integrated test is done.\n\nVerified with SparkPi both in spark cluster and client mode.\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #12597 from jerryshao/SPARK-14836.","distinct":false,"url":"https://api.github.com/repos/yanboliang/spark/commits/2398e3d69c9a675d651c192107953de8e6c2aecd"},{"sha":"9c7c42bc6a35679cfffcdfb6feb26af834fec2e1","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"Revert \"[SPARK-14613][ML] Add @Since into the matrix and vector classes in spark-mllib-local\"\n\nThis reverts commit dae538a4d7c36191c1feb02ba87ffc624ab960dc.","distinct":false,"url":"https://api.github.com/repos/yanboliang/spark/commits/9c7c42bc6a35679cfffcdfb6feb26af834fec2e1"}]}