{"push_id":1056954815,"size":6,"distinct_size":6,"ref":"refs/heads/master","head":"953ff897e422570a329d0aec98d573d3fb66ab9a","before":"aa852215f82876977d164f371627e894e86baacc","commits":[{"sha":"ae1db91d158d1ae62a0ab7ea74467679ca050101","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-14410][SQL] Push functions existence check into catalog\n\n## What changes were proposed in this pull request?\n\nThis is a followup to #12117 and addresses some of the TODOs introduced there. In particular, the resolution of database is now pushed into session catalog, which knows about the current database. Further, the logic for checking whether a function exists is pushed into the external catalog.\n\nNo change in functionality is expected.\n\n## How was this patch tested?\n\n`SessionCatalogSuite`, `DDLSuite`\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #12198 from andrewor14/function-exists.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/ae1db91d158d1ae62a0ab7ea74467679ca050101"},{"sha":"49fb237081bbca0d811aa48aa06f4728fea62781","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-14270][SQL] whole stage codegen support for typed filter\n\n## What changes were proposed in this pull request?\n\nWe implement typed filter by `MapPartitions`, which doesn't work well with whole stage codegen. This PR use `Filter` to implement typed filter and we can get the whole stage codegen support for free.\n\nThis PR also introduced `DeserializeToObject` and `SerializeFromObject`, to seperate serialization logic from object operator, so that it's eaiser to write optimization rules for adjacent object operators.\n\n## How was this patch tested?\n\nexisting tests.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #12061 from cloud-fan/whole-stage-codegen.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/49fb237081bbca0d811aa48aa06f4728fea62781"},{"sha":"30e980ad8e6443dddd54f3c2d48b3904499545cf","author":{"email":"7661b9e1f005294f1a4d931c89010c59872cd7af@mesosphere.io","name":"Michael Gummelt"},"message":"[DOCS][MINOR] Remove sentence about Mesos not supporting cluster mode.\n\nDocs change to remove the sentence about Mesos not supporting cluster mode.\n\nIt was not.\n\nAuthor: Michael Gummelt <mgummelt@mesosphere.io>\n\nCloses #12249 from mgummelt/fix-mesos-cluster-docs.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/30e980ad8e6443dddd54f3c2d48b3904499545cf"},{"sha":"3e29e372ff518827bae9dcd26087946fde476843","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"Andrew Or"},"message":"[SPARK-14468] Always enable OutputCommitCoordinator\n\n## What changes were proposed in this pull request?\n\n`OutputCommitCoordinator` was introduced to deal with concurrent task attempts racing to write output, leading to data loss or corruption. For more detail, read the [JIRA description](https://issues.apache.org/jira/browse/SPARK-14468).\n\nBefore: `OutputCommitCoordinator` is enabled only if speculation is enabled.\nAfter: `OutputCommitCoordinator` is always enabled.\n\nUsers may still disable this through `spark.hadoop.outputCommitCoordination.enabled`, but they really shouldn't...\n\n## How was this patch tested?\n\n`OutputCommitCoordinator*Suite`\n\nAuthor: Andrew Or <andrew@databricks.com>\n\nCloses #12244 from andrewor14/always-occ.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/3e29e372ff518827bae9dcd26087946fde476843"},{"sha":"692c74840bc53debbb842db5372702f58207412c","author":{"email":"17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","name":"Michael Armbrust"},"message":"[SPARK-14449][SQL] SparkContext should use SparkListenerInterface\n\nCurrently all `SparkFirehoseListener` implementations are broken since we expect listeners to extend `SparkListener`, while the fire hose only extends `SparkListenerInterface`.  This changes the addListener function and the config based injection to use the interface instead.\n\nThe existing tests in SparkListenerSuite are improved such that they would have caught this.\n\nFollow-up to #12142\n\nAuthor: Michael Armbrust <michael@databricks.com>\n\nCloses #12227 from marmbrus/fixListener.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/692c74840bc53debbb842db5372702f58207412c"},{"sha":"953ff897e422570a329d0aec98d573d3fb66ab9a","author":{"email":"461476587780aa9fa5611ea6dc3912c146a91760@databricks.com","name":"Joseph K. Bradley"},"message":"[SPARK-13048][ML][MLLIB] keepLastCheckpoint option for LDA EM optimizer\n\n## What changes were proposed in this pull request?\n\nThe EMLDAOptimizer should generally not delete its last checkpoint since that can cause failures when DistributedLDAModel methods are called (if any partitions need to be recovered from the checkpoint).\n\nThis PR adds a \"deleteLastCheckpoint\" option which defaults to false.  This is a change in behavior from Spark 1.6, in that the last checkpoint will not be removed by default.\n\nThis involves adding the deleteLastCheckpoint option to both spark.ml and spark.mllib, and modifying PeriodicCheckpointer to support the option.\n\nThis also:\n* Makes MLlibTestSparkContext extend TempDirectory and set the checkpointDir to tempDir\n* Updates LibSVMRelationSuite because of a name conflict with \"tempDir\" (and fixes a bug where it failed to delete a temp directory)\n* Adds a MIMA exclude for DistributedLDAModel constructor, which is already ```private[clustering]```\n\n## How was this patch tested?\n\nAdded 2 new unit tests to spark.ml LDASuite, which calls into spark.mllib.\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #12166 from jkbradley/emlda-save-checkpoint.","distinct":true,"url":"https://api.github.com/repos/ueshin/apache-spark/commits/953ff897e422570a329d0aec98d573d3fb66ab9a"}]}