{"push_id":1051283922,"size":30,"distinct_size":30,"ref":"refs/heads/master","head":"7db56244fa3dba92246bad6694f31bbf68ea47ec","before":"d7982a3a9aa804e7e3a2004335e7f314867a5f8a","commits":[{"sha":"67d753516da9b6318cd4001bb7ae91703aaf098d","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[HOTFIX] Fix compilation break.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/67d753516da9b6318cd4001bb7ae91703aaf098d"},{"sha":"06694f1c68cb752ea311144f0dbe50e92e1393cf","author":{"email":"9749d1492af3d43f9c09e04c5c43f27bb909af51@japila.pl","name":"Jacek Laskowski"},"message":"[MINOR] Typo fixes\n\n## What changes were proposed in this pull request?\n\nTypo fixes. No functional changes.\n\n## How was this patch tested?\n\nBuilt the sources and ran with samples.\n\nAuthor: Jacek Laskowski <jacek@japila.pl>\n\nCloses #11802 from jaceklaskowski/typo-fixes.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/06694f1c68cb752ea311144f0dbe50e92e1393cf"},{"sha":"a3e293542a6e7df9bcc7d9bbd22b3c93a81bcc38","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[HOTFIX] Disable StateStoreSuite.maintenance","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/a3e293542a6e7df9bcc7d9bbd22b3c93a81bcc38"},{"sha":"f705037617d55bb479ec60bcb1e55c736224be94","author":{"email":"6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org","name":"Dongjoon Hyun"},"message":"[SPARK-14338][SQL] Improve `SimplifyConditionals` rule to handle `null` in IF/CASEWHEN\n\n## What changes were proposed in this pull request?\n\nCurrently, `SimplifyConditionals` handles `true` and `false` to optimize branches. This PR improves `SimplifyConditionals` to take advantage of `null` conditions for `if` and `CaseWhen` expressions, too.\n\n**Before**\n```\nscala> sql(\"SELECT IF(null, 1, 0)\").explain()\n== Physical Plan ==\nWholeStageCodegen\n:  +- Project [if (null) 1 else 0 AS (IF(CAST(NULL AS BOOLEAN), 1, 0))#4]\n:     +- INPUT\n+- Scan OneRowRelation[]\nscala> sql(\"select case when cast(null as boolean) then 1 else 2 end\").explain()\n== Physical Plan ==\nWholeStageCodegen\n:  +- Project [CASE WHEN null THEN 1 ELSE 2 END AS CASE WHEN CAST(NULL AS BOOLEAN) THEN 1 ELSE 2 END#14]\n:     +- INPUT\n+- Scan OneRowRelation[]\n```\n\n**After**\n```\nscala> sql(\"SELECT IF(null, 1, 0)\").explain()\n== Physical Plan ==\nWholeStageCodegen\n:  +- Project [0 AS (IF(CAST(NULL AS BOOLEAN), 1, 0))#4]\n:     +- INPUT\n+- Scan OneRowRelation[]\nscala> sql(\"select case when cast(null as boolean) then 1 else 2 end\").explain()\n== Physical Plan ==\nWholeStageCodegen\n:  +- Project [2 AS CASE WHEN CAST(NULL AS BOOLEAN) THEN 1 ELSE 2 END#4]\n:     +- INPUT\n+- Scan OneRowRelation[]\n```\n\n**Hive**\n```\nhive> select if(null,1,2);\nOK\n2\nhive> select case when cast(null as boolean) then 1 else 2 end;\nOK\n2\n```\n\n## How was this patch tested?\n\nPass the Jenkins tests (including new extended test cases).\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #12122 from dongjoon-hyun/SPARK-14338.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/f705037617d55bb479ec60bcb1e55c736224be94"},{"sha":"4a6e78abd9d5edc4a5092738dff0006bbe202a89","author":{"email":"6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org","name":"Dongjoon Hyun"},"message":"[MINOR][DOCS] Use multi-line JavaDoc comments in Scala code.\n\n## What changes were proposed in this pull request?\n\nThis PR aims to fix all Scala-Style multiline comments into Java-Style multiline comments in Scala codes.\n(All comment-only changes over 77 files: +786 lines, −747 lines)\n\n## How was this patch tested?\n\nManual.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #12130 from dongjoon-hyun/use_multiine_javadoc_comments.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/4a6e78abd9d5edc4a5092738dff0006bbe202a89"},{"sha":"03d130f9734be66e8aefc4ffaa207ee13e837629","author":{"email":"a6ba1154cbf361d4882390aa08f5a86dcc774822@gmail.com","name":"Liwei Lin"},"message":"[SPARK-14342][CORE][DOCS][TESTS] Remove straggler references to Tachyon\n\n## What changes were proposed in this pull request?\n\nStraggler references to Tachyon were removed:\n- for docs, `tachyon` has been generalized as `off-heap memory`;\n- for Mesos test suits, the key-value `tachyon:true`/`tachyon:false` has been changed to `os:centos`/`os:ubuntu`, since `os` is an example constrain used by the [Mesos official docs](http://mesos.apache.org/documentation/attributes-resources/).\n\n## How was this patch tested?\n\nExisting test suites.\n\nAuthor: Liwei Lin <lwlin7@gmail.com>\n\nCloses #12129 from lw-lin/tachyon-cleanup.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/03d130f9734be66e8aefc4ffaa207ee13e837629"},{"sha":"1cf70183423b938ec064925b20fd4a5b9e355991","author":{"email":"651d9e2ab320d6d21a117141758c2477389f223d@fb.com","name":"Sital Kedia"},"message":"[SPARK-14056] Appends s3 specific configurations and spark.hadoop con…\n\n## What changes were proposed in this pull request?\n\nAppends s3 specific configurations and spark.hadoop configurations to hive configuration.\n\n## How was this patch tested?\n\nTested by running a job on cluster.\n\n…figurations to hive configuration.\n\nAuthor: Sital Kedia <skedia@fb.com>\n\nCloses #11876 from sitalkedia/hiveConf.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/1cf70183423b938ec064925b20fd4a5b9e355991"},{"sha":"c2f25b1a148eeb1791ea7018b14b3a665c13212a","author":{"email":"b226223d7355fe9fd300b8643a53cd1893f12f1f@tw.ibm.com","name":"Liang-Chi Hsieh"},"message":"[SPARK-13996] [SQL] Add more not null attributes for Filter codegen\n\n## What changes were proposed in this pull request?\nJIRA: https://issues.apache.org/jira/browse/SPARK-13996\n\nFilter codegen finds the attributes not null by checking IsNotNull(a) expression with a condition if child.output.contains(a). However, the current approach to checking it is not comprehensive. We can improve it.\n\nE.g., for this plan:\n\n    val rdd = sqlContext.sparkContext.makeRDD(Seq(Row(1, \"1\"), Row(null, \"1\"), Row(2, \"2\")))\n    val schema = new StructType().add(\"k\", IntegerType).add(\"v\", StringType)\n    val smallDF = sqlContext.createDataFrame(rdd, schema)\n    val df = smallDF.filter(\"isnotnull(k + 1)\")\n\nThe code snippet generated without this patch:\n\n    /* 031 */   protected void processNext() throws java.io.IOException {\n    /* 032 */     /*** PRODUCE: Filter isnotnull((k#0 + 1)) */\n    /* 033 */\n    /* 034 */     /*** PRODUCE: INPUT */\n    /* 035 */\n    /* 036 */     while (!shouldStop() && inputadapter_input.hasNext()) {\n    /* 037 */       InternalRow inputadapter_row = (InternalRow) inputadapter_input.next();\n    /* 038 */       /*** CONSUME: Filter isnotnull((k#0 + 1)) */\n    /* 039 */       /* input[0, int] */\n    /* 040 */       boolean filter_isNull = inputadapter_row.isNullAt(0);\n    /* 041 */       int filter_value = filter_isNull ? -1 : (inputadapter_row.getInt(0));\n    /* 042 */\n    /* 043 */       /* isnotnull((input[0, int] + 1)) */\n    /* 044 */       /* (input[0, int] + 1) */\n    /* 045 */       boolean filter_isNull3 = true;\n    /* 046 */       int filter_value3 = -1;\n    /* 047 */\n    /* 048 */       if (!filter_isNull) {\n    /* 049 */         filter_isNull3 = false; // resultCode could change nullability.\n    /* 050 */         filter_value3 = filter_value + 1;\n    /* 051 */\n    /* 052 */       }\n    /* 053 */       if (!(!(filter_isNull3))) continue;\n    /* 054 */\n    /* 055 */       filter_metricValue.add(1);\n\nWith this patch:\n\n    /* 031 */   protected void processNext() throws java.io.IOException {\n    /* 032 */     /*** PRODUCE: Filter isnotnull((k#0 + 1)) */\n    /* 033 */\n    /* 034 */     /*** PRODUCE: INPUT */\n    /* 035 */\n    /* 036 */     while (!shouldStop() && inputadapter_input.hasNext()) {\n    /* 037 */       InternalRow inputadapter_row = (InternalRow) inputadapter_input.next();\n    /* 038 */       /*** CONSUME: Filter isnotnull((k#0 + 1)) */\n    /* 039 */       /* input[0, int] */\n    /* 040 */       boolean filter_isNull = inputadapter_row.isNullAt(0);\n    /* 041 */       int filter_value = filter_isNull ? -1 : (inputadapter_row.getInt(0));\n    /* 042 */\n    /* 043 */       if (filter_isNull) continue;\n    /* 044 */\n    /* 045 */       filter_metricValue.add(1);\n\n## How was this patch tested?\n\nExisting tests.\n\nAuthor: Liang-Chi Hsieh <simonh@tw.ibm.com>\n\nCloses #11810 from viirya/add-more-not-null-attrs.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/c2f25b1a148eeb1791ea7018b14b3a665c13212a"},{"sha":"7be46205083fc688249ee619ac7758904f7aa55d","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[HOTFIX] Fix Scala 2.10 compilation","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/7be46205083fc688249ee619ac7758904f7aa55d"},{"sha":"2262a93358c2f6d4cfb73645c4ebc963c5640ec8","author":{"email":"2680965ad5aae29275c8fb10150cb10f5554e78b@gmail.com","name":"hyukjinkwon"},"message":"[SPARK-14231] [SQL] JSON data source infers floating-point values as a double when they do not fit in a decimal\n\n## What changes were proposed in this pull request?\n\nhttps://issues.apache.org/jira/browse/SPARK-14231\n\nCurrently, JSON data source supports to infer `DecimalType` for big numbers and `floatAsBigDecimal` option which reads floating-point values as `DecimalType`.\n\nBut there are few restrictions in Spark `DecimalType` below:\n\n1. The precision cannot be bigger than 38.\n2. scale cannot be bigger than precision.\n\nCurrently, both restrictions are not being handled.\n\nThis PR handles the cases by inferring them as `DoubleType`. Also, the option name was changed from `floatAsBigDecimal` to `prefersDecimal` as suggested [here](https://issues.apache.org/jira/browse/SPARK-14231?focusedCommentId=15215579&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15215579).\n\nSo, the codes below:\n\n```scala\ndef doubleRecords: RDD[String] =\n  sqlContext.sparkContext.parallelize(\n    s\"\"\"{\"a\": 1${\"0\" * 38}, \"b\": 0.01}\"\"\" ::\n    s\"\"\"{\"a\": 2${\"0\" * 38}, \"b\": 0.02}\"\"\" :: Nil)\n\nval jsonDF = sqlContext.read\n  .option(\"prefersDecimal\", \"true\")\n  .json(doubleRecords)\njsonDF.printSchema()\n```\n\nproduces below:\n\n- **Before**\n\n```scala\norg.apache.spark.sql.AnalysisException: Decimal scale (2) cannot be greater than precision (1).;\n\tat org.apache.spark.sql.types.DecimalType.<init>(DecimalType.scala:44)\n\tat org.apache.spark.sql.execution.datasources.json.InferSchema$.org$apache$spark$sql$execution$datasources$json$InferSchema$$inferField(InferSchema.scala:144)\n\tat org.apache.spark.sql.execution.datasources.json.InferSchema$.org$apache$spark$sql$execution$datasources$json$InferSchema$$inferField(InferSchema.scala:108)\n\tat\n...\n```\n\n- **After**\n\n```scala\nroot\n |-- a: double (nullable = true)\n |-- b: double (nullable = true)\n```\n\n## How was this patch tested?\n\nUnit tests were used and `./dev/run_tests` for coding style tests.\n\nAuthor: hyukjinkwon <gurwls223@gmail.com>\n\nCloses #12030 from HyukjinKwon/SPARK-14231.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/2262a93358c2f6d4cfb73645c4ebc963c5640ec8"},{"sha":"1f0c5dcebba1f9d1149043a496e0175f78252bae","author":{"email":"6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org","name":"Dongjoon Hyun"},"message":"[SPARK-14350][SQL] EXPLAIN output should be in a single cell\n\n## What changes were proposed in this pull request?\n\nEXPLAIN output should be in a single cell.\n\n**Before**\n```\nscala> sql(\"explain select 1\").collect()\nres0: Array[org.apache.spark.sql.Row] = Array([== Physical Plan ==], [WholeStageCodegen], [:  +- Project [1 AS 1#1]], [:     +- INPUT], [+- Scan OneRowRelation[]])\n```\n\n**After**\n```\nscala> sql(\"explain select 1\").collect()\nres1: Array[org.apache.spark.sql.Row] =\nArray([== Physical Plan ==\nWholeStageCodegen\n:  +- Project [1 AS 1#4]\n:     +- INPUT\n+- Scan OneRowRelation[]])\n```\nOr,\n```\nscala> sql(\"explain select 1\").head\nres1: org.apache.spark.sql.Row =\n[== Physical Plan ==\nWholeStageCodegen\n:  +- Project [1 AS 1#5]\n:     +- INPUT\n+- Scan OneRowRelation[]]\n```\n\nPlease note that `Spark-shell(Scala-shell)` trims long string output. So, you may need to use `println` to get full strings.\n```\nscala> println(sql(\"explain codegen select 'a' as a group by 1\").head)\n[Found 2 WholeStageCodegen subtrees.\n== Subtree 1 / 2 ==\nWholeStageCodegen\n...\n/* 059 */   }\n/* 060 */ }\n\n]\n```\n\n## How was this patch tested?\n\nPass the Jenkins tests. (Testcases are updated.)\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #12137 from dongjoon-hyun/SPARK-14350.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/1f0c5dcebba1f9d1149043a496e0175f78252bae"},{"sha":"c238cd07448f94bbceb661daad90b6a6d597a846","author":{"email":"1a1ab5abfb65135a4f49eb66e79d9e570e28807d@us.ibm.com","name":"bomeng"},"message":"[SPARK-14341][SQL] Throw exception on unsupported create / drop macro ddl\n\n## What changes were proposed in this pull request?\n\nWe throw an AnalysisException that looks like this:\n\n```\nscala> sqlContext.sql(\"CREATE TEMPORARY MACRO SIGMOID (x DOUBLE) 1.0 / (1.0 + EXP(-x))\")\norg.apache.spark.sql.catalyst.parser.ParseException:\nUnsupported SQL statement\n== SQL ==\nCREATE TEMPORARY MACRO SIGMOID (x DOUBLE) 1.0 / (1.0 + EXP(-x))\n  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.nativeCommand(ParseDriver.scala:66)\n  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:56)\n  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:53)\n  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:86)\n  at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:53)\n  at org.apache.spark.sql.SQLContext.parseSql(SQLContext.scala:198)\n  at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:749)\n  ... 48 elided\n\n```\n\n## How was this patch tested?\n\nAdd test cases in HiveQuerySuite.scala\n\nAuthor: bomeng <bmeng@us.ibm.com>\n\nCloses #12125 from bomeng/SPARK-14341.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/c238cd07448f94bbceb661daad90b6a6d597a846"},{"sha":"9023015f059327b3ce4a7eaf71e57ac77b84ad7b","author":{"email":"1cadf8044321016a5149c363cdbd68e5541576ea@handybook.com","name":"Marcin Tustin"},"message":"[SPARK-14163][CORE] SumEvaluator and countApprox cannot reliably handle RDDs of size 1\n\n## What changes were proposed in this pull request?\n\nThis special cases 0 and 1 counts to avoid passing 0 degrees of freedom.\n\n## How was this patch tested?\n\nTests run successfully. New test added.\n\n## Note:\nThis recreates #11982 which was closed to due to non-updated diff. rxin srowen Commented there.\nThis also adds tests, reworks the code to perform the special casing (based on srowen's comments), and adds equality machinery for BoundedDouble, as well as changing how it is transformed to string.\n\nAuthor: Marcin Tustin <mtustin@handybook.com>\nAuthor: Marcin Tustin <mtustin@handy.com>\n\nCloses #12016 from mtustin-handy/SPARK-14163.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/9023015f059327b3ce4a7eaf71e57ac77b84ad7b"},{"sha":"3f749f7ed443899d667c9e2b2a11bc595d6fc7f6","author":{"email":"6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org","name":"Dongjoon Hyun"},"message":"[SPARK-14355][BUILD] Fix typos in Exception/Testcase/Comments and static analysis results\n\n## What changes were proposed in this pull request?\n\nThis PR contains the following 5 types of maintenance fix over 59 files (+94 lines, -93 lines).\n- Fix typos(exception/log strings, testcase name, comments) in 44 lines.\n- Fix lint-java errors (MaxLineLength) in 6 lines. (New codes after SPARK-14011)\n- Use diamond operators in 40 lines. (New codes after SPARK-13702)\n- Fix redundant semicolon in 5 lines.\n- Rename class `InferSchemaSuite` to `CSVInferSchemaSuite` in CSVInferSchemaSuite.scala.\n\n## How was this patch tested?\n\nManual and pass the Jenkins tests.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #12139 from dongjoon-hyun/SPARK-14355.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/3f749f7ed443899d667c9e2b2a11bc595d6fc7f6"},{"sha":"76f3c735aa300d7ea6b17e64cc22d7e8fc3a8322","author":{"email":"c481c5e387de8107aa9fe99c851e11e1ff44d0c3@databricks.com","name":"Matei Zaharia"},"message":"[SPARK-14356] Update spark.sql.execution.debug to work on Datasets\n\n## What changes were proposed in this pull request?\n\nUpdate DebugQuery to work on Datasets of any type, not just DataFrames.\n\n## How was this patch tested?\n\nAdded unit tests, checked in spark-shell.\n\nAuthor: Matei Zaharia <matei@databricks.com>\n\nCloses #12140 from mateiz/debug-dataset.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/76f3c735aa300d7ea6b17e64cc22d7e8fc3a8322"},{"sha":"0340b3d279de6be4903673bbf3e6a1a2653de6c0","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-14360][SQL] QueryExecution.debug.codegen() to dump codegen\n\n## What changes were proposed in this pull request?\nWe recently added the ability to dump the generated code for a given query. However, the method is only available through an implicit after an import. It'd slightly simplify things if it can be called directly in queryExecution.\n\n## How was this patch tested?\nManually tested in spark-shell.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #12144 from rxin/SPARK-14360.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/0340b3d279de6be4903673bbf3e6a1a2653de6c0"},{"sha":"745425332f41e2ae94649f9d1ad675243f36f743","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-14137] [SQL] Cleanup hash join\n\n## What changes were proposed in this pull request?\n\nThis PR did a few cleanup on HashedRelation and HashJoin:\n\n1) Merge HashedRelation and UniqueHashedRelation together\n2) Return an iterator from HashedRelation, so we donot need a create many UnsafeRow objects.\n3) Return a copy of HashedRelation for thread-safety in BroadcastJoin, so we can re-use the UnafeRow objects.\n4) Cleanup HashJoin, share most of the code between BroadcastHashJoin and ShuffleHashJoin\n5) Removed UniqueLongHashedRelation, which will be replaced by LongUnsafeMap (another PR).\n6) Update benchmark, before this patch, the selectivity of joins are too high.\n\n## How was this patch tested?\n\nExisting tests.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #12102 from davies/cleanup_hash.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/745425332f41e2ae94649f9d1ad675243f36f743"},{"sha":"89f3befab6c150f87de2fb91b50ea8b414c69095","author":{"email":"461476587780aa9fa5611ea6dc3912c146a91760@databricks.com","name":"Joseph K. Bradley"},"message":"[SPARK-13784][ML] Persistence for RandomForestClassifier, RandomForestRegressor\n\n## What changes were proposed in this pull request?\n\n**Main change**: Added save/load for RandomForestClassifier, RandomForestRegressor (implementation details below)\n\nModified numTrees method (*deprecation*)\n* Goal: Use default implementations of unit tests which assume Estimators and Models share the same set of Params.\n* What this PR does: Moves method numTrees outside of trait TreeEnsembleModel.  Adds it to GBT and RF Models.  Deprecates it in RF Models in favor of new method getNumTrees.  In Spark 2.1, we can have RF Models include Param numTrees.\n\nMinor items\n* Fixes bugs in GBTClassificationModel, GBTRegressionModel fromOld methods where they assign the wrong old UID.\n\n**Implementation details**\n* Split DecisionTreeModelReadWrite.loadTreeNodes into 2 methods in order to reuse some code for ensembles.\n* Added EnsembleModelReadWrite object with save/load implementations usable for RFs and GBTs\n  * These store all trees' nodes in a single DataFrame, and all trees' metadata in a second DataFrame.\n* Split trait RandomForestParams into parts in order to add more Estimator Params to RF models\n* Split DefaultParamsWriter.saveMetadata into two methods to allow ensembles to store sub-models' metadata in a single DataFrame.  Same for DefaultParamsReader.loadMetadata\n\n## How was this patch tested?\n\nAdds standard unit tests for RF save/load\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\nAuthor: GayathriMurali <gayathri.m.softie@gmail.com>\n\nCloses #12118 from jkbradley/GayathriMurali-SPARK-13784.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/89f3befab6c150f87de2fb91b50ea8b414c69095"},{"sha":"855ed44ed31210d2001d7ce67c8fa99f8416edd3","author":{"email":"b59cd5ce8cb5ae44d26fc6984be1f84cf74771a7@databricks.com","name":"Shixiong Zhu"},"message":"[SPARK-14176][SQL] Add DataFrameWriter.trigger to set the stream batch period\n\n## What changes were proposed in this pull request?\n\nAdd a processing time trigger to control the batch processing speed\n\n## How was this patch tested?\n\nUnit tests\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #11976 from zsxwing/trigger.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/855ed44ed31210d2001d7ce67c8fa99f8416edd3"},{"sha":"5743c6476dbef50852b7f9873112a2d299966ebd","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-12981] [SQL] extract Pyhton UDF in physical plan\n\n## What changes were proposed in this pull request?\n\nCurrently we extract Python UDFs into a special logical plan EvaluatePython in analyzer, But EvaluatePython is not part of catalyst, many rules have no knowledge of it , which will break many things (for example, filter push down or column pruning).\n\nWe should treat Python UDFs as normal expressions, until we want to evaluate in physical plan, we could extract them in end of optimizer, or physical plan.\n\nThis PR extract Python UDFs in physical plan.\n\nCloses #10935\n\n## How was this patch tested?\n\nAdded regression tests.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #12127 from davies/py_udf.","distinct":true,"url":"https://api.github.com/repos/pravingadakh/spark/commits/5743c6476dbef50852b7f9873112a2d299966ebd"}]}