{"push_id":1252644032,"size":328,"distinct_size":25,"ref":"refs/heads/feature/SPARK-11373","head":"745d532b67b64149e86d2fdf0464cab616023eac","before":"4b2046fd47d58bc5814f6271aa1d67a6cdd0918c","commits":[{"sha":"d8220c1e5e94abbdb9643672b918f0d748206db9","author":{"name":"jerryshao","email":"4c4fc98b369ca68b84d880910d6c592827f07854@hortonworks.com"},"message":"[SPARK-16435][YARN][MINOR] Add warning log if initialExecutors is less than minExecutors\n\n## What changes were proposed in this pull request?\n\nCurrently if `spark.dynamicAllocation.initialExecutors` is less than `spark.dynamicAllocation.minExecutors`, Spark will automatically pick the minExecutors without any warning. While in 1.6 Spark will throw exception if configured like this. So here propose to add warning log if these parameters are configured invalidly.\n\n## How was this patch tested?\n\nUnit test added to verify the scenario.\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #14149 from jerryshao/SPARK-16435.","distinct":false,"url":"https://api.github.com/repos/steveloughran/spark/commits/d8220c1e5e94abbdb9643672b918f0d748206db9"},{"sha":"01f09b161217193b797c8c85969d17054c958615","author":{"name":"Joseph K. Bradley","email":"9b6e24a661c030e39d84e80f958c2d82d5dc3d65@databricks.com"},"message":"[SPARK-14812][ML][MLLIB][PYTHON] Experimental, DeveloperApi annotation audit for ML\n\n## What changes were proposed in this pull request?\n\nGeneral decisions to follow, except where noted:\n* spark.mllib, pyspark.mllib: Remove all Experimental annotations.  Leave DeveloperApi annotations alone.\n* spark.ml, pyspark.ml\n** Annotate Estimator-Model pairs of classes and companion objects the same way.\n** For all algorithms marked Experimental with Since tag <= 1.6, remove Experimental annotation.\n** For all algorithms marked Experimental with Since tag = 2.0, leave Experimental annotation.\n* DeveloperApi annotations are left alone, except where noted.\n* No changes to which types are sealed.\n\nExceptions where I am leaving items Experimental in spark.ml, pyspark.ml, mainly because the items are new:\n* Model Summary classes\n* MLWriter, MLReader, MLWritable, MLReadable\n* Evaluator and subclasses: There is discussion of changes around evaluating multiple metrics at once for efficiency.\n* RFormula: Its behavior may need to change slightly to match R in edge cases.\n* AFTSurvivalRegression\n* MultilayerPerceptronClassifier\n\nDeveloperApi changes:\n* ml.tree.Node, ml.tree.Split, and subclasses should no longer be DeveloperApi\n\n## How was this patch tested?\n\nN/A\n\nNote to reviewers:\n* spark.ml.clustering.LDA underwent significant changes (additional methods), so let me know if you want me to leave it Experimental.\n* Be careful to check for cases where a class should no longer be Experimental but has an Experimental method, val, or other feature.  I did not find such cases, but please verify.\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #14147 from jkbradley/experimental-audit.","distinct":false,"url":"https://api.github.com/repos/steveloughran/spark/commits/01f09b161217193b797c8c85969d17054c958615"},{"sha":"0744d84c91d6e494dea77a35e6410bc4b1849e71","author":{"name":"Burak Yavuz","email":"39bd9266538731f16447e7d968c0ba25275ede1e@gmail.com"},"message":"[SPARK-16531][SQL][TEST] Remove timezone setting from DataFrameTimeWindowingSuite\n\n## What changes were proposed in this pull request?\n\nIt's unnecessary. `QueryTest` already sets it.\n\nAuthor: Burak Yavuz <brkyvz@gmail.com>\n\nCloses #14170 from brkyvz/test-tz.","distinct":false,"url":"https://api.github.com/repos/steveloughran/spark/commits/0744d84c91d6e494dea77a35e6410bc4b1849e71"},{"sha":"51a6706b1339bb761602e33276a469f71be2cd90","author":{"name":"James Thomas","email":"4c3f70f7e07111950bf31cd977787d757ef55837@gmail.com"},"message":"[SPARK-16114][SQL] updated structured streaming guide\n\n## What changes were proposed in this pull request?\n\nUpdated structured streaming programming guide with new windowed example.\n\n## How was this patch tested?\n\nDocs\n\nAuthor: James Thomas <jamesjoethomas@gmail.com>\n\nCloses #14183 from jjthomas/ss_docs_update.","distinct":false,"url":"https://api.github.com/repos/steveloughran/spark/commits/51a6706b1339bb761602e33276a469f71be2cd90"},{"sha":"b4baf086ca380a46d953f2710184ad9eee3a045e","author":{"name":"Felix Cheung","email":"00eb71019ad7acfd67cff51fce666aad32ce6eed@hotmail.com"},"message":"[SPARKR][MINOR] R examples and test updates\n\n## What changes were proposed in this pull request?\n\nMinor example updates\n\n## How was this patch tested?\n\nmanual\n\nshivaram\n\nAuthor: Felix Cheung <felixcheung_m@hotmail.com>\n\nCloses #14171 from felixcheung/rexample.","distinct":false,"url":"https://api.github.com/repos/steveloughran/spark/commits/b4baf086ca380a46d953f2710184ad9eee3a045e"},{"sha":"fb2e8eeb0b1e56bea535165f7a3bec6558b3f4a3","author":{"name":"Felix Cheung","email":"00eb71019ad7acfd67cff51fce666aad32ce6eed@hotmail.com"},"message":"[SPARKR][DOCS][MINOR] R programming guide to include csv data source example\n\n## What changes were proposed in this pull request?\n\nMinor documentation update for code example, code style, and missed reference to \"sparkR.init\"\n\n## How was this patch tested?\n\nmanual\n\nshivaram\n\nAuthor: Felix Cheung <felixcheung_m@hotmail.com>\n\nCloses #14178 from felixcheung/rcsvprogrammingguide.","distinct":false,"url":"https://api.github.com/repos/steveloughran/spark/commits/fb2e8eeb0b1e56bea535165f7a3bec6558b3f4a3"},{"sha":"c5ec879828369ec1d21acd7f18a792306634ff74","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-16482][SQL] Describe Table Command for Tables Requiring Runtime Inferred Schema\n\n#### What changes were proposed in this pull request?\nIf we create a table pointing to a parquet/json datasets without specifying the schema, describe table command does not show the schema at all. It only shows `# Schema of this table is inferred at runtime`. In 1.6, describe table does show the schema of such a table.\n\n~~For data source tables, to infer the schema, we need to load the data source tables at runtime. Thus, this PR calls the function `lookupRelation`.~~\n\nFor data source tables, we infer the schema before table creation. Thus, this PR set the inferred schema as the table schema when table creation.\n\n#### How was this patch tested?\nAdded test cases\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #14148 from gatorsmile/describeSchema.","distinct":false,"url":"https://api.github.com/repos/steveloughran/spark/commits/c5ec879828369ec1d21acd7f18a792306634ff74"},{"sha":"a5f51e21627c1bcfc62829a3a962707abf41a452","author":{"name":"Joseph K. Bradley","email":"9b6e24a661c030e39d84e80f958c2d82d5dc3d65@databricks.com"},"message":"[SPARK-16485][ML][DOC] Fix privacy of GLM members, rename sqlDataTypes for ML, doc fixes\n\n## What changes were proposed in this pull request?\n\nFixing issues found during 2.0 API checks:\n* GeneralizedLinearRegressionModel: linkObj, familyObj, familyAndLink should not be exposed\n* sqlDataTypes: name does not follow conventions. Do we need to expose it?\n* Evaluator: inconsistent doc between evaluate and isLargerBetter\n* MinMaxScaler: math rendering --> hard to make it great, but I'll change it a little\n* GeneralizedLinearRegressionSummary: aic doc is incorrect --> will change to use more common name\n\n## How was this patch tested?\n\nExisting unit tests.  Docs generated locally.  (MinMaxScaler is improved a tiny bit.)\n\nAuthor: Joseph K. Bradley <joseph@databricks.com>\n\nCloses #14187 from jkbradley/final-api-check-2.0.","distinct":false,"url":"https://api.github.com/repos/steveloughran/spark/commits/a5f51e21627c1bcfc62829a3a962707abf41a452"},{"sha":"9c530576a44cbeb956db94e7fdd1fad50bd62973","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-16536][SQL][PYSPARK][MINOR] Expose `sql` in PySpark Shell\n\n## What changes were proposed in this pull request?\n\nThis PR exposes `sql` in PySpark Shell like Scala/R Shells for consistency.\n\n**Background**\n * Scala\n ```scala\nscala> sql(\"select 1 a\")\nres0: org.apache.spark.sql.DataFrame = [a: int]\n```\n\n * R\n ```r\n> sql(\"select 1\")\nSparkDataFrame[1:int]\n```\n\n**Before**\n * Python\n\n ```python\n>>> sql(\"select 1 a\")\nTraceback (most recent call last):\n  File \"<stdin>\", line 1, in <module>\nNameError: name 'sql' is not defined\n```\n\n**After**\n * Python\n\n ```python\n>>> sql(\"select 1 a\")\nDataFrame[a: int]\n```\n\n## How was this patch tested?\n\nManual.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #14190 from dongjoon-hyun/SPARK-16536.","distinct":false,"url":"https://api.github.com/repos/steveloughran/spark/commits/9c530576a44cbeb956db94e7fdd1fad50bd62973"},{"sha":"39c836e976fcae51568bed5ebab28e148383b5d4","author":{"name":"Liwei Lin","email":"878182135bcb2086b9b1c72277449d6e06bbf76f@gmail.com"},"message":"[SPARK-16503] SparkSession should provide Spark version\n\n## What changes were proposed in this pull request?\n\nThis patch enables SparkSession to provide spark version.\n\n## How was this patch tested?\n\nManual test:\n\n```\nscala> sc.version\nres0: String = 2.1.0-SNAPSHOT\n\nscala> spark.version\nres1: String = 2.1.0-SNAPSHOT\n```\n\n```\n>>> sc.version\nu'2.1.0-SNAPSHOT'\n>>> spark.version\nu'2.1.0-SNAPSHOT'\n```\n\nAuthor: Liwei Lin <lwlin7@gmail.com>\n\nCloses #14165 from lw-lin/add-version.","distinct":false,"url":"https://api.github.com/repos/steveloughran/spark/commits/39c836e976fcae51568bed5ebab28e148383b5d4"},{"sha":"db7317ac3c2fd2a11088d10060f168178dc99664","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-16448] RemoveAliasOnlyProject should not remove alias with metadata\n\n## What changes were proposed in this pull request?\n\n`Alias` with metadata is not a no-op and we should not strip it in `RemoveAliasOnlyProject` rule.\nThis PR also did some improvement for this rule:\n\n1. extend the semantic of `alias-only`. Now we allow the project list to be partially aliased.\n2. add unit test for this rule.\n\n## How was this patch tested?\n\nnew `RemoveAliasOnlyProjectSuite`\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #14106 from cloud-fan/bug.","distinct":false,"url":"https://api.github.com/repos/steveloughran/spark/commits/db7317ac3c2fd2a11088d10060f168178dc99664"},{"sha":"252d4f27f23b547777892bcea25a2cea62d8cbab","author":{"name":"WeichenXu","email":"7a440c883b23bc43b7a6fdb423cbfd8e75175a83@outlook.com"},"message":"[SPARK-16500][ML][MLLIB][OPTIMIZER] add LBFGS convergence warning for all used place in MLLib\n\n## What changes were proposed in this pull request?\n\nAdd warning_for the following case when LBFGS training not actually convergence:\n\n1) LogisticRegression\n2) AFTSurvivalRegression\n3) LBFGS algorithm wrapper in mllib package\n\n## How was this patch tested?\n\nN/A\n\nAuthor: WeichenXu <WeichenXu123@outlook.com>\n\nCloses #14157 from WeichenXu123/add_lbfgs_convergence_warning_for_all_used_place.","distinct":false,"url":"https://api.github.com/repos/steveloughran/spark/commits/252d4f27f23b547777892bcea25a2cea62d8cbab"},{"sha":"e3f8a033679261aaee15bda0f970a1890411e743","author":{"name":"Bryan Cutler","email":"62bf89cc0e70e22c1fc5a3d8c3b77c0895eb33c3@gmail.com"},"message":"[SPARK-16403][EXAMPLES] Cleanup to remove unused imports, consistent style, minor fixes\n\n## What changes were proposed in this pull request?\n\nCleanup of examples, mostly from PySpark-ML to fix minor issues:  unused imports, style consistency, pipeline_example is a duplicate, use future print funciton, and a spelling error.\n\n* The \"Pipeline Example\" is duplicated by \"Simple Text Classification Pipeline\" in Scala, Python, and Java.\n\n* \"Estimator Transformer Param Example\" is duplicated by \"Simple Params Example\" in Scala, Python and Java\n\n* Synced random_forest_classifier_example.py with Scala by adding IndexToString label converted\n\n* Synced train_validation_split.py (in Scala ModelSelectionViaTrainValidationExample) by adjusting data split, adding grid for intercept.\n\n* RegexTokenizer was doing nothing in tokenizer_example.py and JavaTokenizerExample.java, synced with Scala version\n\n## How was this patch tested?\nlocal tests and run modified examples\n\nAuthor: Bryan Cutler <cutlerb@gmail.com>\n\nCloses #14081 from BryanCutler/examples-cleanup-SPARK-16403.","distinct":false,"url":"https://api.github.com/repos/steveloughran/spark/commits/e3f8a033679261aaee15bda0f970a1890411e743"},{"sha":"c4bc2ed844ea045d2e8218154690b5b2b023f1e5","author":{"name":"jerryshao","email":"4c4fc98b369ca68b84d880910d6c592827f07854@hortonworks.com"},"message":"[SPARK-14963][MINOR][YARN] Fix typo in YarnShuffleService recovery file name\n\n## What changes were proposed in this pull request?\n\nDue to the changes of [SPARK-14963](https://issues.apache.org/jira/browse/SPARK-14963), external shuffle recovery file name is changed mistakenly, so here change it back to the previous file name.\n\nThis only affects the master branch, branch-2.0 is correct [here](https://github.com/apache/spark/blob/branch-2.0/common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java#L195).\n\n## How was this patch tested?\n\nN/A\n\nAuthor: jerryshao <sshao@hortonworks.com>\n\nCloses #14197 from jerryshao/fix-typo-file-name.","distinct":false,"url":"https://api.github.com/repos/steveloughran/spark/commits/c4bc2ed844ea045d2e8218154690b5b2b023f1e5"},{"sha":"b7b5e17876f65c6644505c356f1a0db24ce1d142","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"[SPARK-16505][YARN] Optionally propagate error during shuffle service startup.\n\nThis prevents the NM from starting when something is wrong, which would\nlead to later errors which are confusing and harder to debug.\n\nAdded a unit test to verify startup fails if something is wrong.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #14162 from vanzin/SPARK-16505.","distinct":false,"url":"https://api.github.com/repos/steveloughran/spark/commits/b7b5e17876f65c6644505c356f1a0db24ce1d142"},{"sha":"1b5c9e52a7d5cdd3b4da1334ddff0518a8e14505","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-16530][SQL][TRIVIAL] Wrong Parser Keyword in ALTER TABLE CHANGE COLUMN\n\n#### What changes were proposed in this pull request?\nBased on the [Hive SQL syntax](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ChangeColumnName/Type/Position/Comment), the command to change column name/type/position/comment is `ALTER TABLE CHANGE COLUMN`. However, in our .g4 file, it is `ALTER TABLE CHANGE COLUMNS`. Because it is the last optional keyword, it does not take any effect. Thus, I put the issue as a Trivial level.\n\ncc hvanhovell\n\n#### How was this patch tested?\nExisting test cases\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #14186 from gatorsmile/changeColumns.","distinct":false,"url":"https://api.github.com/repos/steveloughran/spark/commits/1b5c9e52a7d5cdd3b4da1334ddff0518a8e14505"},{"sha":"56183b84fb64ea13977d89ec55a9dd3997b4dacf","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-16543][SQL] Rename the columns of `SHOW PARTITION/COLUMNS` commands\n\n## What changes were proposed in this pull request?\n\nThis PR changes the name of columns returned by `SHOW PARTITION` and `SHOW COLUMNS` commands. Currently, both commands uses `result` as a column name.\n\n**Comparison: Column Name**\n\nCommand|Spark(Before)|Spark(After)|Hive\n----------|--------------|------------|-----\nSHOW PARTITIONS|result|partition|partition\nSHOW COLUMNS|result|col_name|field\n\nNote that Spark/Hive uses `col_name` in `DESC TABLES`. So, this PR chooses `col_name` for consistency among Spark commands.\n\n**Before**\n```scala\nscala> sql(\"show partitions p\").show()\n+------+\n|result|\n+------+\n|   b=2|\n+------+\n\nscala> sql(\"show columns in p\").show()\n+------+\n|result|\n+------+\n|     a|\n|     b|\n+------+\n```\n\n**After**\n```scala\nscala> sql(\"show partitions p\").show\n+---------+\n|partition|\n+---------+\n|      b=2|\n+---------+\n\nscala> sql(\"show columns in p\").show\n+--------+\n|col_name|\n+--------+\n|       a|\n|       b|\n+--------+\n```\n\n## How was this patch tested?\n\nManual.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #14199 from dongjoon-hyun/SPARK-16543.","distinct":false,"url":"https://api.github.com/repos/steveloughran/spark/commits/56183b84fb64ea13977d89ec55a9dd3997b4dacf"},{"sha":"093ebbc628699b40f091b5b7083c119fffa9314b","author":{"name":"Sun Rui","email":"6f09da93fe840a44f8b7d461e5cd331e1e929387@gmail.com"},"message":"[SPARK-16509][SPARKR] Rename window.partitionBy and window.orderBy to windowPartitionBy and windowOrderBy.\n\n## What changes were proposed in this pull request?\nRename window.partitionBy and window.orderBy to windowPartitionBy and windowOrderBy to pass CRAN package check.\n\n## How was this patch tested?\nSparkR unit tests.\n\nAuthor: Sun Rui <sunrui2016@gmail.com>\n\nCloses #14192 from sun-rui/SPARK-16509.","distinct":false,"url":"https://api.github.com/repos/steveloughran/spark/commits/093ebbc628699b40f091b5b7083c119fffa9314b"},{"sha":"12005c88fb24168d57b577cff73eddcd9d8963fc","author":{"name":"Felix Cheung","email":"00eb71019ad7acfd67cff51fce666aad32ce6eed@hotmail.com"},"message":"[SPARK-16538][SPARKR] fix R call with namespace operator on SparkSession functions\n\n## What changes were proposed in this pull request?\n\nFix function routing to work with and without namespace operator `SparkR::createDataFrame`\n\n## How was this patch tested?\n\nmanual, unit tests\n\nshivaram\n\nAuthor: Felix Cheung <felixcheung_m@hotmail.com>\n\nCloses #14195 from felixcheung/rroutedefault.","distinct":false,"url":"https://api.github.com/repos/steveloughran/spark/commits/12005c88fb24168d57b577cff73eddcd9d8963fc"},{"sha":"c576f9fb90853cce2e8e5dcc32a536a0f49cbbd8","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-16529][SQL][TEST] `withTempDatabase` should set `default` database before dropping\n\n## What changes were proposed in this pull request?\n\n`SQLTestUtils.withTempDatabase` is a frequently used test harness to setup a temporary table and clean up finally. This issue improves like the following for usability.\n\n```scala\n-    try f(dbName) finally spark.sql(s\"DROP DATABASE $dbName CASCADE\")\n+    try f(dbName) finally {\n+      if (spark.catalog.currentDatabase == dbName) {\n+        spark.sql(s\"USE ${DEFAULT_DATABASE}\")\n+      }\n+      spark.sql(s\"DROP DATABASE $dbName CASCADE\")\n+    }\n```\n\nIn case of forgetting to reset the databaes, `withTempDatabase` will not raise Exception.\n\n## How was this patch tested?\n\nThis improves test harness.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #14184 from dongjoon-hyun/SPARK-16529.","distinct":false,"url":"https://api.github.com/repos/steveloughran/spark/commits/c576f9fb90853cce2e8e5dcc32a536a0f49cbbd8"}]}
{"actor":{"display_login":"steveloughran"}}