{"push_id":1096628383,"size":4,"distinct_size":4,"ref":"refs/heads/master","head":"a45647746d1efb90cb8bc142c2ef110a0db9bc9f","before":"bc3760d405cc8c3ffcd957b188afa8b7e3b1f824","commits":[{"sha":"6c12e801e84565265d2939b920901d1805d5935f","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-15029] improve error message for Generate\n\n## What changes were proposed in this pull request?\n\nThis PR improve the error message for `Generate` in 3 cases:\n\n1. generator is nested in expressions, e.g. `SELECT explode(list) + 1 FROM tbl`\n2. generator appears more than one time in SELECT, e.g. `SELECT explode(list), explode(list) FROM tbl`\n3. generator appears in other operator which is not project, e.g. `SELECT * FROM tbl SORT BY explode(list)`\n\n## How was this patch tested?\n\nnew tests in `AnalysisErrorSuite`\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #12810 from cloud-fan/bug.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/6c12e801e84565265d2939b920901d1805d5935f"},{"sha":"f152fae306dc75565cb4648ee1211416d7c0bb23","author":{"email":"e6f14efef9fae0ae8f92d04007faea68035873ce@databricks.com","name":"Cheng Lian"},"message":"[SPARK-14127][SQL] Native \"DESC [EXTENDED | FORMATTED] <table>\" DDL command\n\n## What changes were proposed in this pull request?\n\nThis PR implements native `DESC [EXTENDED | FORMATTED] <table>` DDL command. Sample output:\n\n```\nscala> spark.sql(\"desc extended src\").show(100, truncate = false)\n+----------------------------+---------------------------------+-------+\n|col_name                    |data_type                        |comment|\n+----------------------------+---------------------------------+-------+\n|key                         |int                              |       |\n|value                       |string                           |       |\n|                            |                                 |       |\n|# Detailed Table Information|CatalogTable(`default`.`src`, ...|       |\n+----------------------------+---------------------------------+-------+\n\nscala> spark.sql(\"desc formatted src\").show(100, truncate = false)\n+----------------------------+----------------------------------------------------------+-------+\n|col_name                    |data_type                                                 |comment|\n+----------------------------+----------------------------------------------------------+-------+\n|key                         |int                                                       |       |\n|value                       |string                                                    |       |\n|                            |                                                          |       |\n|# Detailed Table Information|                                                          |       |\n|Database:                   |default                                                   |       |\n|Owner:                      |lian                                                      |       |\n|Create Time:                |Mon Jan 04 17:06:00 CST 2016                              |       |\n|Last Access Time:           |Thu Jan 01 08:00:00 CST 1970                              |       |\n|Location:                   |hdfs://localhost:9000/user/hive/warehouse_hive121/src     |       |\n|Table Type:                 |MANAGED                                                   |       |\n|Table Parameters:           |                                                          |       |\n|  transient_lastDdlTime     |1451898360                                                |       |\n|                            |                                                          |       |\n|# Storage Information       |                                                          |       |\n|SerDe Library:              |org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe        |       |\n|InputFormat:                |org.apache.hadoop.mapred.TextInputFormat                  |       |\n|OutputFormat:               |org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat|       |\n|Num Buckets:                |-1                                                        |       |\n|Bucket Columns:             |[]                                                        |       |\n|Sort Columns:               |[]                                                        |       |\n|Storage Desc Parameters:    |                                                          |       |\n|  serialization.format      |1                                                         |       |\n+----------------------------+----------------------------------------------------------+-------+\n```\n\n## How was this patch tested?\n\nA test case is added to `HiveDDLSuite` to check command output.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #12844 from liancheng/spark-14127-desc-table.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/f152fae306dc75565cb4648ee1211416d7c0bb23"},{"sha":"abecbcd5e9598471b705a2f701731af1adc9d48b","author":{"email":"736b6fbd35c0eb103ed5c6229fc55be973c048e9@codilime.com","name":"Dominik Jastrzębski"},"message":"[SPARK-14844][ML] Add setFeaturesCol and setPredictionCol to KMeansM…\n\n## What changes were proposed in this pull request?\n\nIntroduction of setFeaturesCol and setPredictionCol methods to KMeansModel in ML library.\n\n## How was this patch tested?\n\nBy running KMeansSuite.\n\nAuthor: Dominik Jastrzębski <dominik.jastrzebski@codilime.com>\n\nCloses #12609 from dominik-jastrzebski/master.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/abecbcd5e9598471b705a2f701731af1adc9d48b"},{"sha":"a45647746d1efb90cb8bc142c2ef110a0db9bc9f","author":{"email":"56556026bd87097401a2aa2d3035b4c3c9d670a7@gmail.com","name":"Dhruve Ashar"},"message":"[SPARK-4224][CORE][YARN] Support group acls\n\n## What changes were proposed in this pull request?\nCurrently only a list of users can be specified for view and modify acls. This change enables a group of admins/devs/users to be provisioned for viewing and modifying Spark jobs.\n\n**Changes Proposed in the fix**\nThree new corresponding config entries have been added where the user can specify the groups to be given access.\n\n```\nspark.admin.acls.groups\nspark.modify.acls.groups\nspark.ui.view.acls.groups\n```\n\nNew config entries were added because specifying the users and groups explicitly is a better and cleaner way compared to specifying them in the existing config entry using a delimiter.\n\nA generic trait has been introduced to provide the user to group mapping which makes it pluggable to support a variety of mapping protocols - similar to the one used in hadoop. A default unix shell based implementation has been provided.\nCustom user to group mapping protocol can be specified and configured by the entry ```spark.user.groups.mapping```\n\n**How the patch was Tested**\nWe ran different spark jobs setting the config entries in combinations of admin, modify and ui acls. For modify acls we tried killing the job stages from the ui and using yarn commands. For view acls we tried accessing the UI tabs and the logs. Headless accounts were used to launch these jobs and different users tried to modify and view the jobs to ensure that the groups mapping applied correctly.\n\nAdditional Unit tests have been added without modifying the existing ones. These test for different ways of setting the acls through configuration and/or API and validate the expected behavior.\n\nAuthor: Dhruve Ashar <dhruveashar@gmail.com>\n\nCloses #12760 from dhruve/impr/SPARK-4224.","distinct":true,"url":"https://api.github.com/repos/zzcclp/spark/commits/a45647746d1efb90cb8bc142c2ef110a0db9bc9f"}]}