{"push_id":1241854918,"size":134,"distinct_size":134,"ref":"refs/heads/master","head":"0e89b8c5f764e14b2f7f0a944d6305f07a5af4ae","before":"dcd7e949f783288cbd814cc360d4c86dcce097c9","commits":[{"sha":"7119bfa1e902da2d7346ce627bcf68b0d83298a0","author":{"name":"WeichenXu","email":"7a440c883b23bc43b7a6fdb423cbfd8e75175a83@outlook.com"},"message":"update.","distinct":true,"url":"https://api.github.com/repos/WeichenXu123/wchen-spark/commits/7119bfa1e902da2d7346ce627bcf68b0d83298a0"},{"sha":"a9ad875dc687da6212544d4b8aa88691b9123f26","author":{"name":"WeichenXu","email":"7a440c883b23bc43b7a6fdb423cbfd8e75175a83@outlook.com"},"message":"add robust regression","distinct":true,"url":"https://api.github.com/repos/WeichenXu123/wchen-spark/commits/a9ad875dc687da6212544d4b8aa88691b9123f26"},{"sha":"0bd76e872b60cb80295fc12654e370cf22390056","author":{"name":"Liwei Lin","email":"878182135bcb2086b9b1c72277449d6e06bbf76f@gmail.com"},"message":"[SPARK-16620][CORE] Add back the tokenization process in `RDD.pipe(command: String)`\n\n## What changes were proposed in this pull request?\n\nCurrently `RDD.pipe(command: String)`:\n- works only when the command is specified without any options, such as `RDD.pipe(\"wc\")`\n- does NOT work when the command is specified with some options, such as `RDD.pipe(\"wc -l\")`\n\nThis is a regression from Spark 1.6.\n\nThis patch adds back the tokenization process in `RDD.pipe(command: String)` to fix this regression.\n\n## How was this patch tested?\nAdded a test which:\n- would pass in `1.6`\n- _[prior to this patch]_ would fail in `master`\n- _[after this patch]_ would pass in `master`\n\nAuthor: Liwei Lin <lwlin7@gmail.com>\n\nCloses #14256 from lw-lin/rdd-pipe.","distinct":true,"url":"https://api.github.com/repos/WeichenXu123/wchen-spark/commits/0bd76e872b60cb80295fc12654e370cf22390056"},{"sha":"162d04a30e38bb83d35865679145f8ea80b84c26","author":{"name":"Dongjoon Hyun","email":"1bf3a3c57610939bd1558dffb564906c959abecb@apache.org"},"message":"[SPARK-16602][SQL] `Nvl` function should support numeric-string cases\n\n## What changes were proposed in this pull request?\n\n`Nvl` function should support numeric-straing cases like Hive/Spark1.6. Currently, `Nvl` finds the tightest common types among numeric types. This PR extends that to consider `String` type, too.\n\n```scala\n- TypeCoercion.findTightestCommonTypeOfTwo(left.dataType, right.dataType).map { dtype =>\n+ TypeCoercion.findTightestCommonTypeToString(left.dataType, right.dataType).map { dtype =>\n```\n\n**Before**\n```scala\nscala> sql(\"select nvl('0', 1)\").collect()\norg.apache.spark.sql.AnalysisException: cannot resolve `nvl(\"0\", 1)` due to data type mismatch:\ninput to function coalesce should all be the same type, but it's [string, int]; line 1 pos 7\n```\n\n**After**\n```scala\nscala> sql(\"select nvl('0', 1)\").collect()\nres0: Array[org.apache.spark.sql.Row] = Array([0])\n```\n\n## How was this patch tested?\n\nPass the Jenkins tests.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #14251 from dongjoon-hyun/SPARK-16602.","distinct":true,"url":"https://api.github.com/repos/WeichenXu123/wchen-spark/commits/162d04a30e38bb83d35865679145f8ea80b84c26"},{"sha":"2ae7b88a07140e012b6c60db3c4a2a8ca360c684","author":{"name":"Yin Huai","email":"7e8362bd012819bee68b9375d9cefa24afbef722@databricks.com"},"message":"[SPARK-15705][SQL] Change the default value of spark.sql.hive.convertMetastoreOrc to false.\n\n## What changes were proposed in this pull request?\nIn 2.0, we add a new logic to convert HiveTableScan on ORC tables to Spark's native code path. However, during this conversion, we drop the original metastore schema (https://issues.apache.org/jira/browse/SPARK-15705). Because of this regression, I am changing the default value of `spark.sql.hive.convertMetastoreOrc` to false.\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #14267 from yhuai/SPARK-15705-changeDefaultValue.","distinct":true,"url":"https://api.github.com/repos/WeichenXu123/wchen-spark/commits/2ae7b88a07140e012b6c60db3c4a2a8ca360c684"},{"sha":"004e29cba518684d239d2d1661dce7c894a79f14","author":{"name":"Andrew Duffy","email":"0ddec2d7e784f56fe0bd4fe295b4b53464af379c@aduffy.org"},"message":"[SPARK-14702] Make environment of SparkLauncher launched process more configurable\n\n## What changes were proposed in this pull request?\n\nAdds a few public methods to `SparkLauncher` to allow configuring some extra features of the `ProcessBuilder`, including the working directory, output and error stream redirection.\n\n## How was this patch tested?\n\nUnit testing + simple Spark driver programs\n\nAuthor: Andrew Duffy <root@aduffy.org>\n\nCloses #14201 from andreweduffy/feature/launcher.","distinct":true,"url":"https://api.github.com/repos/WeichenXu123/wchen-spark/commits/004e29cba518684d239d2d1661dce7c894a79f14"},{"sha":"9674af6f6f81066139ea675de724f951bd0d49c9","author":{"name":"WeichenXu","email":"7a440c883b23bc43b7a6fdb423cbfd8e75175a83@outlook.com"},"message":"[SPARK-16568][SQL][DOCUMENTATION] update sql programming guide refreshTable API in python code\n\n## What changes were proposed in this pull request?\n\nupdate `refreshTable` API in python code of the sql-programming-guide.\n\nThis API is added in SPARK-15820\n\n## How was this patch tested?\n\nN/A\n\nAuthor: WeichenXu <WeichenXu123@outlook.com>\n\nCloses #14220 from WeichenXu123/update_sql_doc_catalog.","distinct":true,"url":"https://api.github.com/repos/WeichenXu123/wchen-spark/commits/9674af6f6f81066139ea675de724f951bd0d49c9"},{"sha":"fc23263623d5dcd1167fa93c094fe41ace77c326","author":{"name":"Shivaram Venkataraman","email":"a88946ecf9c024d7b7eca4dac2be9c7e472cd2f5@cs.berkeley.edu"},"message":"[SPARK-10683][SPARK-16510][SPARKR] Move SparkR include jar test to SparkSubmitSuite\n\n## What changes were proposed in this pull request?\n\nThis change moves the include jar test from R to SparkSubmitSuite and uses a dynamically compiled jar. This helps us remove the binary jar from the R package and solves both the CRAN warnings and the lack of source being available for this jar.\n\n## How was this patch tested?\nSparkR unit tests, SparkSubmitSuite, check-cran.sh\n\nAuthor: Shivaram Venkataraman <shivaram@cs.berkeley.edu>\n\nCloses #14243 from shivaram/sparkr-jar-move.","distinct":true,"url":"https://api.github.com/repos/WeichenXu123/wchen-spark/commits/fc23263623d5dcd1167fa93c094fe41ace77c326"},{"sha":"75146be6ba5e9f559f5f15430310bb476ee0812c","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"[SPARK-16632][SQL] Respect Hive schema when merging parquet schema.\n\nWhen Hive (or at least certain versions of Hive) creates parquet files\ncontaining tinyint or smallint columns, it stores them as int32, but\ndoesn't annotate the parquet field as containing the corresponding\nint8 / int16 data. When Spark reads those files using the vectorized\nreader, it follows the parquet schema for these fields, but when\nactually reading the data it tries to use the type fetched from\nthe metastore, and then fails because data has been loaded into the\nwrong fields in OnHeapColumnVector.\n\nSo instead of blindly trusting the parquet schema, check whether the\nCatalyst-provided schema disagrees with it, and adjust the types so\nthat the necessary metadata is present when loading the data into\nthe ColumnVector instance.\n\nTested with unit tests and with tests that create byte / short columns\nin Hive and try to read them from Spark.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #14272 from vanzin/SPARK-16632.","distinct":true,"url":"https://api.github.com/repos/WeichenXu123/wchen-spark/commits/75146be6ba5e9f559f5f15430310bb476ee0812c"},{"sha":"aaef5d3dfc34c2840b7075a957b37cd2400976e2","author":{"name":"WeichenXu","email":"7a440c883b23bc43b7a6fdb423cbfd8e75175a83@outlook.com"},"message":"Merge branch 'master' into wchen-spark","distinct":true,"url":"https://api.github.com/repos/WeichenXu123/wchen-spark/commits/aaef5d3dfc34c2840b7075a957b37cd2400976e2"},{"sha":"0dc79ffd1cbb45e69a35e3f5334c9a13290037a0","author":{"name":"Anthony Truchet","email":"26d8dd5b3efdf5761f8c8003329681520fd47d3c@criteo.com"},"message":"[SPARK-16440][MLLIB] Destroy broadcasted variables even on driver\n\n## What changes were proposed in this pull request?\nForgotten broadcasted variables were persisted into a previous #PR 14153). This PR turns those `unpersist()` into `destroy()` so that memory is freed even on the driver.\n\n## How was this patch tested?\nUnit Tests in Word2VecSuite were run locally.\n\nThis contribution is done on behalf of Criteo, according to the\nterms of the Apache license 2.0.\n\nAuthor: Anthony Truchet <a.truchet@criteo.com>\n\nCloses #14268 from AnthonyTruchet/SPARK-16440.","distinct":true,"url":"https://api.github.com/repos/WeichenXu123/wchen-spark/commits/0dc79ffd1cbb45e69a35e3f5334c9a13290037a0"},{"sha":"95abbe537751929353d18b733f6267c3287b6047","author":{"name":"Weiqing Yang","email":"c0c88f1153e01312d8f77a3074d15074c5d139c9@gmail.com"},"message":"[SPARK-15923][YARN] Spark Application rest api returns 'no such app: …\n\n## What changes were proposed in this pull request?\nUpdate monitoring.md.\n\n…<appId>'\n\nAuthor: Weiqing Yang <yangweiqing001@gmail.com>\n\nCloses #14163 from Sherry302/master.","distinct":true,"url":"https://api.github.com/repos/WeichenXu123/wchen-spark/commits/95abbe537751929353d18b733f6267c3287b6047"},{"sha":"4b079dc3964dbe0f4d7839d39512d0400122b520","author":{"name":"Sean Owen","email":"4c59f357315334980589bde1049040bf6de6532c@cloudera.com"},"message":"[SPARK-16613][CORE] RDD.pipe returns values for empty partitions\n\n## What changes were proposed in this pull request?\n\nDocument RDD.pipe semantics; don't execute process for empty input partitions.\n\nNote this includes the fix in https://github.com/apache/spark/pull/14256 because it's necessary to even test this. One or the other will merge the fix.\n\n## How was this patch tested?\n\nJenkins tests including new test.\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #14260 from srowen/SPARK-16613.","distinct":true,"url":"https://api.github.com/repos/WeichenXu123/wchen-spark/commits/4b079dc3964dbe0f4d7839d39512d0400122b520"},{"sha":"b9bab4dcf6cec5ec9e9860871f9dd1ec633a1d22","author":{"name":"Kishor Patil","email":"53fdec9933651013cff2c5b00d76e765b1cc9456@yahoo-inc.com"},"message":"[SPARK-15951] Change Executors Page to use datatables to support sorting columns and searching\n\n1. Create the executorspage-template.html for displaying application information in datables.\n2. Added REST API endpoint \"allexecutors\" to be able to see all executors created for particular job.\n3. The executorspage.js uses jQuery to access the data from /api/v1/applications/appid/allexecutors REST API, and use DataTable to display executors for the application. It also, generates summary of dead/live and total executors created during life of the application.\n4. Similar changes applicable to Executors Page on history server for a given application.\n\nSnapshots for how it looks like now:\n<img width=\"938\" alt=\"screen shot 2016-06-14 at 2 45 44 pm\" src=\"https://cloud.githubusercontent.com/assets/6090397/16060092/ad1de03a-324b-11e6-8469-9eaa3f2548b5.png\">\n\nNew Executors Page screenshot looks like this:\n<img width=\"1436\" alt=\"screen shot 2016-06-15 at 10 12 01 am\" src=\"https://cloud.githubusercontent.com/assets/6090397/16085514/ee7004f0-32e1-11e6-9340-33d91e407f2b.png\">\n\nAuthor: Kishor Patil <kpatil@yahoo-inc.com>\n\nCloses #13670 from kishorvpatil/execTemplates.","distinct":true,"url":"https://api.github.com/repos/WeichenXu123/wchen-spark/commits/b9bab4dcf6cec5ec9e9860871f9dd1ec633a1d22"},{"sha":"e3cd5b3050711af69fc1dfc518b11bf1a86b6a4c","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"[SPARK-16634][SQL] Workaround JVM bug by moving some code out of ctor.\n\nSome 1.7 JVMs have a bug that is triggered by certain Scala-generated\nbytecode. GenericArrayData suffers from that and fails to load in certain\nJVMs.\n\nMoving the offending code out of the constructor and into a helper method\navoids the issue.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #14271 from vanzin/SPARK-16634.","distinct":true,"url":"https://api.github.com/repos/WeichenXu123/wchen-spark/commits/e3cd5b3050711af69fc1dfc518b11bf1a86b6a4c"},{"sha":"e651900bd562cc29a3eb13e92a5147979e347f61","author":{"name":"Cheng Lian","email":"924ffd5d9c6cd2cddca5f635d9a74ad9e1c46f4d@databricks.com"},"message":"[SPARK-16344][SQL] Decoding Parquet array of struct with a single field named \"element\"\n\n## What changes were proposed in this pull request?\n\nDue to backward-compatibility reasons, the following Parquet schema is ambiguous:\n\n```\noptional group f (LIST) {\n  repeated group list {\n    optional group element {\n      optional int32 element;\n    }\n  }\n}\n```\n\nAccording to the parquet-format spec, when interpreted as a standard 3-level layout, this type is equivalent to the following SQL type:\n\n```\nARRAY<STRUCT<element: INT>>\n```\n\nHowever, when interpreted as a legacy 2-level layout, it's equivalent to\n\n```\nARRAY<STRUCT<element: STRUCT<element: INT>>>\n```\n\nHistorically, to disambiguate these cases, we employed two methods:\n\n- `ParquetSchemaConverter.isElementType()`\n\n  Used to disambiguate the above cases while converting Parquet types to Spark types.\n\n- `ParquetRowConverter.isElementType()`\n\n  Used to disambiguate the above cases while instantiating row converters that convert Parquet records to Spark rows.\n\nUnfortunately, these two methods make different decision about the above problematic Parquet type, and caused SPARK-16344.\n\n`ParquetRowConverter.isElementType()` is necessary for Spark 1.4 and earlier versions because Parquet requested schemata are directly converted from Spark schemata in these versions. The converted Parquet schemata may be incompatible with actual schemata of the underlying physical files when the files are written by a system/library that uses a schema conversion scheme that is different from Spark when writing Parquet LIST and MAP fields.\n\nIn Spark 1.5, Parquet requested schemata are always properly tailored from schemata of physical files to be read. Thus `ParquetRowConverter.isElementType()` is no longer necessary. This PR replaces this method with a simply yet accurate scheme: whenever an ambiguous Parquet type is hit, convert the type in question back to a Spark type using `ParquetSchemaConverter` and check whether it matches the corresponding Spark type.\n\n## How was this patch tested?\n\nNew test cases added in `ParquetHiveCompatibilitySuite` and `ParquetQuerySuite`.\n\nAuthor: Cheng Lian <lian@databricks.com>\n\nCloses #14014 from liancheng/spark-16344-for-master-and-2.0.","distinct":true,"url":"https://api.github.com/repos/WeichenXu123/wchen-spark/commits/e651900bd562cc29a3eb13e92a5147979e347f61"},{"sha":"75a06aa256aa256c112555609a93c1e1dbb1cb4b","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"[SPARK-16272][CORE] Allow config values to reference conf, env, system props.\n\nThis allows configuration to be more flexible, for example, when the cluster does\nnot have a homogeneous configuration (e.g. packages are installed on different\npaths in different nodes). By allowing one to reference the environment from\nthe conf, it becomes possible to work around those in certain cases.\n\nAs part of the implementation, ConfigEntry now keeps track of all \"known\" configs\n(i.e. those created through the use of ConfigBuilder), since that list is used\nby the resolution code. This duplicates some code in SQLConf, which could potentially\nbe merged with this now. It will also make it simpler to implement some missing\nfeatures such as filtering which configs show up in the UI or in event logs - which\nare not part of this change.\n\nAnother change is in the way ConfigEntry reads config data; it now takes a string\nmap and a function that reads env variables, so that it can be called both from\nSparkConf and SQLConf. This makes it so both places follow the same read path,\ninstead of having to replicate certain logic in SQLConf. There are still a\ncouple of methods in SQLConf that peek into fields of ConfigEntry directly,\nthough.\n\nTested via unit tests, and by using the new variable expansion functionality\nin a shell session with a custom spark.sql.hive.metastore.jars value.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #14022 from vanzin/SPARK-16272.","distinct":true,"url":"https://api.github.com/repos/WeichenXu123/wchen-spark/commits/75a06aa256aa256c112555609a93c1e1dbb1cb4b"},{"sha":"cfa5ae84ed0f48b3b108d0614dbf6fcd79ef5179","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-16644][SQL] Aggregate should not propagate constraints containing aggregate expressions\n\n## What changes were proposed in this pull request?\n\naggregate expressions can only be executed inside `Aggregate`, if we propagate it up with constraints, the parent operator can not execute it and will fail at runtime.\n\n## How was this patch tested?\n\nnew test in SQLQuerySuite\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #14281 from cloud-fan/bug.","distinct":true,"url":"https://api.github.com/repos/WeichenXu123/wchen-spark/commits/cfa5ae84ed0f48b3b108d0614dbf6fcd79ef5179"},{"sha":"1bf13ba3a24aa002bfaa660588490e527c5bf243","author":{"name":"Holden Karau","email":"0dd5affe763abbc7f4de3fb3797050edd8c481d3@us.ibm.com"},"message":"[MINOR][DOCS][STREAMING] Minor docfix schema of csv rather than parquet in comments\n\n## What changes were proposed in this pull request?\n\nFix parquet to csv in a comment to match the input format being read.\n\n## How was this patch tested?\nN/A (doc change only)\n\nAuthor: Holden Karau <holden@us.ibm.com>\n\nCloses #14274 from holdenk/minor-docfix-schema-of-csv-rather-than-parquet.","distinct":true,"url":"https://api.github.com/repos/WeichenXu123/wchen-spark/commits/1bf13ba3a24aa002bfaa660588490e527c5bf243"},{"sha":"864b764eafa57a1418b683ccf6899b01bab28fba","author":{"name":"Sean Owen","email":"4c59f357315334980589bde1049040bf6de6532c@cloudera.com"},"message":"[SPARK-16226][SQL] Weaken JDBC isolation level to avoid locking when writing partitions\n\n## What changes were proposed in this pull request?\n\nSaving partitions to JDBC in transaction can use a weaker transaction isolation level to reduce locking. Use better method to check if transactions are supported.\n\n## How was this patch tested?\n\nExisting Jenkins tests.\n\nAuthor: Sean Owen <sowen@cloudera.com>\n\nCloses #14054 from srowen/SPARK-16226.","distinct":true,"url":"https://api.github.com/repos/WeichenXu123/wchen-spark/commits/864b764eafa57a1418b683ccf6899b01bab28fba"}]}
{"actor":{"display_login":"WeichenXu123"}}