{"push_id":1145698302,"size":96,"distinct_size":96,"ref":"refs/heads/master","head":"00ad4f054cd044e17d29b7c2c62efd8616462619","before":"2d34183b273af1125181f04c49725efc2fa351af","commits":[{"sha":"5b21139dbf3bd09cb3a590bd0ffb857ea92dc23c","author":{"name":"Devaraj K","email":"b8ebf05c6cb8ebae39c552537bfd7d84ebc26f96@apache.org"},"message":"[SPARK-10530][CORE] Kill other task attempts when one taskattempt belonging the same task is succeeded in speculation\n\n## What changes were proposed in this pull request?\n\nWith this patch, TaskSetManager kills other running attempts when any one of the attempt succeeds for the same task. Also killed tasks will not be considered as failed tasks and they get listed separately in the UI and also shows the task state as KILLED instead of FAILED.\n\n## How was this patch tested?\n\ncore\\src\\test\\scala\\org\\apache\\spark\\ui\\jobs\\JobProgressListenerSuite.scala\ncore\\src\\test\\scala\\org\\apache\\spark\\util\\JsonProtocolSuite.scala\n\nI have verified this patch manually by enabling spark.speculation as true, when any attempt gets succeeded then other running attempts are getting killed for the same task and other pending tasks are getting assigned in those. And also when any attempt gets killed then they are considered as KILLED tasks and not considered as FAILED tasks. Please find the attached screen shots for the reference.\n\n![stage-tasks-table](https://cloud.githubusercontent.com/assets/3174804/14075132/394c6a12-f4f4-11e5-8638-20ff7b8cc9bc.png)\n![stages-table](https://cloud.githubusercontent.com/assets/3174804/14075134/3b60f412-f4f4-11e5-9ea6-dd0dcc86eb03.png)\n\nRef : https://github.com/apache/spark/pull/11916\n\nAuthor: Devaraj K <devaraj@apache.org>\n\nCloses #11996 from devaraj-kavali/SPARK-10530.","distinct":true,"url":"https://api.github.com/repos/javadba/spark/commits/5b21139dbf3bd09cb3a590bd0ffb857ea92dc23c"},{"sha":"675921040ee4802aa9914457de62af746bc3657d","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-15638][SQL] Audit Dataset, SparkSession, and SQLContext\n\n## What changes were proposed in this pull request?\nThis patch contains a list of changes as a result of my auditing Dataset, SparkSession, and SQLContext. The patch audits the categorization of experimental APIs, function groups, and deprecations. For the detailed list of changes, please see the diff.\n\n## How was this patch tested?\nN/A\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #13370 from rxin/SPARK-15638.","distinct":true,"url":"https://api.github.com/repos/javadba/spark/commits/675921040ee4802aa9914457de62af746bc3657d"},{"sha":"6878f3e2eafe0bd2e19a14ddffe6e0c1e545a42d","author":{"name":"catapan","email":"3fadd1ec13d6811a1e2cdfd2ee5dda2ecd627b15@gmail.com"},"message":"[SPARK-15641] HistoryServer to not show invalid date for incomplete application\n\n## What changes were proposed in this pull request?\nFor incomplete applications in HistoryServer, the complete column will show \"-\" instead of incorrect date.\n\n## How was this patch tested?\nmanually tested.\n\nAuthor: catapan <cedarpan86@gmail.com>\nAuthor: Ziying Pan <cedarpan@Ziyings-MacBook.local>\n\nCloses #13396 from catapan/SPARK-15641_fix_completed_column.","distinct":true,"url":"https://api.github.com/repos/javadba/spark/commits/6878f3e2eafe0bd2e19a14ddffe6e0c1e545a42d"},{"sha":"95db8a44f3e2d79913cbe0d29297796b4c3b0d1b","author":{"name":"Takeshi YAMAMURO","email":"c77105bb36ce8e17a5fe8a8e3953cfa3aa730380@gmail.com"},"message":"[SPARK-15528][SQL] Fix race condition in NumberConverter\n\n## What changes were proposed in this pull request?\nA local variable in NumberConverter is wrongly shared between threads.\nThis pr fixes the race condition.\n\n## How was this patch tested?\nManually checked.\n\nAuthor: Takeshi YAMAMURO <linguin.m.s@gmail.com>\n\nCloses #13391 from maropu/SPARK-15528.","distinct":true,"url":"https://api.github.com/repos/javadba/spark/commits/95db8a44f3e2d79913cbe0d29297796b4c3b0d1b"},{"sha":"2bfc4f15214a870b3e067f06f37eb506b0070a1f","author":{"name":"Lianhui Wang","email":"909921094b5feb344e808fcae78e541057760afe@gmail.com"},"message":"[SPARK-15649][SQL] Avoid to serialize MetastoreRelation in HiveTableScanExec\n\n## What changes were proposed in this pull request?\nin HiveTableScanExec, schema is lazy and is related with relation.attributeMap. So it needs to serialize MetastoreRelation when serializing task binary bytes.It can avoid to serialize MetastoreRelation.\n\n## How was this patch tested?\n\nAuthor: Lianhui Wang <lianhuiwang09@gmail.com>\n\nCloses #13397 from lianhuiwang/avoid-serialize.","distinct":true,"url":"https://api.github.com/repos/javadba/spark/commits/2bfc4f15214a870b3e067f06f37eb506b0070a1f"},{"sha":"d67c82e4b647dacd0b789d72c9eaf4dc7d329dbd","author":{"name":"gatorsmile","email":"ea6589371ad7a107e4268706d7d8403e3ba585bb@gmail.com"},"message":"[SPARK-15647][SQL] Fix Boundary Cases in OptimizeCodegen Rule\n\n#### What changes were proposed in this pull request?\n\nThe following condition in the Optimizer rule `OptimizeCodegen` is not right.\n```Scala\nbranches.size < conf.maxCaseBranchesForCodegen\n```\n\n- The number of branches in case when clause should be `branches.size + elseBranch.size`.\n- `maxCaseBranchesForCodegen` is the maximum boundary for enabling codegen. Thus, we should use `<=` instead of `<`.\n\nThis PR is to fix this boundary case and also add missing test cases for verifying the conf `MAX_CASES_BRANCHES`.\n\n#### How was this patch tested?\nAdded test cases in `SQLConfSuite`\n\nAuthor: gatorsmile <gatorsmile@gmail.com>\n\nCloses #13392 from gatorsmile/maxCaseWhen.","distinct":true,"url":"https://api.github.com/repos/javadba/spark/commits/d67c82e4b647dacd0b789d72c9eaf4dc7d329dbd"},{"sha":"2bfed1a0c5be7d0718fd574a4dad90f4f6b44be7","author":{"name":"Wenchen Fan","email":"51df79715bab6f96b68cf2b8813ef1aaa5547bdb@databricks.com"},"message":"[SPARK-15658][SQL] UDT serializer should declare its data type as udt instead of udt.sqlType\n\n## What changes were proposed in this pull request?\n\nWhen we build serializer for UDT object, we should declare its data type as udt instead of udt.sqlType, or if we deserialize it again, we lose the information that it's a udt object and throw analysis exception.\n\n## How was this patch tested?\n\nnew test in `UserDefiendTypeSuite`\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #13402 from cloud-fan/udt.","distinct":true,"url":"https://api.github.com/repos/javadba/spark/commits/2bfed1a0c5be7d0718fd574a4dad90f4f6b44be7"},{"sha":"c6de5832bfad423e7d6f7e0a92a48170265f25cd","author":{"name":"Yin Huai","email":"7e8362bd012819bee68b9375d9cefa24afbef722@databricks.com"},"message":"[SPARK-15622][SQL] Wrap the parent classloader of Janino's classloader in the ParentClassLoader.\n\n## What changes were proposed in this pull request?\nAt https://github.com/aunkrig/janino/blob/janino_2.7.8/janino/src/org/codehaus/janino/ClassLoaderIClassLoader.java#L80-L85, Janino's classloader throws the exception when its parent throws a ClassNotFoundException with a cause set. However, it does not throw the exception when there is no cause set. Seems we need to use a special ClassLoader to wrap the actual parent classloader set to Janino handle this behavior.\n\n## How was this patch tested?\nI have reverted the workaround made by https://issues.apache.org/jira/browse/SPARK-11636 ( https://github.com/apache/spark/compare/master...yhuai:SPARK-15622?expand=1#diff-bb538fda94224dd0af01d0fd7e1b4ea0R81) and `test-only *ReplSuite -- -z \"SPARK-2576 importing implicits\"` still passes the test (without the change in `CodeGenerator`, this test does not pass with the change in `ExecutorClassLoader `).\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #13366 from yhuai/SPARK-15622.","distinct":true,"url":"https://api.github.com/repos/javadba/spark/commits/c6de5832bfad423e7d6f7e0a92a48170265f25cd"},{"sha":"9a74de18a13d84805e1a448f858bb05ce30de87e","author":{"name":"Shixiong Zhu","email":"eb2f6da57ec54996cebae8c2a82b9fbfd527e14c@databricks.com"},"message":"Revert \"[SPARK-11753][SQL][TEST-HADOOP2.2] Make allowNonNumericNumbers option work\n\n## What changes were proposed in this pull request?\n\nThis reverts commit c24b6b679c3efa053f7de19be73eb36dc70d9930. Sent a PR to run Jenkins tests due to the revert conflicts of `dev/deps/spark-deps-hadoop*`.\n\n## How was this patch tested?\n\nJenkins unit tests, integration tests, manual tests)\n\nAuthor: Shixiong Zhu <shixiong@databricks.com>\n\nCloses #13417 from zsxwing/revert-SPARK-11753.","distinct":true,"url":"https://api.github.com/repos/javadba/spark/commits/9a74de18a13d84805e1a448f858bb05ce30de87e"},{"sha":"594484cd8343f870c53fbc829ed4fb889016a8cf","author":{"name":"Yanbo Liang","email":"b6929798b4e4fba75014af8265904a7818902ec7@gmail.com"},"message":"[MINOR][DOC][ML] ml.clustering scala & python api doc sync\n\n## What changes were proposed in this pull request?\nSince we done Scala API audit for ml.clustering at #13148, we should also fix and update the corresponding Python API docs to keep them in sync.\n\n## How was this patch tested?\nDocs change, no tests.\n\nAuthor: Yanbo Liang <ybliang8@gmail.com>\n\nCloses #13291 from yanboliang/spark-15361-followup.","distinct":true,"url":"https://api.github.com/repos/javadba/spark/commits/594484cd8343f870c53fbc829ed4fb889016a8cf"},{"sha":"2df6ca848e99b90acd11c3a3de342fa4d77015d6","author":{"name":"Davies Liu","email":"589979bd19a058b75730100a5a88539708000f52@databricks.com"},"message":"[SPARK-15327] [SQL] fix split expression in whole stage codegen\n\n## What changes were proposed in this pull request?\n\nRight now, we will split the code for expressions into multiple functions when it exceed 64k, which requires that the the expressions are using Row object, but this is not true for whole-state codegen, it will fail to compile after splitted.\n\nThis PR will not split the code in whole-stage codegen.\n\n## How was this patch tested?\n\nAdded regression tests.\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #13235 from davies/fix_nested_codegen.","distinct":true,"url":"https://api.github.com/repos/javadba/spark/commits/2df6ca848e99b90acd11c3a3de342fa4d77015d6"},{"sha":"dfe2cbeb437a4fa69bec3eca4ac9242f3eb51c81","author":{"name":"Dilip Biswal","email":"6ab79ed79ae7ed4bf1dfe07d1c3e4c433e1f76a0@us.ibm.com"},"message":"[SPARK-15557] [SQL] cast the string into DoubleType when it's used together with decimal\n\nIn this case, the result type of the expression becomes DECIMAL(38, 36) as we promote the individual string literals to DECIMAL(38, 18) when we handle string promotions for `BinaryArthmaticExpression`.\n\nI think we need to cast the string literals to Double type instead. I looked at the history and found that  this was changed to use decimal instead of double to avoid potential loss of precision when we cast decimal to double.\n\nTo double check i ran the query against hive, mysql. This query returns non NULL result for both the databases and both promote the expression to use double.\nHere is the output.\n\n- Hive\n```SQL\nhive> create table l2 as select (cast(99 as decimal(19,6)) + '2') from l1;\nOK\nhive> describe l2;\nOK\n_c0                 \tdouble\n```\n- MySQL\n```SQL\nmysql> create table foo2 as select (cast(99 as decimal(19,6)) + '2') from test;\nQuery OK, 1 row affected (0.01 sec)\nRecords: 1  Duplicates: 0  Warnings: 0\n\nmysql> describe foo2;\n+-----------------------------------+--------+------+-----+---------+-------+\n| Field                             | Type   | Null | Key | Default | Extra |\n+-----------------------------------+--------+------+-----+---------+-------+\n| (cast(99 as decimal(19,6)) + '2') | double | NO   |     | 0       |       |\n+-----------------------------------+--------+------+-----+---------+-------+\n```\n\n## How was this patch tested?\nAdded a new test in SQLQuerySuite\n\nAuthor: Dilip Biswal <dbiswal@us.ibm.com>\n\nCloses #13368 from dilipbiswal/spark-15557.","distinct":true,"url":"https://api.github.com/repos/javadba/spark/commits/dfe2cbeb437a4fa69bec3eca4ac9242f3eb51c81"},{"sha":"90b11439b3d4540f48985e87dcc99749f0369287","author":{"name":"Tathagata Das","email":"58b3b6773dfe925407947ae07f6b206d48277ab4@gmail.com"},"message":"[SPARK-15517][SQL][STREAMING] Add support for complete output mode in Structure Streaming\n\n## What changes were proposed in this pull request?\nCurrently structured streaming only supports append output mode.  This PR adds the following.\n\n- Added support for Complete output mode in the internal state store, analyzer and planner.\n- Added public API in Scala and Python for users to specify output mode\n- Added checks for unsupported combinations of output mode and DF operations\n  - Plans with no aggregation should support only Append mode\n  - Plans with aggregation should support only Update and Complete modes\n  - Default output mode is Append mode (**Question: should we change this to automatically set to Complete mode when there is aggregation?**)\n- Added support for Complete output mode in Memory Sink. So Memory Sink internally supports append and complete, update. But from public API only Complete and Append output modes are supported.\n\n## How was this patch tested?\nUnit tests in various test suites\n- StreamingAggregationSuite: tests for complete mode\n- MemorySinkSuite: tests for checking behavior in Append and Complete modes.\n- UnsupportedOperationSuite: tests for checking unsupported combinations of DF ops and output modes\n- DataFrameReaderWriterSuite: tests for checking that output mode cannot be called on static DFs\n- Python doc test and existing unit tests modified to call write.outputMode.\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #13286 from tdas/complete-mode.","distinct":true,"url":"https://api.github.com/repos/javadba/spark/commits/90b11439b3d4540f48985e87dcc99749f0369287"},{"sha":"57adb77e6b0208dc6d29488a674b6ad78702262f","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"[SPARK-15451][BUILD] Use jdk7's rt.jar when available.\n\nThis helps with preventing jdk8-specific calls being checked in,\nbecause PR builders are running the compiler with the wrong settings.\n\nIf the JAVA_7_HOME env variable is set, assume it points at\na jdk7 and use its rt.jar when invoking javac. For zinc, just run\nit with jdk7, and disable it when building jdk8-specific code.\n\nA big note for sbt usage: adding the bootstrap options forces sbt\nto fork the compiler, and that disables incremental compilation.\nThat means that it's really not convenient to use for normal\ndevelopment, but should be ok for automated builds.\n\nTested with JAVA_HOME=jdk8 and JAVA_7_HOME=jdk7:\n- mvn + zinc\n- mvn sans zinc\n- sbt\n\nVerified that in all cases, jdk8-specific library calls fail to\ncompile.\n\nAuthor: Marcelo Vanzin <vanzin@cloudera.com>\n\nCloses #13272 from vanzin/SPARK-15451.","distinct":true,"url":"https://api.github.com/repos/javadba/spark/commits/57adb77e6b0208dc6d29488a674b6ad78702262f"},{"sha":"6954704299386cdcfdbc3f7fa6967c0880c9b7b8","author":{"name":"Jacek Laskowski","email":"2ae8b6a7c53eb98fbee4a4146dda0b032e92cea4@japila.pl"},"message":"[CORE][MINOR][DOC] Removing incorrect scaladoc\n\n## What changes were proposed in this pull request?\n\nI don't think the method will ever throw an exception so removing a false comment. Sorry srowen and rxin again -- I simply couldn't resist.\n\nI wholeheartedly support merging the change with a bigger one (and trashing this PR).\n\n## How was this patch tested?\n\nManual build\n\nAuthor: Jacek Laskowski <jacek@japila.pl>\n\nCloses #13384 from jaceklaskowski/blockinfomanager.","distinct":true,"url":"https://api.github.com/repos/javadba/spark/commits/6954704299386cdcfdbc3f7fa6967c0880c9b7b8"},{"sha":"223f1d58c4f4b6eb0f0037a118a0bb635ae20bb1","author":{"name":"Reynold Xin","email":"9a371236ace08d82aa86864cade97762c12d4503@databricks.com"},"message":"[SPARK-15662][SQL] Add since annotation for classes in sql.catalog\n\n## What changes were proposed in this pull request?\nThis patch does a few things:\n\n1. Adds since version annotation to methods and classes in sql.catalog.\n2. Fixed a typo in FilterFunction and a whitespace issue in spark/api/java/function/package.scala\n3. Added \"database\" field to Function class.\n\n## How was this patch tested?\nUpdated unit test case for \"database\" field in Function class.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #13406 from rxin/SPARK-15662.","distinct":true,"url":"https://api.github.com/repos/javadba/spark/commits/223f1d58c4f4b6eb0f0037a118a0bb635ae20bb1"},{"sha":"8ca01a6feb4935b1a3815cfbff1b90ccc6f60984","author":{"name":"Josh Rosen","email":"1fa93d1db1fb7c1c51f2467ef55c960ec1b64919@databricks.com"},"message":"[SPARK-15680][SQL] Disable comments in generated code in order to avoid perf. issues\n\n## What changes were proposed in this pull request?\n\nIn benchmarks involving tables with very wide and complex schemas (thousands of columns, deep nesting), I noticed that significant amounts of time (order of tens of seconds per task) were being spent generating comments during the code generation phase.\n\nThe root cause of the performance problem stems from the fact that calling toString() on a complex expression can involve thousands of string concatenations, resulting in huge amounts (tens of gigabytes) of character array allocation and copying.\n\nIn the long term, we can avoid this problem by passing StringBuilders down the tree and using them to accumulate output. As a short-term workaround, this patch guards comment generation behind a flag and disables comments by default (for wide tables / complex queries, these comments were being truncated prior to display and thus were not very useful).\n\n## How was this patch tested?\n\nThis was tested manually by running a Spark SQL query over an empty table with a very wide schema obtained from a real workload. Disabling comments brought the per-task time down from about 16 seconds to 600 milliseconds.\n\nAuthor: Josh Rosen <joshrosen@databricks.com>\n\nCloses #13421 from JoshRosen/disable-line-comments-in-codegen.","distinct":true,"url":"https://api.github.com/repos/javadba/spark/commits/8ca01a6feb4935b1a3815cfbff1b90ccc6f60984"},{"sha":"0f24713468088fa7617d208572179d558e1f286b","author":{"name":"Jacek Laskowski","email":"2ae8b6a7c53eb98fbee4a4146dda0b032e92cea4@japila.pl"},"message":"[CORE][DOC][MINOR] typos + links\n\n## What changes were proposed in this pull request?\n\nA very tiny change to javadoc (which I don't mind if gets merged with a bigger change). I've just found it annoying and couldn't resist proposing a pull request. Sorry srowen and rxin.\n\n## How was this patch tested?\n\nManual build\n\nAuthor: Jacek Laskowski <jacek@japila.pl>\n\nCloses #13383 from jaceklaskowski/memory-consumer.","distinct":true,"url":"https://api.github.com/repos/javadba/spark/commits/0f24713468088fa7617d208572179d558e1f286b"},{"sha":"06514d689cb346e2bcfe3f37a037049d4410befa","author":{"name":"Sean Zhong","email":"5320fbfa67c9b6c0e01431b7f024db2a7ff8bdaf@databricks.com"},"message":"[SPARK-12988][SQL] Can't drop top level columns that contain dots\n\n## What changes were proposed in this pull request?\n\nFixes \"Can't drop top level columns that contain dots\".\n\nThis work is based on dilipbiswal's https://github.com/apache/spark/pull/10943.\nThis PR fixes problems like:\n\n```\nscala> Seq((1, 2)).toDF(\"a.b\", \"a.c\").drop(\"a.b\")\norg.apache.spark.sql.AnalysisException: cannot resolve '`a.c`' given input columns: [a.b, a.c];\n```\n\n`drop(columnName)` can only be used to drop top level column, so, we should parse the column name literally WITHOUT interpreting dot \".\"\n\nWe should also NOT interpret back tick \"`\", otherwise it is hard to understand what\n\n```\n​```aaa```bbb``\n```\n\nactually means.\n\n## How was this patch tested?\n\nUnit tests.\n\nAuthor: Sean Zhong <seanzhong@databricks.com>\n\nCloses #13306 from clockfly/fix_drop_column.","distinct":true,"url":"https://api.github.com/repos/javadba/spark/commits/06514d689cb346e2bcfe3f37a037049d4410befa"},{"sha":"dad5a68818436eb7feaeb762b72433248eff298f","author":{"name":"WeichenXu","email":"7a440c883b23bc43b7a6fdb423cbfd8e75175a83@outlook.com"},"message":"[SPARK-15670][JAVA API][SPARK CORE] label_accumulator_deprecate_in_java_spark_context\n\n## What changes were proposed in this pull request?\n\nAdd deprecate annotation for acumulator V1 interface in JavaSparkContext class\n\n## How was this patch tested?\n\nN/A\n\nAuthor: WeichenXu <WeichenXu123@outlook.com>\n\nCloses #13412 from WeichenXu123/label_accumulator_deprecate_in_java_spark_context.","distinct":true,"url":"https://api.github.com/repos/javadba/spark/commits/dad5a68818436eb7feaeb762b72433248eff298f"}]}
{"actor":{"display_login":"javadba"}}