{"push_id":2750758290,"size":5,"distinct_size":5,"ref":"refs/heads/master","head":"2c82745686f4456c4d5c84040a431dcb5b6cb60b","before":"d2e7deb59f641e93778b763d5396f73d38f9a785","commits":[{"sha":"c9b233d4144790c3e57e1a1d1602ad5dc354e8a8","author":{"name":"Takuya UESHIN","email":"f070f8f0c50caeda5841963fc7b57e73e59dad56@databricks.com"},"message":"[SPARK-24878][SQL] Fix reverse function for array type of primitive type containing null.\n\n## What changes were proposed in this pull request?\n\nIf we use `reverse` function for array type of primitive type containing `null` and the child array is `UnsafeArrayData`, the function returns a wrong result because `UnsafeArrayData` doesn't define the behavior of re-assignment, especially we can't set a valid value after we set `null`.\n\n## How was this patch tested?\n\nAdded some tests.\n\nAuthor: Takuya UESHIN <ueshin@databricks.com>\n\nCloses #21830 from ueshin/issues/SPARK-24878/fix_reverse.","distinct":true,"url":"https://api.github.com/repos/lightbend/spark/commits/c9b233d4144790c3e57e1a1d1602ad5dc354e8a8"},{"sha":"58353d7f4baa8102c3d2f4777a5c407f14993306","author":{"name":"Dongjoon Hyun","email":"6ff48b33cbb828151035af58a92f18a39f1d1036@apache.org"},"message":"[SPARK-24924][SQL] Add mapping for built-in Avro data source\n\n## What changes were proposed in this pull request?\n\nThis PR aims to the followings.\n1. Like `com.databricks.spark.csv` mapping, we had better map `com.databricks.spark.avro` to built-in Avro data source.\n2. Remove incorrect error message, `Please find an Avro package at ...`.\n\n## How was this patch tested?\n\nPass the newly added tests.\n\nAuthor: Dongjoon Hyun <dongjoon@apache.org>\n\nCloses #21878 from dongjoon-hyun/SPARK-24924.","distinct":true,"url":"https://api.github.com/repos/lightbend/spark/commits/58353d7f4baa8102c3d2f4777a5c407f14993306"},{"sha":"5ed7660d14022eb65396e28496c06e47c1dbab1d","author":{"name":"maryannxue","email":"a27c3be666bde3a2f598c62e1c810c9c07319551@apache.org"},"message":"[SPARK-24802][SQL][FOLLOW-UP] Add a new config for Optimization Rule Exclusion\n\n## What changes were proposed in this pull request?\n\nThis is an extension to the original PR, in which rule exclusion did not work for classes derived from Optimizer, e.g., SparkOptimizer.\nTo solve this issue, Optimizer and its derived classes will define/override `defaultBatches` and `nonExcludableRules` in order to define its default rule set as well as rules that cannot be excluded by the SQL config. In the meantime, Optimizer's `batches` method is dedicated to the rule exclusion logic and is defined \"final\".\n\n## How was this patch tested?\n\nAdded UT.\n\nAuthor: maryannxue <maryannxue@apache.org>\n\nCloses #21876 from maryannxue/rule-exclusion.","distinct":true,"url":"https://api.github.com/repos/lightbend/spark/commits/5ed7660d14022eb65396e28496c06e47c1dbab1d"},{"sha":"e3486e1b9556e00bc9c392a5b8440ab366780f9b","author":{"name":"Xingbo Jiang","email":"19a442ba0283f6f4c465cc281093aec56203af87@databricks.com"},"message":"[SPARK-24795][CORE] Implement barrier execution mode\n\n## What changes were proposed in this pull request?\n\nPropose new APIs and modify job/task scheduling to support barrier execution mode, which requires all tasks in a same barrier stage start at the same time, and retry all tasks in case some tasks fail in the middle. The barrier execution mode is useful for some ML/DL workloads.\n\nThe proposed API changes include:\n\n- `RDDBarrier` that marks an RDD as barrier (Spark must launch all the tasks together for the current stage).\n- `BarrierTaskContext` that support global sync of all tasks in a barrier stage, and provide extra `BarrierTaskInfo`s.\n\nIn DAGScheduler, we retry all tasks of a barrier stage in case some tasks fail in the middle, this is achieved by unregistering map outputs for a shuffleId (for ShuffleMapStage) or clear the finished partitions in an active job (for ResultStage).\n\n## How was this patch tested?\n\nAdd `RDDBarrierSuite` to ensure we convert RDDs correctly;\nAdd new test cases in `DAGSchedulerSuite` to ensure we do task scheduling correctly;\nAdd new test cases in `SparkContextSuite` to ensure the barrier execution mode actually works (both under local mode and local cluster mode).\nAdd new test cases in `TaskSchedulerImplSuite` to ensure we schedule tasks for barrier taskSet together.\n\nAuthor: Xingbo Jiang <xingbo.jiang@databricks.com>\n\nCloses #21758 from jiangxb1987/barrier-execution-mode.","distinct":true,"url":"https://api.github.com/repos/lightbend/spark/commits/e3486e1b9556e00bc9c392a5b8440ab366780f9b"},{"sha":"2c82745686f4456c4d5c84040a431dcb5b6cb60b","author":{"name":"Imran Rashid","email":"b54fbb0e17aa598fdc04bb3d79ba76b5f0737666@cloudera.com"},"message":"[SPARK-24307][CORE] Add conf to revert to old code.\n\nIn case there are any issues in converting FileSegmentManagedBuffer to\nChunkedByteBuffer, add a conf to go back to old code path.\n\nFollowup to 7e847646d1f377f46dc3154dea37148d4e557a03\n\nAuthor: Imran Rashid <irashid@cloudera.com>\n\nCloses #21867 from squito/SPARK-24307-p2.","distinct":true,"url":"https://api.github.com/repos/lightbend/spark/commits/2c82745686f4456c4d5c84040a431dcb5b6cb60b"}]}
{"actor":{"display_login":"fdp-ci"}}