{"push_id":2733035616,"size":2,"distinct_size":2,"ref":"refs/heads/master","head":"b3d88ac02940eff4c867d3acb79fe5ff9d724e83","before":"6a9a058e09abb1b629680a546c3d6358b49f723a","commits":[{"sha":"8d707b06003bc97d06630b22e6ae7c35f99b3cdd","author":{"name":"Hieu Huynh","email":"2e92adf85ffa4a9093a828fcda49bc22d6379f22@oath.com”"},"message":"[SPARK-24755][CORE] Executor loss can cause task to not be resubmitted\n\n**Description**\nAs described in [SPARK-24755](https://issues.apache.org/jira/browse/SPARK-24755), when speculation is enabled, there is scenario that executor loss can cause task to not be resubmitted.\nThis patch changes the variable killedByOtherAttempt to keeps track of the taskId of tasks that are killed by other attempt. By doing this, we can still prevent resubmitting task killed by other attempt while resubmit successful attempt when executor lost.\n\n**How was this patch tested?**\nA UT is added based on the UT written by xuanyuanking with modification to simulate the scenario described in SPARK-24755.\n\nAuthor: Hieu Huynh <“Hieu.huynh@oath.com”>\n\nCloses #21729 from hthuynh2/SPARK_24755.","distinct":true,"url":"https://api.github.com/repos/attilapiros/spark/commits/8d707b06003bc97d06630b22e6ae7c35f99b3cdd"},{"sha":"b3d88ac02940eff4c867d3acb79fe5ff9d724e83","author":{"name":"Tathagata Das","email":"4330a85689689e23b81e88bdf5853b25c33529eb@gmail.com"},"message":"[SPARK-22187][SS] Update unsaferow format for saved state in flatMapGroupsWithState to allow timeouts with deleted state\n\n## What changes were proposed in this pull request?\n\nCurrently, the group state of user-defined-type is encoded as top-level columns in the UnsafeRows stores in the state store. The timeout timestamp is also saved as (when needed) as the last top-level column. Since the group state is serialized to top-level columns, you cannot save \"null\" as a value of state (setting null in all the top-level columns is not equivalent). So we don't let the user set the timeout without initializing the state for a key. Based on user experience, this leads to confusion.\n\nThis PR is to change the row format such that the state is saved as nested columns. This would allow the state to be set to null, and avoid these confusing corner cases. However, queries recovering from existing checkpoint will use the previous format to maintain compatibility with existing production queries.\n\n## How was this patch tested?\nRefactored existing end-to-end tests and added new tests for explicitly testing obj-to-row conversion for both state formats.\n\nAuthor: Tathagata Das <tathagata.das1565@gmail.com>\n\nCloses #21739 from tdas/SPARK-22187-1.","distinct":true,"url":"https://api.github.com/repos/attilapiros/spark/commits/b3d88ac02940eff4c867d3acb79fe5ff9d724e83"}]}
{"actor":{"display_login":"attilapiros"}}