{"push_id":2836379945,"size":1,"distinct_size":1,"ref":"refs/heads/master","head":"ec3e9986385880adce1648eae30007eccff862ba","before":"6b1b10ca85f17fc8cd8acc9d6705bd14115ba6b4","commits":[{"sha":"ec3e9986385880adce1648eae30007eccff862ba","author":{"name":"Thomas Graves","email":"a1a405b82800b492a7268cba6bc58270d2b527de@unharmedunarmed.corp.ne1.yahoo.com"},"message":"[SPARK-24909][CORE] Always unregister pending partition on task completion.\n\nSpark scheduler can hang when fetch failures, executor lost, task running on lost executor, and multiple stage attempts. To fix this we change to always unregister the pending partition on task completion.\n\n## What changes were proposed in this pull request?\nthis PR is actually reverting the change in SPARK-19263, so that it always does shuffleStage.pendingPartitions -= task.partitionId.   The change in SPARK-23433, should fix the issue originally from SPARK-19263.\n\n## How was this patch tested?\n\nUnit tests.  The condition happens on a race which I haven't reproduced on a real customer, just see it sometimes on customers jobs in a real cluster.\nI am also working on adding spark scheduler integration tests.\n\nCloses #21976 from tgravescs/SPARK-24909.\n\nAuthored-by: Thomas Graves <tgraves@unharmedunarmed.corp.ne1.yahoo.com>\nSigned-off-by: Marcelo Vanzin <vanzin@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/dongjoon-hyun/spark/commits/ec3e9986385880adce1648eae30007eccff862ba"}]}
{"actor":{"display_login":"dongjoon-hyun"}}