{"push_id":934341804,"size":2,"distinct_size":2,"ref":"refs/heads/task-metrics-to-accums","head":"596348f048584db7a5f8a90effec704258755c0e","before":"1323574e2007dddc4d576da460a4f7704fa1aa73","commits":[{"sha":"63a36608ea9b9c710dd78752600d8f05076934fd","author":{"email":"02e0a999c50b1f88df7a8f5a04e1b76b35ea6a88@databricks.com","name":"andrewor14"},"message":"Merge github.com:apache/spark into task-metrics-to-accums","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/63a36608ea9b9c710dd78752600d8f05076934fd"},{"sha":"596348f048584db7a5f8a90effec704258755c0e","author":{"email":"a5c478d606458f6c868a5126471748acd4730a6f@gmail.com","name":"andrewor14"},"message":"Fix HistoryServerSuite\n\nThe problem was that the executor run time from a failed task was\nnot included in the one aggregated across the stage. This is\nbecause the old test log did not have accumulator updates, but\nonly TaskMetrics, in the ExceptionFailure class.\n\nIt would seem that an easy fix would be to fallback to parsing\nTaskMetrics and get accumulator updates from there. However, this\nis insufficient because we could not reconstruct TaskMetrics\nwithout having previously registered them on the driver.\n\nThis commit adds the functionality to reconstruct TaskMetrics\nwithout any prior state. This implicitly requires Spark to be\nable to reconstruct the Accumulator object from AccumulableInfo.\nThis was difficult because this would mean that we need to\nkeep track of the accumulator value type, which was not previously\nencoded anywhere.\n\nAfter this commit, we now use the correct type to de/serialize\nthe values of AccumulableInfo's that represent the internal\naccumulators used by TaskMetrics. The two main benefits are:\n\n- ExceptionFailure no longer breaks compatibility in event logs\n- AccumulableInfo is type-safe in event logs for internal accums","distinct":true,"url":"https://api.github.com/repos/andrewor14/spark/commits/596348f048584db7a5f8a90effec704258755c0e"}]}