{"push_id":1394708558,"size":1,"distinct_size":1,"ref":"refs/heads/master","head":"3f62e1b5d9e75dc07bac3aa4db3e8d0615cc3cc3","before":"64fbdf1aa90b66269daec29f62dc9431c1173bab","commits":[{"sha":"3f62e1b5d9e75dc07bac3aa4db3e8d0615cc3cc3","author":{"name":"Tyson Condie","email":"99eb1d353d139b48ee673525980bd0d3f965b4fd@gmail.com"},"message":"[SPARK-17829][SQL] Stable format for offset log\n\n## What changes were proposed in this pull request?\n\nCurrently we use java serialization for the WAL that stores the offsets contained in each batch. This has two main issues:\nIt can break across spark releases (though this is not the only thing preventing us from upgrading a running query)\nIt is unnecessarily opaque to the user.\nI'd propose we require offsets to provide a user readable serialization and use that instead. JSON is probably a good option.\n## How was this patch tested?\n\nTests were added for KafkaSourceOffset in [KafkaSourceOffsetSuite](external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaSourceOffsetSuite.scala) and for LongOffset in [OffsetSuite](sql/core/src/test/scala/org/apache/spark/sql/streaming/OffsetSuite.scala)\n\nPlease review https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark before opening a pull request.\n\nzsxwing marmbrus\n\nAuthor: Tyson Condie <tcondie@gmail.com>\nAuthor: Tyson Condie <tcondie@clash.local>\n\nCloses #15626 from tcondie/spark-8360.","distinct":true,"url":"https://api.github.com/repos/dbtsai/spark/commits/3f62e1b5d9e75dc07bac3aa4db3e8d0615cc3cc3"}]}
{"actor":{"display_login":"dbtsai-bot"}}