{"push_id":1072775361,"size":1,"distinct_size":1,"ref":"refs/heads/KUDU-1377-8f","head":"0a6fac33f6fde93a5721efa3a2673947bd569fd4","before":"ef8ae58c47fd0498177557b4780fcb34e716a6cc","commits":[{"sha":"0a6fac33f6fde93a5721efa3a2673947bd569fd4","author":{"email":"e07a1fd797b473a9c849e618aa2ce06d8055760b@apache.org","name":"Mike Percy"},"message":"KUDU-1377 (part 2): Add version 2 of protobuf container file\n\nThe protobuf container file format V1 was missing some features that\nmade particular corruption cases impossible to differentiate:\n\n1. Since the length did not have its own checksum, a corrupted length\n   field (huge number) can look like a truncated file.\n2. The file header itself, including the version field, did not have a\n   checksum. This may compromise version-specific logic.\n\nVersion 2 of the protobuf container file format adds these fields to the\nfile format. While the default version of protobuf container files has\nbeen changed to version 2 with this patch, support for writing the\noriginal V1 format has not been dropped. The API for creating new V1\nformat files is contrained only to tests, but an existing V1 format file\nmay be opened, read, and appended to.\n\nChanges in this patch:\n\n* Document and implement the version 2 file format.\n* Return Status::Incomplete for partial record read at end of file\n  (for V2 format only).\n  * This allows the protobuf container file API to express the\n    difference between clean EOF and partial-record EOF.\n* Rearrange code necessary for reading the file header and version in\n  order to share it between the Readable* and Writable* classes.\n* Change from WritableFile to RWFile in the pb writer to allow reading\n  the version information from the file header before appending\n  additional records to the file.\n* Add Reopen() method to the writer so it can detect its format version.\n  Callers that wish to append to an existing PB container file must now\n  call Reopen() first. Callers creating new files still call Init().\n* Rename the reader Init() method to Open() since Init() in the writer\n  is for creating new files and this makes the classes more symmetric.\n* Parameterize the pb_util unit tests to test writing and reading both\n  version 1 and version 2 of the file format.\n* Add tests for partial record truncation and continuing to append when\n  using the PB container format in an append-only, log-style use case.\n\nChange-Id: I239c7b99a55a74a6a658ff418830c1f9be13eaa6","distinct":true,"url":"https://api.github.com/repos/mpercy/kudu/commits/0a6fac33f6fde93a5721efa3a2673947bd569fd4"}]}