{"action":"created","issue":{"url":"https://api.github.com/repos/apache/arrow/issues/1500","repository_url":"https://api.github.com/repos/apache/arrow","labels_url":"https://api.github.com/repos/apache/arrow/issues/1500/labels{/name}","comments_url":"https://api.github.com/repos/apache/arrow/issues/1500/comments","events_url":"https://api.github.com/repos/apache/arrow/issues/1500/events","html_url":"https://github.com/apache/arrow/pull/1500","id":291347315,"number":1500,"title":"ARROW-1860: [C++] Introduce ipc::PreparedMessage data structure to avoid making multiple passes over record batches","user":{"login":"wesm","id":329591,"avatar_url":"https://avatars2.githubusercontent.com/u/329591?v=4","gravatar_id":"","url":"https://api.github.com/users/wesm","html_url":"https://github.com/wesm","followers_url":"https://api.github.com/users/wesm/followers","following_url":"https://api.github.com/users/wesm/following{/other_user}","gists_url":"https://api.github.com/users/wesm/gists{/gist_id}","starred_url":"https://api.github.com/users/wesm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wesm/subscriptions","organizations_url":"https://api.github.com/users/wesm/orgs","repos_url":"https://api.github.com/users/wesm/repos","events_url":"https://api.github.com/users/wesm/events{/privacy}","received_events_url":"https://api.github.com/users/wesm/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-01-24T20:23:46Z","updated_at":"2018-05-21T22:08:33Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/apache/arrow/pulls/1500","html_url":"https://github.com/apache/arrow/pull/1500","diff_url":"https://github.com/apache/arrow/pull/1500.diff","patch_url":"https://github.com/apache/arrow/pull/1500.patch"},"body":"The purpose of this is to decompose a record batch into its serialized Flatbuffer metadata and sequence of buffers that form its body so that the total output size can be computed, e.g. for writing to a shared memory segment. Prior to this, one would have to call `GetRecordBatchSize`, allocate memory, then `WriteRecordBatch` which duplicates work. \r\n\r\nThis also introduces a change to how padding is handled in unaligned streams to make the message contents deterministic. This makes record batches consistent with the way that tensors were already being written, with alignment bytes being written first to move the stream position to a multiple of 8, then beginning to write the metadata and message body."},"comment":{"url":"https://api.github.com/repos/apache/arrow/issues/comments/390798036","html_url":"https://github.com/apache/arrow/pull/1500#issuecomment-390798036","issue_url":"https://api.github.com/repos/apache/arrow/issues/1500","id":390798036,"user":{"login":"wesm","id":329591,"avatar_url":"https://avatars2.githubusercontent.com/u/329591?v=4","gravatar_id":"","url":"https://api.github.com/users/wesm","html_url":"https://github.com/wesm","followers_url":"https://api.github.com/users/wesm/followers","following_url":"https://api.github.com/users/wesm/following{/other_user}","gists_url":"https://api.github.com/users/wesm/gists{/gist_id}","starred_url":"https://api.github.com/users/wesm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wesm/subscriptions","organizations_url":"https://api.github.com/users/wesm/orgs","repos_url":"https://api.github.com/users/wesm/repos","events_url":"https://api.github.com/users/wesm/events{/privacy}","received_events_url":"https://api.github.com/users/wesm/received_events","type":"User","site_admin":false},"created_at":"2018-05-21T22:08:33Z","updated_at":"2018-05-21T22:08:33Z","author_association":"MEMBER","body":"> why do you want to pad data? \r\n\r\nI'll need to revisit this patch from a few months ago and remember why I thought this. The basic idea is for messages to be relocatable and deterministic (e.g. if you were to hash the metadata, you would always get the same hash). By always aligning the stream before writing a message, you guarantee that the message metadata and body have the exact same contents.\r\n\r\nGiven the time that has passed, it'll probably make sense to start this patch over. There's a unit test here that can be taken over, but the rest of the refactoring should probably be started from from the codebase as it is now. "}}
{"actor":{"display_login":"wesm"}}