{"action":"created","issue":{"url":"https://api.github.com/repos/pinterest/secor/issues/226","repository_url":"https://api.github.com/repos/pinterest/secor","labels_url":"https://api.github.com/repos/pinterest/secor/issues/226/labels{/name}","comments_url":"https://api.github.com/repos/pinterest/secor/issues/226/comments","events_url":"https://api.github.com/repos/pinterest/secor/issues/226/events","html_url":"https://github.com/pinterest/secor/pull/226","id":162913532,"number":226,"title":"Added support for writing protobuf messages directly to Parquet","user":{"login":"spektom","id":158011,"avatar_url":"https://avatars.githubusercontent.com/u/158011?v=3","gravatar_id":"","url":"https://api.github.com/users/spektom","html_url":"https://github.com/spektom","followers_url":"https://api.github.com/users/spektom/followers","following_url":"https://api.github.com/users/spektom/following{/other_user}","gists_url":"https://api.github.com/users/spektom/gists{/gist_id}","starred_url":"https://api.github.com/users/spektom/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spektom/subscriptions","organizations_url":"https://api.github.com/users/spektom/orgs","repos_url":"https://api.github.com/users/spektom/repos","events_url":"https://api.github.com/users/spektom/events{/privacy}","received_events_url":"https://api.github.com/users/spektom/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-06-29T12:32:54Z","updated_at":"2016-06-29T19:51:43Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/pinterest/secor/pulls/226","html_url":"https://github.com/pinterest/secor/pull/226","diff_url":"https://github.com/pinterest/secor/pull/226.diff","patch_url":"https://github.com/pinterest/secor/pull/226.patch"},"body":""},"comment":{"url":"https://api.github.com/repos/pinterest/secor/issues/comments/229468445","html_url":"https://github.com/pinterest/secor/pull/226#issuecomment-229468445","issue_url":"https://api.github.com/repos/pinterest/secor/issues/226","id":229468445,"user":{"login":"spektom","id":158011,"avatar_url":"https://avatars.githubusercontent.com/u/158011?v=3","gravatar_id":"","url":"https://api.github.com/users/spektom","html_url":"https://github.com/spektom","followers_url":"https://api.github.com/users/spektom/followers","following_url":"https://api.github.com/users/spektom/following{/other_user}","gists_url":"https://api.github.com/users/spektom/gists{/gist_id}","starred_url":"https://api.github.com/users/spektom/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spektom/subscriptions","organizations_url":"https://api.github.com/users/spektom/orgs","repos_url":"https://api.github.com/users/spektom/repos","events_url":"https://api.github.com/users/spektom/events{/privacy}","received_events_url":"https://api.github.com/users/spektom/received_events","type":"User","site_admin":false},"created_at":"2016-06-29T19:51:43Z","updated_at":"2016-06-29T19:51:43Z","body":"1. I thought about decoupling, but there's no much to decouple. If you look at the `ProtobufParquetFileWriter` and `ProtobufParquetFileReader` implementations, they are very difficult to make abstract, since quite every line is implementation (Protobuf) specific. It's easy to create Thrift or Avro implementation using relevant `ParquetWriter` or `ParquetReader` implementations. Anyway, I think premature decoupling is not worth in this case.\r\n\r\n2. Protobuf schema is propagated to the Parquet format using `ProtoWriteSupport` class, where protobuf types are mapped to the relevant Parquet column types. Message schema is determined from the message class, which contains the schema in its static descriptor data fields.\r\n\r\n3. I didn't find a better way than calling static method `parseFrom` of protobuf class for creating message instance out of byte array (if this is what you mean). It's a bit of an overhead, I agree, but it seems like there's no way to avoid reflection.\r\n\r\nI'll be glad to hear your suggestions, if you have any."}}
{"actor":{"display_login":"spektom"}}