{"action":"created","issue":{"url":"https://api.github.com/repos/scalapb/sparksql-scalapb/issues/10","repository_url":"https://api.github.com/repos/scalapb/sparksql-scalapb","labels_url":"https://api.github.com/repos/scalapb/sparksql-scalapb/issues/10/labels{/name}","comments_url":"https://api.github.com/repos/scalapb/sparksql-scalapb/issues/10/comments","events_url":"https://api.github.com/repos/scalapb/sparksql-scalapb/issues/10/events","html_url":"https://github.com/scalapb/sparksql-scalapb/pull/10","id":296726955,"number":10,"title":"Add parquet list support","user":{"login":"greg-nicolle","id":6479491,"avatar_url":"https://avatars3.githubusercontent.com/u/6479491?v=4","gravatar_id":"","url":"https://api.github.com/users/greg-nicolle","html_url":"https://github.com/greg-nicolle","followers_url":"https://api.github.com/users/greg-nicolle/followers","following_url":"https://api.github.com/users/greg-nicolle/following{/other_user}","gists_url":"https://api.github.com/users/greg-nicolle/gists{/gist_id}","starred_url":"https://api.github.com/users/greg-nicolle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/greg-nicolle/subscriptions","organizations_url":"https://api.github.com/users/greg-nicolle/orgs","repos_url":"https://api.github.com/users/greg-nicolle/repos","events_url":"https://api.github.com/users/greg-nicolle/events{/privacy}","received_events_url":"https://api.github.com/users/greg-nicolle/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-02-13T12:54:49Z","updated_at":"2018-02-13T18:01:04Z","closed_at":null,"author_association":"NONE","pull_request":{"url":"https://api.github.com/repos/scalapb/sparksql-scalapb/pulls/10","html_url":"https://github.com/scalapb/sparksql-scalapb/pull/10","diff_url":"https://github.com/scalapb/sparksql-scalapb/pull/10.diff","patch_url":"https://github.com/scalapb/sparksql-scalapb/pull/10.patch"},"body":"Add reading and writing support to [Hive](https://github.com/apache/hive) or other tools that use [parquet list](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#lists) struture.\r\n\r\nGiven the following protobuf schema:\r\n`\r\nmessage ListOfPrimitives {\r\n    repeated int64 my_repeated_id = 1;\r\n}\r\n`\r\n\r\nOld parquet schema was:\r\n`\r\nmessage ListOfPrimitives {\r\n  repeated int64 my_repeated_id = 1;\r\n}\r\n`\r\n\r\nNew parquet schema is:\r\n`\r\nmessage ListOfPrimitives {\r\n  required group my_repeated_id (LIST) = 1 {\r\n    repeated group list {\r\n      required int64 element;\r\n    }\r\n  }\r\n}\r\n`"},"comment":{"url":"https://api.github.com/repos/scalapb/sparksql-scalapb/issues/comments/365350973","html_url":"https://github.com/scalapb/sparksql-scalapb/pull/10#issuecomment-365350973","issue_url":"https://api.github.com/repos/scalapb/sparksql-scalapb/issues/10","id":365350973,"user":{"login":"thesamet","id":307373,"avatar_url":"https://avatars0.githubusercontent.com/u/307373?v=4","gravatar_id":"","url":"https://api.github.com/users/thesamet","html_url":"https://github.com/thesamet","followers_url":"https://api.github.com/users/thesamet/followers","following_url":"https://api.github.com/users/thesamet/following{/other_user}","gists_url":"https://api.github.com/users/thesamet/gists{/gist_id}","starred_url":"https://api.github.com/users/thesamet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thesamet/subscriptions","organizations_url":"https://api.github.com/users/thesamet/orgs","repos_url":"https://api.github.com/users/thesamet/repos","events_url":"https://api.github.com/users/thesamet/events{/privacy}","received_events_url":"https://api.github.com/users/thesamet/received_events","type":"User","site_admin":false},"created_at":"2018-02-13T18:01:04Z","updated_at":"2018-02-13T18:01:04Z","author_association":"CONTRIBUTOR","body":"@greg-nicolle, thanks for sending this PR. A few questions:\r\n\r\n1. What is the problem that this PR is fixing? Does the current parquet cause any problems with Hive? Can you add details on that to the PR?\r\n2. Is this change breaking compatibility in the sense that Parquet files that were written by an old version would not be readable with the new version? If so, can you make a flag that controls the writing mode?\r\n3. My preference for the generated schema is that the outer list would be nullable, and the elements required:\r\n```message ListOfPrimitives { optional group my_repeated_id (LIST) = 1 { repeated group list { required int64 element; } } }```\r\nThis is the second style example at https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#lists"}}
{"actor":{"display_login":"thesamet"}}