{"action":"created","comment":{"url":"https://api.github.com/repos/yandex/ClickHouse/pulls/comments/191072007","pull_request_review_id":123579251,"id":191072007,"diff_hunk":"@@ -0,0 +1,52 @@\n+#pragma once\n+\n+#include <DataStreams/IBlockOutputStream.h>\n+#include <DataTypes/DataTypesNumber.h>\n+\n+namespace DB\n+{\n+\n+\n+class ParquetBlockOutputStream : public IBlockOutputStream\n+{\n+public:\n+    ParquetBlockOutputStream(WriteBuffer & ostr_, const Block & header_);\n+\n+    Block getHeader() const override { return header; }\n+    void write(const Block & block) override;\n+    void flush() override;\n+\n+    String getContentType() const override { return \"application/octet-stream\"; }\n+\n+private:\n+    WriteBuffer & ostr;\n+    Block header;\n+\n+    template <typename NumericType, typename ArrowBuilderType>\n+    void fillArrowArrayWithNumericColumnData(ColumnPtr write_column, std::shared_ptr<arrow::Array> & arrow_array);\n+\n+    void fillArrowArrayWithDateColumnData(ColumnPtr write_column, std::shared_ptr<arrow::Array> & arrow_array);\n+    void fillArrowArrayWithStringColumnData(ColumnPtr write_column, std::shared_ptr<arrow::Array> & arrow_array);\n+\n+    std::unordered_map<String, std::shared_ptr<arrow::DataType>> internal_type_to_arrow_type = {\n+        {\"UInt8\",   arrow::uint8()},\n+        {\"Int8\",    arrow::int8()},\n+        {\"UInt16\",  arrow::uint16()},\n+        {\"Int16\",   arrow::int16()},\n+        {\"UInt32\",  arrow::uint32()},\n+        {\"Int32\",   arrow::int32()},\n+        {\"UInt64\",  arrow::uint64()},\n+        {\"Int64\",   arrow::int64()},\n+        {\"Float32\", arrow::float32()},\n+        {\"Float64\", arrow::float64()},\n+\n+        {\"Date\",  arrow::date32()},\n+\n+        {\"String\", arrow::utf8()}//,","path":"dbms/src/DataStreams/ParquetBlockOutputStream.h","position":45,"original_position":45,"commit_id":"882e246e4f9fdf9fe73293afc731427531b6e5da","original_commit_id":"882e246e4f9fdf9fe73293afc731427531b6e5da","user":{"login":"ludv1x","id":1247636,"avatar_url":"https://avatars3.githubusercontent.com/u/1247636?v=4","gravatar_id":"","url":"https://api.github.com/users/ludv1x","html_url":"https://github.com/ludv1x","followers_url":"https://api.github.com/users/ludv1x/followers","following_url":"https://api.github.com/users/ludv1x/following{/other_user}","gists_url":"https://api.github.com/users/ludv1x/gists{/gist_id}","starred_url":"https://api.github.com/users/ludv1x/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ludv1x/subscriptions","organizations_url":"https://api.github.com/users/ludv1x/orgs","repos_url":"https://api.github.com/users/ludv1x/repos","events_url":"https://api.github.com/users/ludv1x/events{/privacy}","received_events_url":"https://api.github.com/users/ludv1x/received_events","type":"User","site_admin":false},"body":"I am not sure about it. ClickHouse String could store strings that are not valid utf-8 sequences.","created_at":"2018-05-27T10:30:49Z","updated_at":"2018-05-27T10:30:49Z","html_url":"https://github.com/yandex/ClickHouse/pull/2427#discussion_r191072007","pull_request_url":"https://api.github.com/repos/yandex/ClickHouse/pulls/2427","author_association":"COLLABORATOR","_links":{"self":{"href":"https://api.github.com/repos/yandex/ClickHouse/pulls/comments/191072007"},"html":{"href":"https://github.com/yandex/ClickHouse/pull/2427#discussion_r191072007"},"pull_request":{"href":"https://api.github.com/repos/yandex/ClickHouse/pulls/2427"}}},"pull_request":{"url":"https://api.github.com/repos/yandex/ClickHouse/pulls/2427","id":190759479,"html_url":"https://github.com/yandex/ClickHouse/pull/2427","diff_url":"https://github.com/yandex/ClickHouse/pull/2427.diff","patch_url":"https://github.com/yandex/ClickHouse/pull/2427.patch","issue_url":"https://api.github.com/repos/yandex/ClickHouse/issues/2427","number":2427,"state":"open","locked":false,"title":"[WIP] Add parquet support","user":{"login":"luc1ph3r","id":3271477,"avatar_url":"https://avatars3.githubusercontent.com/u/3271477?v=4","gravatar_id":"","url":"https://api.github.com/users/luc1ph3r","html_url":"https://github.com/luc1ph3r","followers_url":"https://api.github.com/users/luc1ph3r/followers","following_url":"https://api.github.com/users/luc1ph3r/following{/other_user}","gists_url":"https://api.github.com/users/luc1ph3r/gists{/gist_id}","starred_url":"https://api.github.com/users/luc1ph3r/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/luc1ph3r/subscriptions","organizations_url":"https://api.github.com/users/luc1ph3r/orgs","repos_url":"https://api.github.com/users/luc1ph3r/repos","events_url":"https://api.github.com/users/luc1ph3r/events{/privacy}","received_events_url":"https://api.github.com/users/luc1ph3r/received_events","type":"User","site_admin":false},"body":"In this PR [Parquet](https://parquet.apache.org/) data format support is implemented.\r\nThe input and output processing is implemented using [parquet-cpp](https://github.com/apache/parquet-cpp/) library in conjunction with [arrow](https://github.com/apache/arrow).\r\n\r\nCurrent state:\r\n- New `Format` is created — `Parquet`.  Common usages would be:\r\n    Sending data from a file to a server:\r\n    ```bash\r\n    $ cat data.parquet | clickhouse-client --query=\"INSERT INTO table FORMAT Parquet\"\r\n    ```\r\n\r\n    Storing table data into a parquet file:\r\n    ```bash\r\n    $ clickhouse-client --query=\"SELECT * FROM table FORMAT Parquet\" > data.parquet\r\n    ```\r\n- Supported input [types](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md)* (Parquet type -> ClickHouse type):\r\n    - [x] All numerical types (`INT_8`, `INT_16`, `INT_32`, `INT_64`, `UINT_8`, `UINT_16`, `UINT_32`, `UINT_64`, `FLOAT`, `DOUBLE`)\r\n    - [x] `BOOLEAN` -> `UInt8`\r\n    - [x] `DATE` -> `Date` (**Note:** **arrow** library stores `DATE` type as Int32, while ClickHouse stores `Date` as UInt16. Therefore implicit conversion is performed. Also internal dates are limited by the compilation-time variable [DATE_LUT_MAX_DAY_NUM](https://github.com/yandex/ClickHouse/blob/master/libs/libcommon/include/common/DateLUTImpl.h#L10))\r\n    - [x] `UTF8` -> `String`\r\n    - [ ] `UUID` -> `String`\r\n    - [ ] `JSON` -> `String`\r\n    TODO?:\r\n    - [ ] `TIME_MILLIS`\r\n    - [ ] `TIME_MICROS`\r\n- Supported output [types](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md)* (ClickHouse type -> Parquet type):\r\n    - [x] All numerical types (`INT_8`, `INT_16`, `INT_32`, `INT_64`, `UINT_8`, `UINT_16`, `UINT_32`, `UINT_64`, `FLOAT`, `DOUBLE`)\r\n    - [x] `Date` -> `DATE`\r\n    - [x] `String` -> `UTF8`\r\n    - [ ] `FixedString` -> `BYTE_ARRAY`\r\n    - [ ] `DateTime` -> `TIMESTAMP_MILLIS`\r\n\r\n&ast; Types that are not included are probably not convertable into internal (ClickHouse) ones.\r\n\r\nTODO:\r\n- [ ] Support NULLs\r\n- [ ] Optimize file reading (do not store a whole file in memory)\r\n- [ ] Output a file using [row group](https://parquet.apache.org/documentation/latest/) size estimation (a number of rows in a continuous chunk of data)\r\n- [ ] Make parquet support optional on a build step\r\n- [ ] Set up library building within the project using CMake\r\n- [ ] Use compression\r\n- [ ] Write tests\r\n\r\n\r\n---\r\nI hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en\r\n","created_at":"2018-05-26T21:16:52Z","updated_at":"2018-05-27T10:30:49Z","closed_at":null,"merged_at":null,"merge_commit_sha":"9e1f0c495bc61f251ff6be5b605a98ca1eb76e8c","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/yandex/ClickHouse/pulls/2427/commits","review_comments_url":"https://api.github.com/repos/yandex/ClickHouse/pulls/2427/comments","review_comment_url":"https://api.github.com/repos/yandex/ClickHouse/pulls/comments{/number}","comments_url":"https://api.github.com/repos/yandex/ClickHouse/issues/2427/comments","statuses_url":"https://api.github.com/repos/yandex/ClickHouse/statuses/882e246e4f9fdf9fe73293afc731427531b6e5da","head":{"label":"luc1ph3r:add-parquet-support","ref":"add-parquet-support","sha":"882e246e4f9fdf9fe73293afc731427531b6e5da","user":{"login":"luc1ph3r","id":3271477,"avatar_url":"https://avatars3.githubusercontent.com/u/3271477?v=4","gravatar_id":"","url":"https://api.github.com/users/luc1ph3r","html_url":"https://github.com/luc1ph3r","followers_url":"https://api.github.com/users/luc1ph3r/followers","following_url":"https://api.github.com/users/luc1ph3r/following{/other_user}","gists_url":"https://api.github.com/users/luc1ph3r/gists{/gist_id}","starred_url":"https://api.github.com/users/luc1ph3r/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/luc1ph3r/subscriptions","organizations_url":"https://api.github.com/users/luc1ph3r/orgs","repos_url":"https://api.github.com/users/luc1ph3r/repos","events_url":"https://api.github.com/users/luc1ph3r/events{/privacy}","received_events_url":"https://api.github.com/users/luc1ph3r/received_events","type":"User","site_admin":false},"repo":{"id":110275494,"name":"ClickHouse","full_name":"luc1ph3r/ClickHouse","owner":{"login":"luc1ph3r","id":3271477,"avatar_url":"https://avatars3.githubusercontent.com/u/3271477?v=4","gravatar_id":"","url":"https://api.github.com/users/luc1ph3r","html_url":"https://github.com/luc1ph3r","followers_url":"https://api.github.com/users/luc1ph3r/followers","following_url":"https://api.github.com/users/luc1ph3r/following{/other_user}","gists_url":"https://api.github.com/users/luc1ph3r/gists{/gist_id}","starred_url":"https://api.github.com/users/luc1ph3r/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/luc1ph3r/subscriptions","organizations_url":"https://api.github.com/users/luc1ph3r/orgs","repos_url":"https://api.github.com/users/luc1ph3r/repos","events_url":"https://api.github.com/users/luc1ph3r/events{/privacy}","received_events_url":"https://api.github.com/users/luc1ph3r/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/luc1ph3r/ClickHouse","description":"ClickHouse is a free analytic DBMS for big data.","fork":true,"url":"https://api.github.com/repos/luc1ph3r/ClickHouse","forks_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/forks","keys_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/keys{/key_id}","collaborators_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/teams","hooks_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/hooks","issue_events_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/issues/events{/number}","events_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/events","assignees_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/assignees{/user}","branches_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/branches{/branch}","tags_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/tags","blobs_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/git/refs{/sha}","trees_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/git/trees{/sha}","statuses_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/statuses/{sha}","languages_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/languages","stargazers_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/stargazers","contributors_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/contributors","subscribers_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/subscribers","subscription_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/subscription","commits_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/commits{/sha}","git_commits_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/git/commits{/sha}","comments_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/comments{/number}","issue_comment_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/issues/comments{/number}","contents_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/contents/{+path}","compare_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/compare/{base}...{head}","merges_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/merges","archive_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/downloads","issues_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/issues{/number}","pulls_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/pulls{/number}","milestones_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/milestones{/number}","notifications_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/labels{/name}","releases_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/releases{/id}","deployments_url":"https://api.github.com/repos/luc1ph3r/ClickHouse/deployments","created_at":"2017-11-10T17:42:24Z","updated_at":"2018-05-18T20:56:27Z","pushed_at":"2018-05-26T20:45:13Z","git_url":"git://github.com/luc1ph3r/ClickHouse.git","ssh_url":"git@github.com:luc1ph3r/ClickHouse.git","clone_url":"https://github.com/luc1ph3r/ClickHouse.git","svn_url":"https://github.com/luc1ph3r/ClickHouse","homepage":"https://clickhouse.yandex","size":86137,"stargazers_count":0,"watchers_count":0,"language":"C++","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":1,"license":{"key":"other","name":"Other","spdx_id":null,"url":null},"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"base":{"label":"yandex:master","ref":"master","sha":"d4b861df899c408ab57f518936d97d4c632e3a49","user":{"login":"yandex","id":7409213,"avatar_url":"https://avatars1.githubusercontent.com/u/7409213?v=4","gravatar_id":"","url":"https://api.github.com/users/yandex","html_url":"https://github.com/yandex","followers_url":"https://api.github.com/users/yandex/followers","following_url":"https://api.github.com/users/yandex/following{/other_user}","gists_url":"https://api.github.com/users/yandex/gists{/gist_id}","starred_url":"https://api.github.com/users/yandex/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yandex/subscriptions","organizations_url":"https://api.github.com/users/yandex/orgs","repos_url":"https://api.github.com/users/yandex/repos","events_url":"https://api.github.com/users/yandex/events{/privacy}","received_events_url":"https://api.github.com/users/yandex/received_events","type":"Organization","site_admin":false},"repo":{"id":60246359,"name":"ClickHouse","full_name":"yandex/ClickHouse","owner":{"login":"yandex","id":7409213,"avatar_url":"https://avatars1.githubusercontent.com/u/7409213?v=4","gravatar_id":"","url":"https://api.github.com/users/yandex","html_url":"https://github.com/yandex","followers_url":"https://api.github.com/users/yandex/followers","following_url":"https://api.github.com/users/yandex/following{/other_user}","gists_url":"https://api.github.com/users/yandex/gists{/gist_id}","starred_url":"https://api.github.com/users/yandex/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yandex/subscriptions","organizations_url":"https://api.github.com/users/yandex/orgs","repos_url":"https://api.github.com/users/yandex/repos","events_url":"https://api.github.com/users/yandex/events{/privacy}","received_events_url":"https://api.github.com/users/yandex/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/yandex/ClickHouse","description":"ClickHouse is a free analytic DBMS for big data.","fork":false,"url":"https://api.github.com/repos/yandex/ClickHouse","forks_url":"https://api.github.com/repos/yandex/ClickHouse/forks","keys_url":"https://api.github.com/repos/yandex/ClickHouse/keys{/key_id}","collaborators_url":"https://api.github.com/repos/yandex/ClickHouse/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/yandex/ClickHouse/teams","hooks_url":"https://api.github.com/repos/yandex/ClickHouse/hooks","issue_events_url":"https://api.github.com/repos/yandex/ClickHouse/issues/events{/number}","events_url":"https://api.github.com/repos/yandex/ClickHouse/events","assignees_url":"https://api.github.com/repos/yandex/ClickHouse/assignees{/user}","branches_url":"https://api.github.com/repos/yandex/ClickHouse/branches{/branch}","tags_url":"https://api.github.com/repos/yandex/ClickHouse/tags","blobs_url":"https://api.github.com/repos/yandex/ClickHouse/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/yandex/ClickHouse/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/yandex/ClickHouse/git/refs{/sha}","trees_url":"https://api.github.com/repos/yandex/ClickHouse/git/trees{/sha}","statuses_url":"https://api.github.com/repos/yandex/ClickHouse/statuses/{sha}","languages_url":"https://api.github.com/repos/yandex/ClickHouse/languages","stargazers_url":"https://api.github.com/repos/yandex/ClickHouse/stargazers","contributors_url":"https://api.github.com/repos/yandex/ClickHouse/contributors","subscribers_url":"https://api.github.com/repos/yandex/ClickHouse/subscribers","subscription_url":"https://api.github.com/repos/yandex/ClickHouse/subscription","commits_url":"https://api.github.com/repos/yandex/ClickHouse/commits{/sha}","git_commits_url":"https://api.github.com/repos/yandex/ClickHouse/git/commits{/sha}","comments_url":"https://api.github.com/repos/yandex/ClickHouse/comments{/number}","issue_comment_url":"https://api.github.com/repos/yandex/ClickHouse/issues/comments{/number}","contents_url":"https://api.github.com/repos/yandex/ClickHouse/contents/{+path}","compare_url":"https://api.github.com/repos/yandex/ClickHouse/compare/{base}...{head}","merges_url":"https://api.github.com/repos/yandex/ClickHouse/merges","archive_url":"https://api.github.com/repos/yandex/ClickHouse/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/yandex/ClickHouse/downloads","issues_url":"https://api.github.com/repos/yandex/ClickHouse/issues{/number}","pulls_url":"https://api.github.com/repos/yandex/ClickHouse/pulls{/number}","milestones_url":"https://api.github.com/repos/yandex/ClickHouse/milestones{/number}","notifications_url":"https://api.github.com/repos/yandex/ClickHouse/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/yandex/ClickHouse/labels{/name}","releases_url":"https://api.github.com/repos/yandex/ClickHouse/releases{/id}","deployments_url":"https://api.github.com/repos/yandex/ClickHouse/deployments","created_at":"2016-06-02T08:28:18Z","updated_at":"2018-05-27T08:57:52Z","pushed_at":"2018-05-26T21:16:52Z","git_url":"git://github.com/yandex/ClickHouse.git","ssh_url":"git@github.com:yandex/ClickHouse.git","clone_url":"https://github.com/yandex/ClickHouse.git","svn_url":"https://github.com/yandex/ClickHouse","homepage":"https://clickhouse.yandex","size":87363,"stargazers_count":4091,"watchers_count":4091,"language":"C++","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":617,"mirror_url":null,"archived":false,"open_issues_count":451,"license":{"key":"other","name":"Other","spdx_id":null,"url":null},"forks":617,"open_issues":451,"watchers":4091,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/yandex/ClickHouse/pulls/2427"},"html":{"href":"https://github.com/yandex/ClickHouse/pull/2427"},"issue":{"href":"https://api.github.com/repos/yandex/ClickHouse/issues/2427"},"comments":{"href":"https://api.github.com/repos/yandex/ClickHouse/issues/2427/comments"},"review_comments":{"href":"https://api.github.com/repos/yandex/ClickHouse/pulls/2427/comments"},"review_comment":{"href":"https://api.github.com/repos/yandex/ClickHouse/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/yandex/ClickHouse/pulls/2427/commits"},"statuses":{"href":"https://api.github.com/repos/yandex/ClickHouse/statuses/882e246e4f9fdf9fe73293afc731427531b6e5da"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"ludv1x"}}