{"action":"created","comment":{"url":"https://api.github.com/repos/apache/parquet-cpp/pulls/comments/51321552","id":51321552,"diff_hunk":"@@ -68,13 +69,14 @@ class ColumnReader {\n   static std::shared_ptr<ColumnReader> Make(const parquet::ColumnMetaData*,\n       const parquet::SchemaElement*, std::unique_ptr<InputStream> stream);\n \n-  virtual bool ReadNewPage() = 0;\n-\n   // Returns true if there are still values in this column.\n   bool HasNext() {\n-    if (num_buffered_values_ == 0) {\n-      ReadNewPage();\n-      if (num_buffered_values_ == 0) return false;\n+    // Either there is no data page available yet, or the data page has been\n+    // exhausted\n+    if (!num_buffered_values_ || num_decoded_values_ == num_buffered_values_) {","path":"src/parquet/column/reader.h","position":30,"original_position":30,"commit_id":"7ea261eb62bbfc8a4cd2434bf14d0c0732f7796f","original_commit_id":"7ea261eb62bbfc8a4cd2434bf14d0c0732f7796f","user":{"login":"wesm","id":329591,"avatar_url":"https://avatars.githubusercontent.com/u/329591?v=3","gravatar_id":"","url":"https://api.github.com/users/wesm","html_url":"https://github.com/wesm","followers_url":"https://api.github.com/users/wesm/followers","following_url":"https://api.github.com/users/wesm/following{/other_user}","gists_url":"https://api.github.com/users/wesm/gists{/gist_id}","starred_url":"https://api.github.com/users/wesm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wesm/subscriptions","organizations_url":"https://api.github.com/users/wesm/orgs","repos_url":"https://api.github.com/users/wesm/repos","events_url":"https://api.github.com/users/wesm/events{/privacy}","received_events_url":"https://api.github.com/users/wesm/received_events","type":"User","site_admin":false},"body":"noted, fixing now","created_at":"2016-01-29T21:57:02Z","updated_at":"2016-01-29T21:57:02Z","html_url":"https://github.com/apache/parquet-cpp/pull/26#discussion_r51321552","pull_request_url":"https://api.github.com/repos/apache/parquet-cpp/pulls/26","_links":{"self":{"href":"https://api.github.com/repos/apache/parquet-cpp/pulls/comments/51321552"},"html":{"href":"https://github.com/apache/parquet-cpp/pull/26#discussion_r51321552"},"pull_request":{"href":"https://api.github.com/repos/apache/parquet-cpp/pulls/26"}}},"pull_request":{"url":"https://api.github.com/repos/apache/parquet-cpp/pulls/26","id":57316060,"html_url":"https://github.com/apache/parquet-cpp/pull/26","diff_url":"https://github.com/apache/parquet-cpp/pull/26.diff","patch_url":"https://github.com/apache/parquet-cpp/pull/26.patch","issue_url":"https://api.github.com/repos/apache/parquet-cpp/issues/26","number":26,"state":"open","locked":false,"title":"PARQUET-435: Change column reader methods to be array-oriented rather than scalar","user":{"login":"wesm","id":329591,"avatar_url":"https://avatars.githubusercontent.com/u/329591?v=3","gravatar_id":"","url":"https://api.github.com/users/wesm","html_url":"https://github.com/wesm","followers_url":"https://api.github.com/users/wesm/followers","following_url":"https://api.github.com/users/wesm/following{/other_user}","gists_url":"https://api.github.com/users/wesm/gists{/gist_id}","starred_url":"https://api.github.com/users/wesm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wesm/subscriptions","organizations_url":"https://api.github.com/users/wesm/orgs","repos_url":"https://api.github.com/users/wesm/repos","events_url":"https://api.github.com/users/wesm/events{/privacy}","received_events_url":"https://api.github.com/users/wesm/received_events","type":"User","site_admin":false},"body":"Column scanning and record reconstruction is independent of the Parquet file format and depends, among other things, on the data structures where the reconstructed data will end up. This is a work-in progress, but the basic idea is:\r\n\r\n- APIs for reading a batch of repetition `ReadRepetitionLevels` or definition levels `ReadDefinitionLevels` into a preallocated `int16_t*`\r\n- APIs for reading arrays of decoded values into preallocated memory (`ReadValues`)\r\n\r\nThese methods are only able to read data within a particular data page. Once you exhaust the data available in the data page (`ReadValues` returns 0), you must call `ReadNewPage`, which returns `true` is there is more data available.\r\n\r\nSeparately, I added a simple `Scanner` class that emulates the scalar value iteration functionality that existed previously. I used this to reimplement the `DebugPrint` method in `parquet_scanner.cc`. This obviously only works currently for flat data. \r\n\r\nI would like to keep the `ColumnReader` low level and primitive, concerned only with providing access to the raw data in a Parquet file as fast as possible. We can devise separate algorithms for inferring nested record structure by examining the arrays of decoded values and repetition/definition levels. The major benefit of separating raw data access from structure inference is that this can be pipelined with threads: one thread decompresses and decodes values and levels, and another thread can turn batches into a nested record- or column-oriented structure.\r\n\r\nComments welcome. Thanks","created_at":"2016-01-27T05:37:41Z","updated_at":"2016-01-29T21:57:02Z","closed_at":null,"merged_at":null,"merge_commit_sha":"5fcf733e36691ba1593b68a2bec2f59494a74d8b","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/apache/parquet-cpp/pulls/26/commits","review_comments_url":"https://api.github.com/repos/apache/parquet-cpp/pulls/26/comments","review_comment_url":"https://api.github.com/repos/apache/parquet-cpp/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/parquet-cpp/issues/26/comments","statuses_url":"https://api.github.com/repos/apache/parquet-cpp/statuses/7ea261eb62bbfc8a4cd2434bf14d0c0732f7796f","head":{"label":"wesm:PARQUET-435","ref":"PARQUET-435","sha":"7ea261eb62bbfc8a4cd2434bf14d0c0732f7796f","user":{"login":"wesm","id":329591,"avatar_url":"https://avatars.githubusercontent.com/u/329591?v=3","gravatar_id":"","url":"https://api.github.com/users/wesm","html_url":"https://github.com/wesm","followers_url":"https://api.github.com/users/wesm/followers","following_url":"https://api.github.com/users/wesm/following{/other_user}","gists_url":"https://api.github.com/users/wesm/gists{/gist_id}","starred_url":"https://api.github.com/users/wesm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wesm/subscriptions","organizations_url":"https://api.github.com/users/wesm/orgs","repos_url":"https://api.github.com/users/wesm/repos","events_url":"https://api.github.com/users/wesm/events{/privacy}","received_events_url":"https://api.github.com/users/wesm/received_events","type":"User","site_admin":false},"repo":{"id":48463753,"name":"parquet-cpp","full_name":"wesm/parquet-cpp","owner":{"login":"wesm","id":329591,"avatar_url":"https://avatars.githubusercontent.com/u/329591?v=3","gravatar_id":"","url":"https://api.github.com/users/wesm","html_url":"https://github.com/wesm","followers_url":"https://api.github.com/users/wesm/followers","following_url":"https://api.github.com/users/wesm/following{/other_user}","gists_url":"https://api.github.com/users/wesm/gists{/gist_id}","starred_url":"https://api.github.com/users/wesm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wesm/subscriptions","organizations_url":"https://api.github.com/users/wesm/orgs","repos_url":"https://api.github.com/users/wesm/repos","events_url":"https://api.github.com/users/wesm/events{/privacy}","received_events_url":"https://api.github.com/users/wesm/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/wesm/parquet-cpp","description":"Mirror of Apache Parquet","fork":true,"url":"https://api.github.com/repos/wesm/parquet-cpp","forks_url":"https://api.github.com/repos/wesm/parquet-cpp/forks","keys_url":"https://api.github.com/repos/wesm/parquet-cpp/keys{/key_id}","collaborators_url":"https://api.github.com/repos/wesm/parquet-cpp/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/wesm/parquet-cpp/teams","hooks_url":"https://api.github.com/repos/wesm/parquet-cpp/hooks","issue_events_url":"https://api.github.com/repos/wesm/parquet-cpp/issues/events{/number}","events_url":"https://api.github.com/repos/wesm/parquet-cpp/events","assignees_url":"https://api.github.com/repos/wesm/parquet-cpp/assignees{/user}","branches_url":"https://api.github.com/repos/wesm/parquet-cpp/branches{/branch}","tags_url":"https://api.github.com/repos/wesm/parquet-cpp/tags","blobs_url":"https://api.github.com/repos/wesm/parquet-cpp/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/wesm/parquet-cpp/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/wesm/parquet-cpp/git/refs{/sha}","trees_url":"https://api.github.com/repos/wesm/parquet-cpp/git/trees{/sha}","statuses_url":"https://api.github.com/repos/wesm/parquet-cpp/statuses/{sha}","languages_url":"https://api.github.com/repos/wesm/parquet-cpp/languages","stargazers_url":"https://api.github.com/repos/wesm/parquet-cpp/stargazers","contributors_url":"https://api.github.com/repos/wesm/parquet-cpp/contributors","subscribers_url":"https://api.github.com/repos/wesm/parquet-cpp/subscribers","subscription_url":"https://api.github.com/repos/wesm/parquet-cpp/subscription","commits_url":"https://api.github.com/repos/wesm/parquet-cpp/commits{/sha}","git_commits_url":"https://api.github.com/repos/wesm/parquet-cpp/git/commits{/sha}","comments_url":"https://api.github.com/repos/wesm/parquet-cpp/comments{/number}","issue_comment_url":"https://api.github.com/repos/wesm/parquet-cpp/issues/comments{/number}","contents_url":"https://api.github.com/repos/wesm/parquet-cpp/contents/{+path}","compare_url":"https://api.github.com/repos/wesm/parquet-cpp/compare/{base}...{head}","merges_url":"https://api.github.com/repos/wesm/parquet-cpp/merges","archive_url":"https://api.github.com/repos/wesm/parquet-cpp/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/wesm/parquet-cpp/downloads","issues_url":"https://api.github.com/repos/wesm/parquet-cpp/issues{/number}","pulls_url":"https://api.github.com/repos/wesm/parquet-cpp/pulls{/number}","milestones_url":"https://api.github.com/repos/wesm/parquet-cpp/milestones{/number}","notifications_url":"https://api.github.com/repos/wesm/parquet-cpp/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/wesm/parquet-cpp/labels{/name}","releases_url":"https://api.github.com/repos/wesm/parquet-cpp/releases{/id}","deployments_url":"https://api.github.com/repos/wesm/parquet-cpp/deployments","created_at":"2015-12-23T02:08:59Z","updated_at":"2016-01-11T19:24:28Z","pushed_at":"2016-01-29T20:07:22Z","git_url":"git://github.com/wesm/parquet-cpp.git","ssh_url":"git@github.com:wesm/parquet-cpp.git","clone_url":"https://github.com/wesm/parquet-cpp.git","svn_url":"https://github.com/wesm/parquet-cpp","homepage":null,"size":382,"stargazers_count":0,"watchers_count":0,"language":"C++","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"5e938171b5b6d1effad27559b001c42f7ee0bb2c","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":20675634,"name":"parquet-cpp","full_name":"apache/parquet-cpp","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/parquet-cpp","description":"Mirror of Apache Parquet","fork":false,"url":"https://api.github.com/repos/apache/parquet-cpp","forks_url":"https://api.github.com/repos/apache/parquet-cpp/forks","keys_url":"https://api.github.com/repos/apache/parquet-cpp/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/parquet-cpp/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/parquet-cpp/teams","hooks_url":"https://api.github.com/repos/apache/parquet-cpp/hooks","issue_events_url":"https://api.github.com/repos/apache/parquet-cpp/issues/events{/number}","events_url":"https://api.github.com/repos/apache/parquet-cpp/events","assignees_url":"https://api.github.com/repos/apache/parquet-cpp/assignees{/user}","branches_url":"https://api.github.com/repos/apache/parquet-cpp/branches{/branch}","tags_url":"https://api.github.com/repos/apache/parquet-cpp/tags","blobs_url":"https://api.github.com/repos/apache/parquet-cpp/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/parquet-cpp/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/parquet-cpp/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/parquet-cpp/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/parquet-cpp/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/parquet-cpp/languages","stargazers_url":"https://api.github.com/repos/apache/parquet-cpp/stargazers","contributors_url":"https://api.github.com/repos/apache/parquet-cpp/contributors","subscribers_url":"https://api.github.com/repos/apache/parquet-cpp/subscribers","subscription_url":"https://api.github.com/repos/apache/parquet-cpp/subscription","commits_url":"https://api.github.com/repos/apache/parquet-cpp/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/parquet-cpp/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/parquet-cpp/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/parquet-cpp/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/parquet-cpp/contents/{+path}","compare_url":"https://api.github.com/repos/apache/parquet-cpp/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/parquet-cpp/merges","archive_url":"https://api.github.com/repos/apache/parquet-cpp/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/parquet-cpp/downloads","issues_url":"https://api.github.com/repos/apache/parquet-cpp/issues{/number}","pulls_url":"https://api.github.com/repos/apache/parquet-cpp/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/parquet-cpp/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/parquet-cpp/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/parquet-cpp/labels{/name}","releases_url":"https://api.github.com/repos/apache/parquet-cpp/releases{/id}","deployments_url":"https://api.github.com/repos/apache/parquet-cpp/deployments","created_at":"2014-06-10T07:00:07Z","updated_at":"2016-01-28T06:39:43Z","pushed_at":"2016-01-29T21:23:35Z","git_url":"git://github.com/apache/parquet-cpp.git","ssh_url":"git@github.com:apache/parquet-cpp.git","clone_url":"https://github.com/apache/parquet-cpp.git","svn_url":"https://github.com/apache/parquet-cpp","homepage":null,"size":480,"stargazers_count":20,"watchers_count":20,"language":"C++","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":18,"mirror_url":"git://git.apache.org/parquet-cpp.git","open_issues_count":6,"forks":18,"open_issues":6,"watchers":20,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/parquet-cpp/pulls/26"},"html":{"href":"https://github.com/apache/parquet-cpp/pull/26"},"issue":{"href":"https://api.github.com/repos/apache/parquet-cpp/issues/26"},"comments":{"href":"https://api.github.com/repos/apache/parquet-cpp/issues/26/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/parquet-cpp/pulls/26/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/parquet-cpp/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/parquet-cpp/pulls/26/commits"},"statuses":{"href":"https://api.github.com/repos/apache/parquet-cpp/statuses/7ea261eb62bbfc8a4cd2434bf14d0c0732f7796f"}}}}