{"action":"created","issue":{"url":"https://api.github.com/repos/mcaceresb/stata-parquet/issues/14","repository_url":"https://api.github.com/repos/mcaceresb/stata-parquet","labels_url":"https://api.github.com/repos/mcaceresb/stata-parquet/issues/14/labels{/name}","comments_url":"https://api.github.com/repos/mcaceresb/stata-parquet/issues/14/comments","events_url":"https://api.github.com/repos/mcaceresb/stata-parquet/issues/14/events","html_url":"https://github.com/mcaceresb/stata-parquet/issues/14","id":378824006,"node_id":"MDU6SXNzdWUzNzg4MjQwMDY=","number":14,"title":"Support reading \"Parquet Datasets\"","user":{"login":"kylebarron","id":15164633,"node_id":"MDQ6VXNlcjE1MTY0NjMz","avatar_url":"https://avatars3.githubusercontent.com/u/15164633?v=4","gravatar_id":"","url":"https://api.github.com/users/kylebarron","html_url":"https://github.com/kylebarron","followers_url":"https://api.github.com/users/kylebarron/followers","following_url":"https://api.github.com/users/kylebarron/following{/other_user}","gists_url":"https://api.github.com/users/kylebarron/gists{/gist_id}","starred_url":"https://api.github.com/users/kylebarron/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kylebarron/subscriptions","organizations_url":"https://api.github.com/users/kylebarron/orgs","repos_url":"https://api.github.com/users/kylebarron/repos","events_url":"https://api.github.com/users/kylebarron/events{/privacy}","received_events_url":"https://api.github.com/users/kylebarron/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-11-08T16:51:38Z","updated_at":"2018-11-08T23:29:53Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Parquet files written by Spark or with `fastparquet` with the option `file_scheme=hive` are written into folders with the name `filename.parquet/`, with multiple files inside, one for each row group, to help with parallel reads. This isn't currently supported here.\r\n\r\n```\r\n> ll med1999.parquet\r\ntotal 16M\r\n-rw-r--r-- 1 barronk-dua51929 dua51929 1.1M Nov  3 11:17 part-00000-28fcf1da-b8f8-49ca-ab2f-c5bd65e2d65c-c000.snappy.parquet\r\n-rw-r--r-- 1 barronk-dua51929 dua51929 1.1M Nov  3 11:17 part-00001-28fcf1da-b8f8-49ca-ab2f-c5bd65e2d65c-c000.snappy.parquet\r\n-rw-r--r-- 1 barronk-dua51929 dua51929 1.1M Nov  3 11:17 part-00002-28fcf1da-b8f8-49ca-ab2f-c5bd65e2d65c-c000.snappy.parquet\r\n-rw-r--r-- 1 barronk-dua51929 dua51929 1.1M Nov  3 11:17 part-00003-28fcf1da-b8f8-49ca-ab2f-c5bd65e2d65c-c000.snappy.parquet\r\n-rw-r--r-- 1 barronk-dua51929 dua51929 1.1M Nov  3 11:17 part-00004-28fcf1da-b8f8-49ca-ab2f-c5bd65e2d65c-c000.snappy.parquet\r\n-rw-r--r-- 1 barronk-dua51929 dua51929 1.1M Nov  3 11:17 part-00005-28fcf1da-b8f8-49ca-ab2f-c5bd65e2d65c-c000.snappy.parquet\r\n-rw-r--r-- 1 barronk-dua51929 dua51929 1.1M Nov  3 11:17 part-00006-28fcf1da-b8f8-49ca-ab2f-c5bd65e2d65c-c000.snappy.parquet\r\n-rw-r--r-- 1 barronk-dua51929 dua51929 1.1M Nov  3 11:17 part-00007-28fcf1da-b8f8-49ca-ab2f-c5bd65e2d65c-c000.snappy.parquet\r\n-rw-r--r-- 1 barronk-dua51929 dua51929 1.1M Nov  3 11:17 part-00008-28fcf1da-b8f8-49ca-ab2f-c5bd65e2d65c-c000.snappy.parquet\r\n-rw-r--r-- 1 barronk-dua51929 dua51929 1.1M Nov  3 11:17 part-00009-28fcf1da-b8f8-49ca-ab2f-c5bd65e2d65c-c000.snappy.parquet\r\n-rw-r--r-- 1 barronk-dua51929 dua51929 1.1M Nov  3 11:17 part-00010-28fcf1da-b8f8-49ca-ab2f-c5bd65e2d65c-c000.snappy.parquet\r\n-rw-r--r-- 1 barronk-dua51929 dua51929 1.1M Nov  3 11:17 part-00011-28fcf1da-b8f8-49ca-ab2f-c5bd65e2d65c-c000.snappy.parquet\r\n-rw-r--r-- 1 barronk-dua51929 dua51929 1.1M Nov  3 11:17 part-00012-28fcf1da-b8f8-49ca-ab2f-c5bd65e2d65c-c000.snappy.parquet\r\n-rw-r--r-- 1 barronk-dua51929 dua51929 1.1M Nov  3 11:17 part-00013-28fcf1da-b8f8-49ca-ab2f-c5bd65e2d65c-c000.snappy.parquet\r\n-rw-r--r-- 1 barronk-dua51929 dua51929 1.1M Nov  3 11:17 part-00014-28fcf1da-b8f8-49ca-ab2f-c5bd65e2d65c-c000.snappy.parquet\r\n-rw-r--r-- 1 barronk-dua51929 dua51929 1.1M Nov  3 11:17 part-00015-28fcf1da-b8f8-49ca-ab2f-c5bd65e2d65c-c000.snappy.parquet\r\n-rw-r--r-- 1 barronk-dua51929 dua51929 940K Nov  3 11:17 part-00016-28fcf1da-b8f8-49ca-ab2f-c5bd65e2d65c-c000.snappy.parquet\r\n-rw-r--r-- 1 barronk-dua51929 dua51929    0 Nov  3 11:17 _SUCCESS\r\n```\r\n\r\n![image](https://user-images.githubusercontent.com/15164633/48212951-b5ae8a80-e34a-11e8-93ea-708360f9d896.png)\r\n\r\nYou can create a similar dataset with `fastparquet` with:\r\n```py\r\nimport pandas as pd\r\nimport numpy as np\r\nimport fastparquet as fp\r\ndf = pd.DataFrame(np.random.randint(0,100,size=(4, 4)), columns=list('ABCD'))\r\nfp.write('test.parquet', df, row_group_offsets=2, file_scheme='hive')\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/mcaceresb/stata-parquet/issues/comments/437194782","html_url":"https://github.com/mcaceresb/stata-parquet/issues/14#issuecomment-437194782","issue_url":"https://api.github.com/repos/mcaceresb/stata-parquet/issues/14","id":437194782,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNzE5NDc4Mg==","user":{"login":"mcaceresb","id":4219197,"node_id":"MDQ6VXNlcjQyMTkxOTc=","avatar_url":"https://avatars0.githubusercontent.com/u/4219197?v=4","gravatar_id":"","url":"https://api.github.com/users/mcaceresb","html_url":"https://github.com/mcaceresb","followers_url":"https://api.github.com/users/mcaceresb/followers","following_url":"https://api.github.com/users/mcaceresb/following{/other_user}","gists_url":"https://api.github.com/users/mcaceresb/gists{/gist_id}","starred_url":"https://api.github.com/users/mcaceresb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mcaceresb/subscriptions","organizations_url":"https://api.github.com/users/mcaceresb/orgs","repos_url":"https://api.github.com/users/mcaceresb/repos","events_url":"https://api.github.com/users/mcaceresb/events{/privacy}","received_events_url":"https://api.github.com/users/mcaceresb/received_events","type":"User","site_admin":false},"created_at":"2018-11-08T23:29:53Z","updated_at":"2018-11-08T23:29:53Z","author_association":"OWNER","body":"Actually, if you update to the latest version in the repo it reads them just fine. There's a ton of missing values, I'm assuming because the rest of the data is in the other files.\r\n\r\n```\r\n. parquet use part-00000-863c36b3-bd34-4eb3-9149-3d7c5243e080-c000.snappy.parquet, clear\r\nScanned column types (0.03 sec).\r\n        File:    part-00000-863c36b3-bd34-4eb3-9149-3d7c5243e080-c000.snappy.parquet\r\n        Groups:  1\r\n        Columns: 365\r\n        Rows:    5408\r\nWarning: 580685 NaN values in string variables coerced to blanks ('').\r\nRead data from disk (0.05 sec).\r\n```"}}
{"actor":{"display_login":"mcaceresb"}}