{"action":"created","issue":{"url":"https://api.github.com/repos/mcaceresb/stata-parquet/issues/7","repository_url":"https://api.github.com/repos/mcaceresb/stata-parquet","labels_url":"https://api.github.com/repos/mcaceresb/stata-parquet/issues/7/labels{/name}","comments_url":"https://api.github.com/repos/mcaceresb/stata-parquet/issues/7/comments","events_url":"https://api.github.com/repos/mcaceresb/stata-parquet/issues/7/events","html_url":"https://github.com/mcaceresb/stata-parquet/issues/7","id":375959666,"node_id":"MDU6SXNzdWUzNzU5NTk2NjY=","number":7,"title":"FixedLenByteArray support","user":{"login":"kylebarron","id":15164633,"node_id":"MDQ6VXNlcjE1MTY0NjMz","avatar_url":"https://avatars3.githubusercontent.com/u/15164633?v=4","gravatar_id":"","url":"https://api.github.com/users/kylebarron","html_url":"https://github.com/kylebarron","followers_url":"https://api.github.com/users/kylebarron/followers","following_url":"https://api.github.com/users/kylebarron/following{/other_user}","gists_url":"https://api.github.com/users/kylebarron/gists{/gist_id}","starred_url":"https://api.github.com/users/kylebarron/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kylebarron/subscriptions","organizations_url":"https://api.github.com/users/kylebarron/orgs","repos_url":"https://api.github.com/users/kylebarron/repos","events_url":"https://api.github.com/users/kylebarron/events{/privacy}","received_events_url":"https://api.github.com/users/kylebarron/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-10-31T13:22:42Z","updated_at":"2018-10-31T20:11:17Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"> Reading FixedLenByteArray and Int96 variables are not supported, as they have no direct Stata counterpart\r\n\r\n`FixedLenByteArray` is actually closer to a direct Stata counterpart. It's my understanding that with `ByteArray`, you have no idea how many bytes are in each row of the column, so you have to set `str255`, import the data, and then compress. \r\n\r\n`FixedLenByteArray` is set to have the same width of bytes in every row.\r\n\r\nI'm not sure how prevalent this field is in real-world Parquet data. Unless you have a column where every row is the same length, like an MD5 hash, the variable length encoding is much more space-efficient.\r\n\r\nIn any case, reading in all string columns as `str255` is kind of untenable for large datasets with lots of strings. I'm not sure what the best way to get around this is... Set each `str#` value low, and then make Stata recast the type to a larger string as you're importing the data?"},"comment":{"url":"https://api.github.com/repos/mcaceresb/stata-parquet/issues/comments/434830803","html_url":"https://github.com/mcaceresb/stata-parquet/issues/7#issuecomment-434830803","issue_url":"https://api.github.com/repos/mcaceresb/stata-parquet/issues/7","id":434830803,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNDgzMDgwMw==","user":{"login":"mcaceresb","id":4219197,"node_id":"MDQ6VXNlcjQyMTkxOTc=","avatar_url":"https://avatars0.githubusercontent.com/u/4219197?v=4","gravatar_id":"","url":"https://api.github.com/users/mcaceresb","html_url":"https://github.com/mcaceresb","followers_url":"https://api.github.com/users/mcaceresb/followers","following_url":"https://api.github.com/users/mcaceresb/following{/other_user}","gists_url":"https://api.github.com/users/mcaceresb/gists{/gist_id}","starred_url":"https://api.github.com/users/mcaceresb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mcaceresb/subscriptions","organizations_url":"https://api.github.com/users/mcaceresb/orgs","repos_url":"https://api.github.com/users/mcaceresb/repos","events_url":"https://api.github.com/users/mcaceresb/events{/privacy}","received_events_url":"https://api.github.com/users/mcaceresb/received_events","type":"User","site_admin":false},"created_at":"2018-10-31T20:11:17Z","updated_at":"2018-10-31T20:11:17Z","author_association":"OWNER","body":"Dan made a good point noting that we would expect FixedLenByteArray to be better at storing string IDs (e.g. encrypted bene id, hashes, etc.). Even if stored as binary instead of strings they would still be fixed width.\r\n\r\nFor arbitrary strings, which I assume are much more common than encrypted IDs and hashes, ByteArray would be the way to go because it saves space. "}}
{"actor":{"display_login":"mcaceresb"}}