{"action":"created","issue":{"url":"https://api.github.com/repos/dask/dask/issues/2308","repository_url":"https://api.github.com/repos/dask/dask","labels_url":"https://api.github.com/repos/dask/dask/issues/2308/labels{/name}","comments_url":"https://api.github.com/repos/dask/dask/issues/2308/comments","events_url":"https://api.github.com/repos/dask/dask/issues/2308/events","html_url":"https://github.com/dask/dask/issues/2308","id":226888959,"number":2308,"title":"Globs don't work when reading multiple hive-format parquet files","user":{"login":"makmanalp","id":161965,"avatar_url":"https://avatars3.githubusercontent.com/u/161965?v=4","gravatar_id":"","url":"https://api.github.com/users/makmanalp","html_url":"https://github.com/makmanalp","followers_url":"https://api.github.com/users/makmanalp/followers","following_url":"https://api.github.com/users/makmanalp/following{/other_user}","gists_url":"https://api.github.com/users/makmanalp/gists{/gist_id}","starred_url":"https://api.github.com/users/makmanalp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/makmanalp/subscriptions","organizations_url":"https://api.github.com/users/makmanalp/orgs","repos_url":"https://api.github.com/users/makmanalp/repos","events_url":"https://api.github.com/users/makmanalp/events{/privacy}","received_events_url":"https://api.github.com/users/makmanalp/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2017-05-07T19:58:25Z","updated_at":"2018-05-22T15:18:13Z","closed_at":null,"author_association":"NONE","body":"Hi,\r\n\r\nWe have a directory full of hive-format parquet files (generated by fastparquet, but I don't think it's relevant here), one for each month, which in totality form a year.\r\n\r\nI'd like to load these files all into one dataframe, with the glob functionality: \r\n\r\n```python\r\ndf  = ddf.read_parquet(\"../data/processed/2016*.parq\")\r\n```\r\n\r\nWhile this works for CSVs and such, it doesn't seem to work for parquet files - I think the machinery that does the glob matching doesn't really know how to deal with the fact that hive-format parquet files are already directories.\r\n\r\nLoading the files individually works OK:\r\n\r\n```python\r\n\r\nIn [12]: df  = ddf.read_parquet(\"../data/processed/201601.parq\")\r\n```\r\n\r\nThe error we get:\r\n\r\n```python\r\n\r\nIn [13]: df  = ddf.read_parquet(\"../data/processed/2016*.parq\")\r\n---------------------------------------------------------------------------\r\nFileNotFoundError                         Traceback (most recent call last)\r\n/nfs/projects_nobackup/c/cidgrowlab/Mali/MBTA/mbtaenv/lib/python3.6/site-packages/fastparquet/api.py in __init__(self, fn, verify, open_with, sep)\r\n     46             self.fn = fn2\r\n---> 47             with open_with(fn2, 'rb') as f:\r\n     48                 self._parse_header(f, verify)\r\n\r\n/nfs/projects_nobackup/c/cidgrowlab/Mali/MBTA/mbtaenv/lib/python3.6/site-packages/dask/bytes/core.py in __enter__(self)\r\n    313         mode = self.mode.replace('t', '').replace('b', '') + 'b'\r\n--> 314         f = f2 = self.myopen(self.path, mode=mode)\r\n    315         CompressFile = merge(seekable_files, compress_files)[self.compression]\r\n\r\n/nfs/projects_nobackup/c/cidgrowlab/Mali/MBTA/mbtaenv/lib/python3.6/site-packages/dask/bytes/local.py in open(self, path, mode, **kwargs)\r\n     62             path = path[len('file://'):]\r\n---> 63         return open(path, mode=mode)\r\n     64 \r\n\r\nFileNotFoundError: [Errno 2] No such file or directory: '../data/processed/2016*.parq/_metadata/_metadata'\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nFileNotFoundError                         Traceback (most recent call last)\r\n/nfs/projects_nobackup/c/cidgrowlab/Mali/MBTA/mbtaenv/lib/python3.6/site-packages/dask/dataframe/io/parquet.py in read_parquet(path, columns, filters, categories, index, storage_options)\r\n     74                                      open_with=myopen,\r\n---> 75                                      sep=myopen.fs.sep)\r\n     76     except:\r\n\r\n/nfs/projects_nobackup/c/cidgrowlab/Mali/MBTA/mbtaenv/lib/python3.6/site-packages/fastparquet/api.py in __init__(self, fn, verify, open_with, sep)\r\n     51             self.fn = fn\r\n---> 52             with open_with(fn, 'rb') as f:\r\n     53                 self._parse_header(f, verify)\r\n\r\n/nfs/projects_nobackup/c/cidgrowlab/Mali/MBTA/mbtaenv/lib/python3.6/site-packages/dask/bytes/core.py in __enter__(self)\r\n    313         mode = self.mode.replace('t', '').replace('b', '') + 'b'\r\n--> 314         f = f2 = self.myopen(self.path, mode=mode)\r\n    315         CompressFile = merge(seekable_files, compress_files)[self.compression]\r\n\r\n/nfs/projects_nobackup/c/cidgrowlab/Mali/MBTA/mbtaenv/lib/python3.6/site-packages/dask/bytes/local.py in open(self, path, mode, **kwargs)\r\n     62             path = path[len('file://'):]\r\n---> 63         return open(path, mode=mode)\r\n     64 \r\n\r\nFileNotFoundError: [Errno 2] No such file or directory: '../data/processed/2016*.parq/_metadata'\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nFileNotFoundError                         Traceback (most recent call last)\r\n/nfs/projects_nobackup/c/cidgrowlab/Mali/MBTA/mbtaenv/lib/python3.6/site-packages/fastparquet/api.py in __init__(self, fn, verify, open_with, sep)\r\n     46             self.fn = fn2\r\n---> 47             with open_with(fn2, 'rb') as f:\r\n     48                 self._parse_header(f, verify)\r\n\r\n/nfs/projects_nobackup/c/cidgrowlab/Mali/MBTA/mbtaenv/lib/python3.6/site-packages/dask/bytes/core.py in __enter__(self)\r\n    313         mode = self.mode.replace('t', '').replace('b', '') + 'b'\r\n--> 314         f = f2 = self.myopen(self.path, mode=mode)\r\n    315         CompressFile = merge(seekable_files, compress_files)[self.compression]\r\n\r\n/nfs/projects_nobackup/c/cidgrowlab/Mali/MBTA/mbtaenv/lib/python3.6/site-packages/dask/bytes/local.py in open(self, path, mode, **kwargs)\r\n     62             path = path[len('file://'):]\r\n---> 63         return open(path, mode=mode)\r\n     64 \r\n\r\nFileNotFoundError: [Errno 2] No such file or directory: '../data/processed/2016*.parq/_metadata'\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nFileNotFoundError                         Traceback (most recent call last)\r\n<ipython-input-13-2d2b23a42644> in <module>()\r\n----> 1 df  = ddf.read_parquet(\"../data/processed/2016*.parq\")\r\n\r\n/nfs/projects_nobackup/c/cidgrowlab/Mali/MBTA/mbtaenv/lib/python3.6/site-packages/dask/dataframe/io/parquet.py in read_parquet(path, columns, filters, categories, index, storage_options)\r\n     75                                      sep=myopen.fs.sep)\r\n     76     except:\r\n---> 77         pf = fastparquet.ParquetFile(path, open_with=myopen, sep=myopen.fs.sep)\r\n     78 \r\n     79     check_column_names(pf.columns, categories)\r\n\r\n/nfs/projects_nobackup/c/cidgrowlab/Mali/MBTA/mbtaenv/lib/python3.6/site-packages/fastparquet/api.py in __init__(self, fn, verify, open_with, sep)\r\n     50         except (IOError, OSError):\r\n     51             self.fn = fn\r\n---> 52             with open_with(fn, 'rb') as f:\r\n     53                 self._parse_header(f, verify)\r\n     54         if all(rg.columns[0].file_path is None for rg in self.row_groups):\r\n\r\n/nfs/projects_nobackup/c/cidgrowlab/Mali/MBTA/mbtaenv/lib/python3.6/site-packages/dask/bytes/core.py in __enter__(self)\r\n    312     def __enter__(self):\r\n    313         mode = self.mode.replace('t', '').replace('b', '') + 'b'\r\n--> 314         f = f2 = self.myopen(self.path, mode=mode)\r\n    315         CompressFile = merge(seekable_files, compress_files)[self.compression]\r\n    316         if PY2:\r\n\r\n/nfs/projects_nobackup/c/cidgrowlab/Mali/MBTA/mbtaenv/lib/python3.6/site-packages/dask/bytes/local.py in open(self, path, mode, **kwargs)\r\n     61         if path.startswith('file://'):\r\n     62             path = path[len('file://'):]\r\n---> 63         return open(path, mode=mode)\r\n     64 \r\n     65     def ukey(self, path):\r\n\r\nFileNotFoundError: [Errno 2] No such file or directory: '../data/processed/2016*.parq'\r\n```\r\n\r\nThe fact that it's looking for `../data/processed/2016*.parq/_metadata/_metadata` suggests that there is something kinda wonky going on.\r\n\r\nSpecifying the file scheme as hive to the storage engine doesn't seem to resolve the issue, and neither does trying different combos of wildcards and slashes. As a last ditch effort, I tried to pass a list of parquet files into read_parquet and that doesnt' seem to be supported (even just this would be super handy).\r\n\r\nMy workaround is probably going to be to do a delayed for-loop read and concat after that, but I'm not sure how that'd perform in comparison to the other way. Regardless, it would be super nice to have this. Thank you for reading!"},"comment":{"url":"https://api.github.com/repos/dask/dask/issues/comments/391030481","html_url":"https://github.com/dask/dask/issues/2308#issuecomment-391030481","issue_url":"https://api.github.com/repos/dask/dask/issues/2308","id":391030481,"user":{"login":"tribeclickgh","id":28625139,"avatar_url":"https://avatars0.githubusercontent.com/u/28625139?v=4","gravatar_id":"","url":"https://api.github.com/users/tribeclickgh","html_url":"https://github.com/tribeclickgh","followers_url":"https://api.github.com/users/tribeclickgh/followers","following_url":"https://api.github.com/users/tribeclickgh/following{/other_user}","gists_url":"https://api.github.com/users/tribeclickgh/gists{/gist_id}","starred_url":"https://api.github.com/users/tribeclickgh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tribeclickgh/subscriptions","organizations_url":"https://api.github.com/users/tribeclickgh/orgs","repos_url":"https://api.github.com/users/tribeclickgh/repos","events_url":"https://api.github.com/users/tribeclickgh/events{/privacy}","received_events_url":"https://api.github.com/users/tribeclickgh/received_events","type":"User","site_admin":false},"created_at":"2018-05-22T15:18:13Z","updated_at":"2018-05-22T15:18:13Z","author_association":"NONE","body":"hey guys, not sure if this issue is stale, but I am facing a similar issue.\r\n\r\nI am reading from pure json line files in s3, partitioned by year/month/day.\r\n\r\nIm spark (which I am trying to replace) I can read them fairly easily like:\r\n\r\n```\r\nspark.read.json(\"s3://BUCKET/items/\") \r\n```\r\n\r\nAnd that reads all the files that match that wildcard, in particular all of the actual files:\r\n```\r\n\"s3://BUCKET/items/year=2018/month=05/day=14/18232132132.js\"\r\n\"s3://BUCKET/items/year=2018/month=05/day=15/FILENAMES.jl\"\r\n\"s3://BUCKET/items/year=2018/month=05/day=16/FILENAMES.jl\"\r\n\"s3://BUCKET/items/year=2018/month=05/day=17/FILENAMES.jl\"\r\n\"s3://BUCKET/items/year=2018/month=05/day=18/18232132132.js\"\r\n...\r\n```\r\nIs there a way to do this with dask?"}}
{"actor":{"display_login":"tribeclickgh"}}