{"action":"created","issue":{"url":"https://api.github.com/repos/dask/dask/issues/1803","repository_url":"https://api.github.com/repos/dask/dask","labels_url":"https://api.github.com/repos/dask/dask/issues/1803/labels{/name}","comments_url":"https://api.github.com/repos/dask/dask/issues/1803/comments","events_url":"https://api.github.com/repos/dask/dask/issues/1803/events","html_url":"https://github.com/dask/dask/pull/1803","id":190849981,"number":1803,"title":"Optimise read_hdf","user":{"login":"chrisburr","id":5220533,"avatar_url":"https://avatars.githubusercontent.com/u/5220533?v=3","gravatar_id":"","url":"https://api.github.com/users/chrisburr","html_url":"https://github.com/chrisburr","followers_url":"https://api.github.com/users/chrisburr/followers","following_url":"https://api.github.com/users/chrisburr/following{/other_user}","gists_url":"https://api.github.com/users/chrisburr/gists{/gist_id}","starred_url":"https://api.github.com/users/chrisburr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chrisburr/subscriptions","organizations_url":"https://api.github.com/users/chrisburr/orgs","repos_url":"https://api.github.com/users/chrisburr/repos","events_url":"https://api.github.com/users/chrisburr/events{/privacy}","received_events_url":"https://api.github.com/users/chrisburr/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-11-21T22:08:42Z","updated_at":"2016-11-22T01:45:09Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/dask/dask/pulls/1803","html_url":"https://github.com/dask/dask/pull/1803","diff_url":"https://github.com/dask/dask/pull/1803.diff","patch_url":"https://github.com/dask/dask/pull/1803.patch"},"body":"This PR helps mitigate the problem described in #1353 where `read_hdf` is very slow when reading from many files/files with many tables. The solution proposed here is comprised of two (and a half) parts:\r\n\r\n1. Only load the keys from the hdf file if the key contains one of the special characters (as defined in [fnmatch](https://docs.python.org/3.5/library/fnmatch.html)).\r\n2. Use `delayed`/`compute` to allow multiple files to be read simultaneously before concatenating.\r\n3. @mrocklin suggested using `h5py` to access the keys within each file, the snippet below results in a 6x speed improvement over just using `HDFStore.keys` on file with ~30 tables. Though, I haven't included this as as it appears to result in a deadlock when used with the threaded scheduler. I also don't know if this is sufficiently general for all use cases.\r\n\r\n```python\r\n# Replace \"keys = [k for k in hdf.keys() if fnmatch(k, key)]\" with\r\ndef get_keys(key, obj):\r\n    # If the object corresponding to this key is a `Dataset` and the key ends in\r\n    # table then the pytables key is contained within `key` as follows:\r\n    # key = \"/{pytables_key}/table\"\r\n    if isinstance(obj, h5py._hl.dataset.Dataset) and key.split('/')[-1] == 'table':\r\n        get_keys.result.append('/' + key[:-len('/table')])\r\n\r\nget_keys.result = []\r\nwith h5py.File(path, mode='r') as h5py_f:\r\n    h5py_f.visititems(get_keys)\r\n\r\nkeys = [k for k in get_keys.result if fnmatch(k, key)]\r\n```"},"comment":{"url":"https://api.github.com/repos/dask/dask/issues/comments/262123645","html_url":"https://github.com/dask/dask/pull/1803#issuecomment-262123645","issue_url":"https://api.github.com/repos/dask/dask/issues/1803","id":262123645,"user":{"login":"TomAugspurger","id":1312546,"avatar_url":"https://avatars.githubusercontent.com/u/1312546?v=3","gravatar_id":"","url":"https://api.github.com/users/TomAugspurger","html_url":"https://github.com/TomAugspurger","followers_url":"https://api.github.com/users/TomAugspurger/followers","following_url":"https://api.github.com/users/TomAugspurger/following{/other_user}","gists_url":"https://api.github.com/users/TomAugspurger/gists{/gist_id}","starred_url":"https://api.github.com/users/TomAugspurger/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/TomAugspurger/subscriptions","organizations_url":"https://api.github.com/users/TomAugspurger/orgs","repos_url":"https://api.github.com/users/TomAugspurger/repos","events_url":"https://api.github.com/users/TomAugspurger/events{/privacy}","received_events_url":"https://api.github.com/users/TomAugspurger/received_events","type":"User","site_admin":false},"created_at":"2016-11-22T01:45:09Z","updated_at":"2016-11-22T01:45:09Z","body":"I wonder if the performance of `HDFStore.keys()` could be improved upstream in pandas or pytables? This is a breakdown for `HDFStore.groups()`, which is what `HDFStore.keys()` spends all its time in.\r\n\r\n```\r\n  1077                                               def groups(self):\r\n  1078                                                   \"\"\"return a list of all the top-level nodes (that are not themselves a\r\n  1079                                                   pandas storage object)\r\n  1080                                                   \"\"\"\r\n  1081         1           18     18.0      0.0          _tables()\r\n  1082         1           33     33.0      0.0          self._check_if_open()\r\n  1083                                                   return [\r\n  1084         1      2309128 2309128.0    100.0              g for g in self._handle.walk_nodes()\r\n  1085                                                       if (getattr(g._v_attrs, 'pandas_type', None) or\r\n  1086                                                           getattr(g, 'table', None) or\r\n  1087                                                           (isinstance(g, _table_mod.table.Table) and\r\n  1088                                                            g._v_name != u('table')))\r\n  1089                                                   ]\r\n```\r\n\r\n`_handle` is a `tables.file.File` so we would need to either speed that up, or find a faster alternative to use in pandas."}}
{"actor":{"display_login":"TomAugspurger"}}