{"action":"created","issue":{"url":"https://api.github.com/repos/dask/dask/issues/1720","repository_url":"https://api.github.com/repos/dask/dask","labels_url":"https://api.github.com/repos/dask/dask/issues/1720/labels{/name}","comments_url":"https://api.github.com/repos/dask/dask/issues/1720/comments","events_url":"https://api.github.com/repos/dask/dask/issues/1720/events","html_url":"https://github.com/dask/dask/issues/1720","id":185904277,"number":1720,"title":"Random access to dask array spanning several HDF5 files sometimes leads to IOErrors.","user":{"login":"axeltidemann","id":6352118,"avatar_url":"https://avatars.githubusercontent.com/u/6352118?v=3","gravatar_id":"","url":"https://api.github.com/users/axeltidemann","html_url":"https://github.com/axeltidemann","followers_url":"https://api.github.com/users/axeltidemann/followers","following_url":"https://api.github.com/users/axeltidemann/following{/other_user}","gists_url":"https://api.github.com/users/axeltidemann/gists{/gist_id}","starred_url":"https://api.github.com/users/axeltidemann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/axeltidemann/subscriptions","organizations_url":"https://api.github.com/users/axeltidemann/orgs","repos_url":"https://api.github.com/users/axeltidemann/repos","events_url":"https://api.github.com/users/axeltidemann/events{/privacy}","received_events_url":"https://api.github.com/users/axeltidemann/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-10-28T11:51:38Z","updated_at":"2016-10-28T12:05:49Z","closed_at":null,"body":"I create a dask array out of several HDF5 files, but I find that some indices lead to an `IOError: Can't read data (Can't open directory)`. The indices should be continuous, no? I attach an example. The creation of HDF5 files mimic the ones I am working on. \r\n\r\n```python \r\nimport glob\r\nimport string\r\nimport random\r\n\r\nimport numpy as np\r\nimport pandas as pd\r\nimport dask.array as da\r\nimport h5py\r\n\r\nshapes = [8000, 839, 8000]\r\nwidth = 2048 # The length of vectors I am working with                                                                                                                               \r\n\r\nfor i, shape in enumerate(shapes):\r\n    h5name = '{}.h5'.format(i)\r\n    with pd.HDFStore(h5name, mode='w', complevel=9, complib='blosc') as store:\r\n        for _ in range(3): # To simulate chunk writes                                                                                                                                \r\n            X = np.random.random((shape, width))\r\n\r\n            # To allow columns this wide, they must be strings, not integers                                                                                                         \r\n            columns = [ 'f{}'.format(i) for i in range(X.shape[1]) ]\r\n\r\n            # String indexes of varying length                                                                                                                                       \r\n            index = [ ''.join(random.choice(string.ascii_uppercase + string.digits)\r\n                              for _ in range(np.random.randint(20,51)))\r\n                      for _ in range(shape) ]\r\n\r\n            df = pd.DataFrame(data=X, index=index, columns=columns)\r\n            df.index.name='filename'\r\n            store.append('data', df)\r\n\r\nh5_files = glob.glob('*.h5')\r\n\r\ndatasets = [ h5py.File(fn)['/data/table'] for fn in h5_files ]\r\narrays = [ da.from_array(data, chunks=1000) for data in datasets ]\r\nX = da.concatenate(arrays, axis=0)\r\n\r\nprint X # Everything seems fine                                                                                                                                                      \r\n\r\ntrouble = X[2515] # This value I found by trial and error                                                                                                                            \r\nprint trouble # Here we can see that the shape is (), so clearly something is wrong\r\nprint trouble.compute() # IOError                                                                                                                                   \r\n```"},"comment":{"url":"https://api.github.com/repos/dask/dask/issues/comments/256903831","html_url":"https://github.com/dask/dask/issues/1720#issuecomment-256903831","issue_url":"https://api.github.com/repos/dask/dask/issues/1720","id":256903831,"user":{"login":"mrocklin","id":306380,"avatar_url":"https://avatars.githubusercontent.com/u/306380?v=3","gravatar_id":"","url":"https://api.github.com/users/mrocklin","html_url":"https://github.com/mrocklin","followers_url":"https://api.github.com/users/mrocklin/followers","following_url":"https://api.github.com/users/mrocklin/following{/other_user}","gists_url":"https://api.github.com/users/mrocklin/gists{/gist_id}","starred_url":"https://api.github.com/users/mrocklin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrocklin/subscriptions","organizations_url":"https://api.github.com/users/mrocklin/orgs","repos_url":"https://api.github.com/users/mrocklin/repos","events_url":"https://api.github.com/users/mrocklin/events{/privacy}","received_events_url":"https://api.github.com/users/mrocklin/received_events","type":"User","site_admin":false},"created_at":"2016-10-28T12:05:49Z","updated_at":"2016-10-28T12:05:49Z","body":"Hrm, unfortunately this is working fine when I try it out locally."}}
{"actor":{"display_login":"mrocklin"}}