{"action":"created","issue":{"url":"https://api.github.com/repos/pydata/xarray/issues/1026","repository_url":"https://api.github.com/repos/pydata/xarray","labels_url":"https://api.github.com/repos/pydata/xarray/issues/1026/labels{/name}","comments_url":"https://api.github.com/repos/pydata/xarray/issues/1026/comments","events_url":"https://api.github.com/repos/pydata/xarray/issues/1026/events","html_url":"https://github.com/pydata/xarray/issues/1026","id":180516114,"number":1026,"title":"multidim groupby on dask arrays: dask.array.reshape error","user":{"login":"rabernat","id":1197350,"avatar_url":"https://avatars.githubusercontent.com/u/1197350?v=3","gravatar_id":"","url":"https://api.github.com/users/rabernat","html_url":"https://github.com/rabernat","followers_url":"https://api.github.com/users/rabernat/followers","following_url":"https://api.github.com/users/rabernat/following{/other_user}","gists_url":"https://api.github.com/users/rabernat/gists{/gist_id}","starred_url":"https://api.github.com/users/rabernat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rabernat/subscriptions","organizations_url":"https://api.github.com/users/rabernat/orgs","repos_url":"https://api.github.com/users/rabernat/repos","events_url":"https://api.github.com/users/rabernat/events{/privacy}","received_events_url":"https://api.github.com/users/rabernat/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-10-02T14:55:25Z","updated_at":"2016-10-02T21:38:30Z","closed_at":null,"body":"If I try to run a groupby operation using a multidimensional group, I get an error from dask about \"dask.array.reshape requires that reshaped dimensions after the first contain at most one chunk\".\r\n\r\nThis error is arises with dask 0.11.0 but NOT dask 0.8.0.\r\n\r\nConsider the following test example:\r\n```python\r\nimport dask.array as da\r\nimport xarray as xr\r\n\r\nnz, ny, nx = (10,20,30)\r\ndata = da.ones((nz,ny,nx), chunks=(5,ny,nx))\r\ncoord_2d = da.random.random((ny,nx), chunks=(ny,nx))>0.5\r\nds = xr.Dataset({'thedata': (('z','y','x'), data)},\r\n                coords={'thegroup': (('y','x'), coord_2d)})\r\n# this works fine\r\nds.thedata.groupby('thegroup')\r\n```\r\n\r\nNow I rechunk one of the later dimensions and group again:\r\n```python\r\nds.chunk({'x': 5}).thedata.groupby('thegroup')\r\n```\r\n\r\nThis raises the following error and stack trace\r\n```\r\nValueError                                Traceback (most recent call last)\r\n<ipython-input-16-1b0095ee24a0> in <module>()\r\n----> 1 ds.chunk({'x': 5}).thedata.groupby('thegroup')\r\n\r\n/Users/rpa/RND/open_source/xray/xarray/core/common.pyc in groupby(self, group, squeeze)\r\n    343         if isinstance(group, basestring):\r\n    344             group = self[group]\r\n--> 345         return self.groupby_cls(self, group, squeeze=squeeze)\r\n    346 \r\n    347     def groupby_bins(self, group, bins, right=True, labels=None, precision=3,\r\n\r\n/Users/rpa/RND/open_source/xray/xarray/core/groupby.pyc in __init__(self, obj, group, squeeze, grouper, bins, cut_kwargs)\r\n    170             # the copy is necessary here, otherwise read only array raises error\r\n    171             # in pandas: https://github.com/pydata/pandas/issues/12813>\r\n--> 172             group = group.stack(**{stacked_dim_name: orig_dims}).copy()\r\n    173             obj = obj.stack(**{stacked_dim_name: orig_dims})\r\n    174             self._stacked_dim = stacked_dim_name\r\n\r\n/Users/rpa/RND/open_source/xray/xarray/core/dataarray.pyc in stack(self, **dimensions)\r\n    857         DataArray.unstack\r\n    858         \"\"\"\r\n--> 859         ds = self._to_temp_dataset().stack(**dimensions)\r\n    860         return self._from_temp_dataset(ds)\r\n    861 \r\n\r\n/Users/rpa/RND/open_source/xray/xarray/core/dataset.pyc in stack(self, **dimensions)\r\n   1359         result = self\r\n   1360         for new_dim, dims in dimensions.items():\r\n-> 1361             result = result._stack_once(dims, new_dim)\r\n   1362         return result\r\n   1363 \r\n\r\n/Users/rpa/RND/open_source/xray/xarray/core/dataset.pyc in _stack_once(self, dims, new_dim)\r\n   1322                     shape = [self.dims[d] for d in vdims]\r\n   1323                     exp_var = var.expand_dims(vdims, shape)\r\n-> 1324                     stacked_var = exp_var.stack(**{new_dim: dims})\r\n   1325                     variables[name] = stacked_var\r\n   1326                 else:\r\n\r\n/Users/rpa/RND/open_source/xray/xarray/core/variable.pyc in stack(self, **dimensions)\r\n    801         result = self\r\n    802         for new_dim, dims in dimensions.items():\r\n--> 803             result = result._stack_once(dims, new_dim)\r\n    804         return result\r\n    805 \r\n\r\n/Users/rpa/RND/open_source/xray/xarray/core/variable.pyc in _stack_once(self, dims, new_dim)\r\n    771 \r\n    772         new_shape = reordered.shape[:len(other_dims)] + (-1,)\r\n--> 773         new_data = reordered.data.reshape(new_shape)\r\n    774         new_dims = reordered.dims[:len(other_dims)] + (new_dim,)\r\n    775 \r\n\r\n/Users/rpa/anaconda/lib/python2.7/site-packages/dask/array/core.pyc in reshape(self, *shape)\r\n   1101         if len(shape) == 1 and not isinstance(shape[0], Number):\r\n   1102             shape = shape[0]\r\n-> 1103         return reshape(self, shape)\r\n   1104 \r\n   1105     @wraps(topk)\r\n\r\n/Users/rpa/anaconda/lib/python2.7/site-packages/dask/array/core.pyc in reshape(array, shape)\r\n   2585 \r\n   2586     if any(len(c) != 1 for c in array.chunks[ndim_same+1:]):\r\n-> 2587         raise ValueError('dask.array.reshape requires that reshaped '\r\n   2588                          'dimensions after the first contain at most one chunk')\r\n   2589 \r\n\r\nValueError: dask.array.reshape requires that reshaped dimensions after the first contain at most one chunk\r\n```\r\n\r\nI am using the latest xarray master and dask version 0.11.0. Note that the example works _fine_ if I use an earlier version of dask (e.g. 0.8.0, the only other one I tested.) This suggests an upstream issue with dask, but I wanted to bring it up here first."},"comment":{"url":"https://api.github.com/repos/pydata/xarray/issues/comments/250997873","html_url":"https://github.com/pydata/xarray/issues/1026#issuecomment-250997873","issue_url":"https://api.github.com/repos/pydata/xarray/issues/1026","id":250997873,"user":{"login":"shoyer","id":1217238,"avatar_url":"https://avatars.githubusercontent.com/u/1217238?v=3","gravatar_id":"","url":"https://api.github.com/users/shoyer","html_url":"https://github.com/shoyer","followers_url":"https://api.github.com/users/shoyer/followers","following_url":"https://api.github.com/users/shoyer/following{/other_user}","gists_url":"https://api.github.com/users/shoyer/gists{/gist_id}","starred_url":"https://api.github.com/users/shoyer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shoyer/subscriptions","organizations_url":"https://api.github.com/users/shoyer/orgs","repos_url":"https://api.github.com/users/shoyer/repos","events_url":"https://api.github.com/users/shoyer/events{/privacy}","received_events_url":"https://api.github.com/users/shoyer/received_events","type":"User","site_admin":false},"created_at":"2016-10-02T21:38:30Z","updated_at":"2016-10-02T21:38:30Z","body":"It would look something like this: \r\n\r\n1. Verify that `chunks` are the same on all dask arrays to be stacked.\r\n2. Use `np.ravel` with `map_blocks` to flatten each block independently.\r\n3. Construct the appropriate (non-sorted) MultiIndex to label the flattened elements."}}
{"actor":{"display_login":"shoyer"}}