{"action":"created","issue":{"url":"https://api.github.com/repos/pydata/xarray/issues/818","repository_url":"https://api.github.com/repos/pydata/xarray","labels_url":"https://api.github.com/repos/pydata/xarray/issues/818/labels{/name}","comments_url":"https://api.github.com/repos/pydata/xarray/issues/818/comments","events_url":"https://api.github.com/repos/pydata/xarray/issues/818/events","html_url":"https://github.com/pydata/xarray/pull/818","id":146182176,"number":818,"title":"Multidimensional groupby","user":{"login":"rabernat","id":1197350,"avatar_url":"https://avatars.githubusercontent.com/u/1197350?v=3","gravatar_id":"","url":"https://api.github.com/users/rabernat","html_url":"https://github.com/rabernat","followers_url":"https://api.github.com/users/rabernat/followers","following_url":"https://api.github.com/users/rabernat/following{/other_user}","gists_url":"https://api.github.com/users/rabernat/gists{/gist_id}","starred_url":"https://api.github.com/users/rabernat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rabernat/subscriptions","organizations_url":"https://api.github.com/users/rabernat/orgs","repos_url":"https://api.github.com/users/rabernat/repos","events_url":"https://api.github.com/users/rabernat/events{/privacy}","received_events_url":"https://api.github.com/users/rabernat/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":26,"created_at":"2016-04-06T04:14:37Z","updated_at":"2016-05-12T05:27:11Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/pydata/xarray/pulls/818","html_url":"https://github.com/pydata/xarray/pull/818","diff_url":"https://github.com/pydata/xarray/pull/818.diff","patch_url":"https://github.com/pydata/xarray/pull/818.patch"},"body":"Many datasets have a two dimensional coordinate variable (e.g. longitude) which is different from the logical grid coordinates (e.g. nx, ny). (See #605.) For plotting purposes, this is solved by #608. However, we still might want to split / apply / combine over such coordinates. That has not been possible, because groupby only supports creating groups on one-dimensional arrays.\r\n\r\nThis PR overcomes that issue by using ``stack`` to collapse multiple dimensions in the group variable. A minimal example of the new functionality is\r\n```python\r\n>>> da = xr.DataArray([[0,1],[2,3]], \r\n                coords={'lon': (['ny','nx'], [[30,40],[40,50]] ),\r\n                        'lat': (['ny','nx'], [[10,10],[20,20]] )},\r\n                dims=['ny','nx'])\r\n>>> da.groupby('lon').sum()\r\n<xarray.DataArray (lon: 3)>\r\narray([0, 3, 3])\r\nCoordinates:\r\n  * lon      (lon) int64 30 40 50\r\n```\r\n\r\nThis feature could have broad applicability for many realistic datasets (particularly model output on irregular grids): for example, averaging non-rectangular grids zonally (i.e. in latitude), binning in temperature, etc.\r\n\r\nIf you think this is worth pursuing, I would love some feedback.\r\n\r\nThe PR is not complete. Some items to address are\r\n- [ ] Create a specialized grouper to allow coarser bins. By default, if no ``grouper`` is specified, the ``GroupBy`` object uses all unique values to define the groups. With a high resolution dataset, this could balloon to a huge number of groups. With the latitude example, we would like to be able to specify e.g. 1-degree bins. Usage would be ``da.groupby('lon', bins=range(-90,90))``.\r\n- [ ] Allow specification of which dims to stack. For example, stack in space but keep time dimension intact. (Currently it just stacks all the dimensions of the group variable.) \r\n- [ ] A nice example for the docs.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/pydata/xarray/issues/comments/218663446","html_url":"https://github.com/pydata/xarray/pull/818#issuecomment-218663446","issue_url":"https://api.github.com/repos/pydata/xarray/issues/818","id":218663446,"user":{"login":"shoyer","id":1217238,"avatar_url":"https://avatars.githubusercontent.com/u/1217238?v=3","gravatar_id":"","url":"https://api.github.com/users/shoyer","html_url":"https://github.com/shoyer","followers_url":"https://api.github.com/users/shoyer/followers","following_url":"https://api.github.com/users/shoyer/following{/other_user}","gists_url":"https://api.github.com/users/shoyer/gists{/gist_id}","starred_url":"https://api.github.com/users/shoyer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shoyer/subscriptions","organizations_url":"https://api.github.com/users/shoyer/orgs","repos_url":"https://api.github.com/users/shoyer/repos","events_url":"https://api.github.com/users/shoyer/events{/privacy}","received_events_url":"https://api.github.com/users/shoyer/received_events","type":"User","site_admin":false},"created_at":"2016-05-12T05:27:11Z","updated_at":"2016-05-12T05:27:11Z","body":"@naught101 I would consider changing:\r\n```\r\nforcing_data.isel(lat=lat, lon=lon)\r\n                            .to_dataframe()\r\n                            .drop(['lat', 'lon'], axis=1)\r\n```\r\nto something more like\r\n```\r\nnp.stack([forcing_data.isel(lat=lat, lon=lon).values, times])\r\n```\r\nbecause there's no point in creating a DataFrame with a bunch of variables you wouldn't use -- pandas will be pretty wasteful in allocating this.\r\n\r\nOtherwise that looks pretty reasonable, given the limitations of current groupby support. Now, ideally you could write something like instead:\r\n```\r\ndef make_prediction(model, forcing_data_time_series, flux_vars, time):\r\n    predicted_values = model.predict(forcing_data_time_series.values)\r\n    return xr.DataArray(predicted_values, [flux_vars, time])\r\n\r\nforcing_data.groupby(['lat', 'lon']).dask_apply(make_prediction)\r\n```\r\nThis would two the 2D groupby, and then apply the predict function in parallel with dask. Sadly we don't have this feature yet, though :)."}}