{"action":"created","issue":{"url":"https://api.github.com/repos/pydata/xarray/issues/2159","repository_url":"https://api.github.com/repos/pydata/xarray","labels_url":"https://api.github.com/repos/pydata/xarray/issues/2159/labels{/name}","comments_url":"https://api.github.com/repos/pydata/xarray/issues/2159/comments","events_url":"https://api.github.com/repos/pydata/xarray/issues/2159/events","html_url":"https://github.com/pydata/xarray/issues/2159","id":324350248,"node_id":"MDU6SXNzdWUzMjQzNTAyNDg=","number":2159,"title":"Concatenate across multiple dimensions with open_mfdataset","user":{"login":"TomNicholas","id":35968931,"node_id":"MDQ6VXNlcjM1OTY4OTMx","avatar_url":"https://avatars2.githubusercontent.com/u/35968931?v=4","gravatar_id":"","url":"https://api.github.com/users/TomNicholas","html_url":"https://github.com/TomNicholas","followers_url":"https://api.github.com/users/TomNicholas/followers","following_url":"https://api.github.com/users/TomNicholas/following{/other_user}","gists_url":"https://api.github.com/users/TomNicholas/gists{/gist_id}","starred_url":"https://api.github.com/users/TomNicholas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/TomNicholas/subscriptions","organizations_url":"https://api.github.com/users/TomNicholas/orgs","repos_url":"https://api.github.com/users/TomNicholas/repos","events_url":"https://api.github.com/users/TomNicholas/events{/privacy}","received_events_url":"https://api.github.com/users/TomNicholas/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":20,"created_at":"2018-05-18T10:10:49Z","updated_at":"2018-11-05T02:03:00Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"#### Code Sample\r\n\r\n```python\r\n# Create 4 datasets containing sections of contiguous (x,y) data\r\nfor i, x in enumerate([1, 3]):\r\n    for j, y in enumerate([10, 40]):\r\n        ds = xr.Dataset({'foo': (('x', 'y'), np.ones((2, 3)))},\r\n                         coords={'x': [x, x+1],\r\n                                 'y': [y, y+10, y+20]})\r\n\r\n        ds.to_netcdf('ds.' + str(i) + str(j) + '.nc')\r\n\r\n# Try to open them all in one go\r\nds_read = xr.open_mfdataset('ds.*.nc')\r\nprint(ds_read)\r\n```\r\n#### Problem description\r\n\r\nCurrently ``xr.open_mfdataset`` will detect a single common dimension and concatenate DataSets along that dimension. However a common use case is a set of NetCDF files which have two or more common dimensions that need to be concatenated along simultaneously (for example collecting the output of any large-scale simulation which parallelizes in more than one dimension simultaneously). For the behaviour of ``xr.open_mfdataset`` to be n-dimensional it should automatically recognise and concatenate along all common dimensions.\r\n\r\n#### Expected Output\r\n```\r\n<xarray.Dataset>\r\nDimensions:  (x: 4, y: 6)\r\nCoordinates:\r\n  * x        (x) int64 1 2 3 4\r\n  * y        (y) int64 10 20 30 40 50 60\r\nData variables:\r\n    foo      (x, y) float64 dask.array<shape=(4, 6), chunksize=(2, 3)>\r\n```\r\n\r\n#### Current output of ``xr.open_mfdataset()``\r\n```\r\n<xarray.Dataset>\r\nDimensions:  (x: 4, y: 12)\r\nCoordinates:\r\n  * x        (x) int64 1 2 3 4\r\n  * y        (y) int64 10 20 30 40 50 60 10 20 30 40 50 60\r\nData variables:\r\n    foo      (x, y) float64 dask.array<shape=(4, 12), chunksize=(4, 3)>\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/pydata/xarray/issues/comments/435733418","html_url":"https://github.com/pydata/xarray/issues/2159#issuecomment-435733418","issue_url":"https://api.github.com/repos/pydata/xarray/issues/2159","id":435733418,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNTczMzQxOA==","user":{"login":"shoyer","id":1217238,"node_id":"MDQ6VXNlcjEyMTcyMzg=","avatar_url":"https://avatars2.githubusercontent.com/u/1217238?v=4","gravatar_id":"","url":"https://api.github.com/users/shoyer","html_url":"https://github.com/shoyer","followers_url":"https://api.github.com/users/shoyer/followers","following_url":"https://api.github.com/users/shoyer/following{/other_user}","gists_url":"https://api.github.com/users/shoyer/gists{/gist_id}","starred_url":"https://api.github.com/users/shoyer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shoyer/subscriptions","organizations_url":"https://api.github.com/users/shoyer/orgs","repos_url":"https://api.github.com/users/shoyer/repos","events_url":"https://api.github.com/users/shoyer/events{/privacy}","received_events_url":"https://api.github.com/users/shoyer/received_events","type":"User","site_admin":false},"created_at":"2018-11-05T02:03:00Z","updated_at":"2018-11-05T02:03:00Z","author_association":"MEMBER","body":"> This is fine though right? We can do all of this, because it should compartmentalise fairly easily shouldn't it? You end up with logic like:\r\n\r\nYes, this seems totally fine to me.\r\n\r\n> We don't need to, but I don't think it would be that hard (if the structure above is feasible), and I think it's a common use case. Also there's an argument for putting in special effort to generalize this function as much as possible, because it lowers the barrier to entry for xarray for new users. Though perhaps I'm just biased because it happens to be my use case...\r\n\r\nSure, no opposition from me if you want to do it! 👍 "}}
{"actor":{"display_login":"shoyer"}}