{"action":"created","issue":{"url":"https://api.github.com/repos/pydata/xarray/issues/798","repository_url":"https://api.github.com/repos/pydata/xarray","labels_url":"https://api.github.com/repos/pydata/xarray/issues/798/labels{/name}","comments_url":"https://api.github.com/repos/pydata/xarray/issues/798/comments","events_url":"https://api.github.com/repos/pydata/xarray/issues/798/events","html_url":"https://github.com/pydata/xarray/issues/798","id":142498006,"number":798,"title":"Integration  with dask/distributed (xarray backend design)","user":{"login":"pwolfram","id":4295853,"avatar_url":"https://avatars.githubusercontent.com/u/4295853?v=3","gravatar_id":"","url":"https://api.github.com/users/pwolfram","html_url":"https://github.com/pwolfram","followers_url":"https://api.github.com/users/pwolfram/followers","following_url":"https://api.github.com/users/pwolfram/following{/other_user}","gists_url":"https://api.github.com/users/pwolfram/gists{/gist_id}","starred_url":"https://api.github.com/users/pwolfram/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pwolfram/subscriptions","organizations_url":"https://api.github.com/users/pwolfram/orgs","repos_url":"https://api.github.com/users/pwolfram/repos","events_url":"https://api.github.com/users/pwolfram/events{/privacy}","received_events_url":"https://api.github.com/users/pwolfram/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":19,"created_at":"2016-03-21T23:18:02Z","updated_at":"2016-04-04T20:40:58Z","closed_at":null,"body":"Dask (https://github.com/dask/dask) currently provides on-node parallelism for medium-size data problems.  However, large climate data sets will require multiple-node parallelism to analyze large climate data sets because this constitutes a big data problem.  A likely solution to this issue is integration of distributed (https://github.com/dask/distributed) with dask.  Distributed is now integrated with dask and its benefits are already starting to be realized, e.g., see http://matthewrocklin.com/blog/work/2016/02/26/dask-distributed-part-3.\r\n\r\nThus, this issue is designed to identify the steps needed to perform this integration, at a high-level. As stated by @shoyer, it will \r\n\r\n\r\n> definitely require some refactoring of the xarray backend system to make this work cleanly, but that's \r\n> OK -- the xarray backend system is indicated as experimental/internal API precisely because we \r\n> hadn't figured out all the use cases yet.\"\r\n\r\n> To be honest, I've never been entirely happy with the design we took there (we use inheritance rather \r\n> than composition for backend classes), but we did get it to work for our use cases. Some refactoring \r\n> with an eye towards compatibility with dask distributed seems like a very worthwhile endeavor. We \r\n> do have the benefit of a pretty large test suite covering existing use cases.\r\n\r\nThus, we have the chance to make xarray big-data capable as well as provide improvements to the backend.\r\n\r\nTo this end, I'm starting this issue to help begin the design process following the xarray mailing list discussion some of us have been having (@shoyer, @mrocklin, @rabernat).\r\n\r\nTask To Do List:\r\n\r\n- [ ] Verify asynchronous access error for `to_netcdf` output is resolved (e.g., https://github.com/pydata/xarray/issues/793)\r\n- [ ] LRU-cached file IO supporting serialization to robustly support HDF/NetCDF reads"},"comment":{"url":"https://api.github.com/repos/pydata/xarray/issues/comments/205484614","html_url":"https://github.com/pydata/xarray/issues/798#issuecomment-205484614","issue_url":"https://api.github.com/repos/pydata/xarray/issues/798","id":205484614,"user":{"login":"shoyer","id":1217238,"avatar_url":"https://avatars.githubusercontent.com/u/1217238?v=3","gravatar_id":"","url":"https://api.github.com/users/shoyer","html_url":"https://github.com/shoyer","followers_url":"https://api.github.com/users/shoyer/followers","following_url":"https://api.github.com/users/shoyer/following{/other_user}","gists_url":"https://api.github.com/users/shoyer/gists{/gist_id}","starred_url":"https://api.github.com/users/shoyer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shoyer/subscriptions","organizations_url":"https://api.github.com/users/shoyer/orgs","repos_url":"https://api.github.com/users/shoyer/repos","events_url":"https://api.github.com/users/shoyer/events{/privacy}","received_events_url":"https://api.github.com/users/shoyer/received_events","type":"User","site_admin":false},"created_at":"2016-04-04T20:40:58Z","updated_at":"2016-04-04T20:40:58Z","body":"@pwolfram I was referring to [this comment](https://github.com/pydata/xarray/issues/798#issuecomment-199545836) for @mrocklin's `netCDF4_Dataset`."}}