{"action":"created","issue":{"url":"https://api.github.com/repos/pydata/xarray/issues/2298","repository_url":"https://api.github.com/repos/pydata/xarray","labels_url":"https://api.github.com/repos/pydata/xarray/issues/2298/labels{/name}","comments_url":"https://api.github.com/repos/pydata/xarray/issues/2298/comments","events_url":"https://api.github.com/repos/pydata/xarray/issues/2298/events","html_url":"https://github.com/pydata/xarray/issues/2298","id":342180429,"node_id":"MDU6SXNzdWUzNDIxODA0Mjk=","number":2298,"title":"Making xarray math lazy","user":{"login":"shoyer","id":1217238,"node_id":"MDQ6VXNlcjEyMTcyMzg=","avatar_url":"https://avatars2.githubusercontent.com/u/1217238?v=4","gravatar_id":"","url":"https://api.github.com/users/shoyer","html_url":"https://github.com/shoyer","followers_url":"https://api.github.com/users/shoyer/followers","following_url":"https://api.github.com/users/shoyer/following{/other_user}","gists_url":"https://api.github.com/users/shoyer/gists{/gist_id}","starred_url":"https://api.github.com/users/shoyer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shoyer/subscriptions","organizations_url":"https://api.github.com/users/shoyer/orgs","repos_url":"https://api.github.com/users/shoyer/repos","events_url":"https://api.github.com/users/shoyer/events{/privacy}","received_events_url":"https://api.github.com/users/shoyer/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-07-18T05:18:53Z","updated_at":"2018-07-20T00:02:34Z","closed_at":null,"author_association":"MEMBER","body":"At SciPy, I had the realization that it would be relatively straightforward to make element-wise math between xarray objects lazy. This would let us support lazy coordinate arrays, a feature that has quite a few use-cases, e.g., for both geoscience and astronomy.\r\n\r\nThe trick would be to write a lazy array class that holds an element-wise vectorized function and passes indexers on to its arguments. I haven't thought too hard about this yet for vectorized indexing, but it could be quite efficient for outer indexing. I have some prototype code but no tests yet.\r\n\r\nThe question is how to hook this into xarray operations. In particular, supposing that the inputs to a function do no hold dask arrays:\r\n- Should we try to make *every* element-wise operation with vectorized functions (ufuncs) lazy by default? This might have negative performance implications and would be a little tricky to implement with xarray's current code, since we still implement binary operations like `+` with separate logic from `apply_ufunc`.\r\n- Should we make every element-wise operation that explicitly uses `apply_ufunc()` lazy by default?\r\n- Or should we only make element-wise operations lazy with `apply_ufunc()` if you use some special flag, e.g., `apply_ufunc(..., lazy=True)`?\r\n\r\nI am leaning towards the last option for now but would welcome other opinions."},"comment":{"url":"https://api.github.com/repos/pydata/xarray/issues/comments/406449411","html_url":"https://github.com/pydata/xarray/issues/2298#issuecomment-406449411","issue_url":"https://api.github.com/repos/pydata/xarray/issues/2298","id":406449411,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNjQ0OTQxMQ==","user":{"login":"shoyer","id":1217238,"node_id":"MDQ6VXNlcjEyMTcyMzg=","avatar_url":"https://avatars2.githubusercontent.com/u/1217238?v=4","gravatar_id":"","url":"https://api.github.com/users/shoyer","html_url":"https://github.com/shoyer","followers_url":"https://api.github.com/users/shoyer/followers","following_url":"https://api.github.com/users/shoyer/following{/other_user}","gists_url":"https://api.github.com/users/shoyer/gists{/gist_id}","starred_url":"https://api.github.com/users/shoyer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shoyer/subscriptions","organizations_url":"https://api.github.com/users/shoyer/orgs","repos_url":"https://api.github.com/users/shoyer/repos","events_url":"https://api.github.com/users/shoyer/events{/privacy}","received_events_url":"https://api.github.com/users/shoyer/received_events","type":"User","site_admin":false},"created_at":"2018-07-20T00:02:34Z","updated_at":"2018-07-20T00:02:34Z","author_association":"MEMBER","body":"> Therefore, personally, I'd like to see this lazy math by implementing a lazy array.\r\n> The API I thought of is .to_lazy() which converts the backend to the lazy array,\r\n> as similar to that .chunk() converts the backend to dask array.\r\n\r\nThis is not a bad idea, but the version of lazy arithmetic that I have been contemplating (see https://github.com/pydata/xarray/pull/2302) is not yet complete. For example, it doesn't have any way to represent a lazy aggregation."}}
{"actor":{"display_login":"shoyer"}}