{"action":"created","issue":{"url":"https://api.github.com/repos/dask/dask/issues/111","repository_url":"https://api.github.com/repos/dask/dask","labels_url":"https://api.github.com/repos/dask/dask/issues/111/labels{/name}","comments_url":"https://api.github.com/repos/dask/dask/issues/111/comments","events_url":"https://api.github.com/repos/dask/dask/issues/111/events","html_url":"https://github.com/dask/dask/issues/111","id":65106214,"node_id":"MDU6SXNzdWU2NTEwNjIxNA==","number":111,"title":"Rules for automatic reblocking","user":{"login":"shoyer","id":1217238,"node_id":"MDQ6VXNlcjEyMTcyMzg=","avatar_url":"https://avatars2.githubusercontent.com/u/1217238?v=4","gravatar_id":"","url":"https://api.github.com/users/shoyer","html_url":"https://github.com/shoyer","followers_url":"https://api.github.com/users/shoyer/followers","following_url":"https://api.github.com/users/shoyer/following{/other_user}","gists_url":"https://api.github.com/users/shoyer/gists{/gist_id}","starred_url":"https://api.github.com/users/shoyer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shoyer/subscriptions","organizations_url":"https://api.github.com/users/shoyer/orgs","repos_url":"https://api.github.com/users/shoyer/repos","events_url":"https://api.github.com/users/shoyer/events{/privacy}","received_events_url":"https://api.github.com/users/shoyer/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2015-03-29T23:16:44Z","updated_at":"2018-06-29T13:58:10Z","closed_at":"2018-06-29T13:58:10Z","author_association":"MEMBER","body":"xref #58, #60\n\nAutomatic reblocking in dask operations will be a super useful follow-on to manual `reblock` functionality. Here are my initial thoughts. For now, I'm going to simplify things by assuming that each array has a fixed blockshape (in practice, we could use the mean of the blockdims along each axis).\n1. For operations where one argument has a strictly larger blockshape (e.g., `(100, 100)` and `(1000, 1000)`), we probably want to reblock to the _larger_ blockshape to minimize growth of the scheduling graph.\n2. For operations with differently shaped blockshapes (e.g., `(100, 10000)` and `(10000, 100)`) but comparable chunk sizes let's consider using the _geometric mean_ of the blockshape along each dimension. This has the nice property of conserving the size of each chunk, and I suspect it would also minimize total amount of concat/slicing work necessary for the reblocking itself. For this example, this yields a blockshape of `(1000, 1000)`.\n3. When given arrays with block shapes `(10000, 1)` and `(1, 10000)`, we probably don't want chunks of shape `(100, 100)` -- that's too small. So shapes calculated from the geometric mean should probably be scaled up to ensure some _minimum size_ (e.g., one million elements, or the largest chunk size currently in the dask).\n\n\"Uncut\" dimensions deserve special consideration, because they are quite likely to arise in practice, both from array broadcasting (they'll have length 1) and from coercing fresh numpy arrays. But I think they are covered reasonably well by these rules.\n\nFor example, consider multiplying arrays with shapes/blockshapes `(100, 1, 1)` and `(1, 1000, 1000)`. With the geometric mean, we get about 10000 chunks of shape `(10, 32, 32)`, which after the scaling adjustment leaves us with about 100 chunks of size `(47, 147, 147)`.\n"},"comment":{"url":"https://api.github.com/repos/dask/dask/issues/comments/401362415","html_url":"https://github.com/dask/dask/issues/111#issuecomment-401362415","issue_url":"https://api.github.com/repos/dask/dask/issues/111","id":401362415,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMTM2MjQxNQ==","user":{"login":"mrocklin","id":306380,"node_id":"MDQ6VXNlcjMwNjM4MA==","avatar_url":"https://avatars3.githubusercontent.com/u/306380?v=4","gravatar_id":"","url":"https://api.github.com/users/mrocklin","html_url":"https://github.com/mrocklin","followers_url":"https://api.github.com/users/mrocklin/followers","following_url":"https://api.github.com/users/mrocklin/following{/other_user}","gists_url":"https://api.github.com/users/mrocklin/gists{/gist_id}","starred_url":"https://api.github.com/users/mrocklin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrocklin/subscriptions","organizations_url":"https://api.github.com/users/mrocklin/orgs","repos_url":"https://api.github.com/users/mrocklin/repos","events_url":"https://api.github.com/users/mrocklin/events{/privacy}","received_events_url":"https://api.github.com/users/mrocklin/received_events","type":"User","site_admin":false},"created_at":"2018-06-29T13:58:10Z","updated_at":"2018-06-29T13:58:10Z","author_association":"MEMBER","body":"I think that rechunking will have to be done on a per-operation basis.  The rules for tensordot will be different from the rules for fft.  \r\n\r\nThis is discussed in https://github.com/dask/dask/issues/3587 .  Closing in favor of that issue.\r\n"}}
{"actor":{"display_login":"mrocklin"}}