{"action":"created","issue":{"url":"https://api.github.com/repos/dask/dask/issues/3648","repository_url":"https://api.github.com/repos/dask/dask","labels_url":"https://api.github.com/repos/dask/dask/issues/3648/labels{/name}","comments_url":"https://api.github.com/repos/dask/dask/issues/3648/comments","events_url":"https://api.github.com/repos/dask/dask/issues/3648/events","html_url":"https://github.com/dask/dask/pull/3648","id":334185653,"node_id":"MDExOlB1bGxSZXF1ZXN0MTk2MjIzNDc2","number":3648,"title":"Break apart uneven array-of-int slicing to separate chunks","user":{"login":"mrocklin","id":306380,"node_id":"MDQ6VXNlcjMwNjM4MA==","avatar_url":"https://avatars3.githubusercontent.com/u/306380?v=4","gravatar_id":"","url":"https://api.github.com/users/mrocklin","html_url":"https://github.com/mrocklin","followers_url":"https://api.github.com/users/mrocklin/followers","following_url":"https://api.github.com/users/mrocklin/following{/other_user}","gists_url":"https://api.github.com/users/mrocklin/gists{/gist_id}","starred_url":"https://api.github.com/users/mrocklin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrocklin/subscriptions","organizations_url":"https://api.github.com/users/mrocklin/orgs","repos_url":"https://api.github.com/users/mrocklin/repos","events_url":"https://api.github.com/users/mrocklin/events{/privacy}","received_events_url":"https://api.github.com/users/mrocklin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2018-06-20T17:48:03Z","updated_at":"2018-06-21T12:19:23Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/dask/dask/pulls/3648","html_url":"https://github.com/dask/dask/pull/3648","diff_url":"https://github.com/dask/dask/pull/3648.diff","patch_url":"https://github.com/dask/dask/pull/3648.patch"},"body":"Previously when slicing into a dask array with a list/array of integers we would do one of two things:\r\n\r\n### Sorted, respect chunking\r\n\r\nIf the array was sorted then it would return an array that was chunked similarly to the input\r\n\r\n```python\r\nimport dask.array as da\r\nx = da.ones((20, 20), chunks=(4, 5))\r\nx.chunks\r\n# ((4, 4, 4, 4, 4), (5, 5, 5, 5))\r\n\r\nimport numpy as np\r\n\r\nx[np.arange(20), :].chunks\r\n# ((4, 4, 4, 4, 4), (5, 5, 5, 5))\r\n\r\nx[np.arange(20) // 2, :].chunks\r\n# ((8, 8, 4), (5, 5, 5, 5))\r\n```\r\n\r\n### Unsorted, single chunk\r\n\r\nHowever if the index wasn't sorted then it would put everything into one big chunk\r\n\r\n```python\r\nindex = np.arange(20) % 10\r\n# array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9])\r\n\r\nx[index, :].chunks\r\n# ((20,), (5, 5, 5, 5))\r\n```\r\n\r\n### Now\r\n\r\nNow, we pick out different pieces of the index and use them to slice into the dask array.  \r\n\r\n```python\r\nx = da.ones((20, 20), chunks=(4, 5))\r\nindex = np.arange(20) % 10\r\n# array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9])\r\n\r\nx[index].chunks\r\n# Before: ((20,), (5, 5, 5, 5))\r\n# Now: ((4, 4, 2, 4, 4, 2), (5, 5, 5, 5))\r\n```\r\n\r\nThis is good because it allows more granular chunking when slicing with a semi-sorted index, but is bad in that it can easily create very many chunks in pathological situations, like indexing with a random index.\r\n\r\ncc @shoyer @rabernat @jakirkham \r\n\r\nFollows on discussion from https://github.com/pydata/xarray/issues/2237"},"comment":{"url":"https://api.github.com/repos/dask/dask/issues/comments/399084031","html_url":"https://github.com/dask/dask/pull/3648#issuecomment-399084031","issue_url":"https://api.github.com/repos/dask/dask/issues/3648","id":399084031,"node_id":"MDEyOklzc3VlQ29tbWVudDM5OTA4NDAzMQ==","user":{"login":"mrocklin","id":306380,"node_id":"MDQ6VXNlcjMwNjM4MA==","avatar_url":"https://avatars3.githubusercontent.com/u/306380?v=4","gravatar_id":"","url":"https://api.github.com/users/mrocklin","html_url":"https://github.com/mrocklin","followers_url":"https://api.github.com/users/mrocklin/followers","following_url":"https://api.github.com/users/mrocklin/following{/other_user}","gists_url":"https://api.github.com/users/mrocklin/gists{/gist_id}","starred_url":"https://api.github.com/users/mrocklin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrocklin/subscriptions","organizations_url":"https://api.github.com/users/mrocklin/orgs","repos_url":"https://api.github.com/users/mrocklin/repos","events_url":"https://api.github.com/users/mrocklin/events{/privacy}","received_events_url":"https://api.github.com/users/mrocklin/received_events","type":"User","site_admin":false},"created_at":"2018-06-21T12:19:23Z","updated_at":"2018-06-21T12:19:23Z","author_association":"MEMBER","body":"Also @rabernat it would be good if you could try this out on a semi-realistic problem and see what the performance is like."}}
{"actor":{"display_login":"mrocklin"}}