{"action":"created","comment":{"url":"https://api.github.com/repos/dask/dask/pulls/comments/171687369","pull_request_review_id":100585483,"id":171687369,"diff_hunk":"@@ -3493,6 +3498,72 @@ def _vindex(x, *indexes):\n     return result_1d.reshape(broadcast_shape + result_1d.shape[1:])\n \n \n+def _vindex_dask_arrays(x, *indices):\n+    \"\"\"Point wise indexing with Dask Arrays\"\"\"\n+\n+    from .creation import arange\n+\n+    x = asarray(x)\n+\n+    oth_inds = []\n+    arr_inds = []\n+    out_axes = []\n+    for i, ind in enumerate(indices):\n+        if isinstance(ind, Array):\n+            oth_inds.append(slice(None))\n+            arr_inds.append(asarray(ind))\n+            if 0 not in out_axes:\n+                out_axes.append(0)\n+        elif isinstance(ind, Integral):\n+            oth_inds.append(ind)\n+        else:\n+            oth_inds.append(ind)\n+            arr_inds.append(slice(None))\n+            out_axes.append(len(arr_inds))\n+    oth_inds = tuple(oth_inds)\n+    arr_inds = tuple(arr_inds)\n+    out_axes = tuple(out_axes)\n+\n+    if not all(o == slice(None) for o in oth_inds):\n+        x = x.vindex[oth_inds]\n+\n+    x_inds = tuple(\n+        arange(s, chunks=c) for s, c in zip(x.shape, x.chunks)\n+    )\n+\n+    idx_vals = atop(\n+        _vindex_dask_arrays_chunk,\n+        out_axes,\n+        x, tuple(range(1, 1 + x.ndim)),\n+        *concat([\n+            (e_0, (0,), e_i, (i,)) if isinstance(e_0, Array) else\n+            (e_0, None, e_i, (i,))\n+            for i, (e_0, e_i) in enumerate(zip(arr_inds, x_inds), start=1)\n+        ]),\n+        dtype=x.dtype,\n+        concatenate=True\n+    )\n+\n+    return idx_vals\n+\n+\n+def _vindex_dask_arrays_chunk(xc, *args):\n+    inds = args[::2]\n+    x_inds = args[1::2]\n+\n+    xc_inds = []\n+    for i, (e_0, e_i) in enumerate(zip(inds, x_inds)):\n+        if isinstance(e_0, np.ndarray):\n+            xc_inds.append(np.equal.outer(e_0, e_i).nonzero()[1])","path":"dask/array/core.py","position":82,"original_position":82,"commit_id":"a84cf8cc4fcdf5346fb97e43f1a1749fa19a908c","original_commit_id":"a84cf8cc4fcdf5346fb97e43f1a1749fa19a908c","user":{"login":"jakirkham","id":3019665,"avatar_url":"https://avatars1.githubusercontent.com/u/3019665?v=4","gravatar_id":"","url":"https://api.github.com/users/jakirkham","html_url":"https://github.com/jakirkham","followers_url":"https://api.github.com/users/jakirkham/followers","following_url":"https://api.github.com/users/jakirkham/following{/other_user}","gists_url":"https://api.github.com/users/jakirkham/gists{/gist_id}","starred_url":"https://api.github.com/users/jakirkham/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jakirkham/subscriptions","organizations_url":"https://api.github.com/users/jakirkham/orgs","repos_url":"https://api.github.com/users/jakirkham/repos","events_url":"https://api.github.com/users/jakirkham/events{/privacy}","received_events_url":"https://api.github.com/users/jakirkham/received_events","type":"User","site_admin":false},"body":"Sure, that makes sense.\r\n\r\nWhen I previously was playing with this in the gist, it used only Dask Array operations without `atop`. So when pushing it all into `atop`, expect that there remains some things from the old approach that are suboptimal.\r\n\r\nHappy to fix any that come up.","created_at":"2018-03-01T20:41:25Z","updated_at":"2018-03-01T20:41:25Z","html_url":"https://github.com/dask/dask/pull/3210#discussion_r171687369","pull_request_url":"https://api.github.com/repos/dask/dask/pulls/3210","author_association":"MEMBER","_links":{"self":{"href":"https://api.github.com/repos/dask/dask/pulls/comments/171687369"},"html":{"href":"https://github.com/dask/dask/pull/3210#discussion_r171687369"},"pull_request":{"href":"https://api.github.com/repos/dask/dask/pulls/3210"}},"in_reply_to_id":171464037},"pull_request":{"url":"https://api.github.com/repos/dask/dask/pulls/3210","id":171295267,"html_url":"https://github.com/dask/dask/pull/3210","diff_url":"https://github.com/dask/dask/pull/3210.diff","patch_url":"https://github.com/dask/dask/pull/3210.patch","issue_url":"https://api.github.com/repos/dask/dask/issues/3210","number":3210,"state":"open","locked":false,"title":"Support using `vindex` with Dask Arrays","user":{"login":"jakirkham","id":3019665,"avatar_url":"https://avatars1.githubusercontent.com/u/3019665?v=4","gravatar_id":"","url":"https://api.github.com/users/jakirkham","html_url":"https://github.com/jakirkham","followers_url":"https://api.github.com/users/jakirkham/followers","following_url":"https://api.github.com/users/jakirkham/following{/other_user}","gists_url":"https://api.github.com/users/jakirkham/gists{/gist_id}","starred_url":"https://api.github.com/users/jakirkham/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jakirkham/subscriptions","organizations_url":"https://api.github.com/users/jakirkham/orgs","repos_url":"https://api.github.com/users/jakirkham/repos","events_url":"https://api.github.com/users/jakirkham/events{/privacy}","received_events_url":"https://api.github.com/users/jakirkham/received_events","type":"User","site_admin":false},"body":"Fixes https://github.com/dask/dask/issues/3096\r\n\r\nChange `vindex` so that it can support using Dask Arrays for indexing. Does this using `atop` to map Dask Array indices to those representing particular values in the chunk. Thus allowing individual values to be pulled from each chunk and combined into the total result. Comes with the added bonus that this will work whether the Dask Arrays providing coordinates have a known length or an unknown length.","created_at":"2018-02-26T06:55:10Z","updated_at":"2018-03-01T20:41:25Z","closed_at":null,"merged_at":null,"merge_commit_sha":"3bad9e45d30bd0c4ba67f827080b5f1aef0bd563","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[{"id":242862305,"url":"https://api.github.com/repos/dask/dask/labels/array","name":"array","color":"006b75","default":false}],"milestone":null,"commits_url":"https://api.github.com/repos/dask/dask/pulls/3210/commits","review_comments_url":"https://api.github.com/repos/dask/dask/pulls/3210/comments","review_comment_url":"https://api.github.com/repos/dask/dask/pulls/comments{/number}","comments_url":"https://api.github.com/repos/dask/dask/issues/3210/comments","statuses_url":"https://api.github.com/repos/dask/dask/statuses/a84cf8cc4fcdf5346fb97e43f1a1749fa19a908c","head":{"label":"jakirkham:support_dask_arrays_vindex","ref":"support_dask_arrays_vindex","sha":"a84cf8cc4fcdf5346fb97e43f1a1749fa19a908c","user":{"login":"jakirkham","id":3019665,"avatar_url":"https://avatars1.githubusercontent.com/u/3019665?v=4","gravatar_id":"","url":"https://api.github.com/users/jakirkham","html_url":"https://github.com/jakirkham","followers_url":"https://api.github.com/users/jakirkham/followers","following_url":"https://api.github.com/users/jakirkham/following{/other_user}","gists_url":"https://api.github.com/users/jakirkham/gists{/gist_id}","starred_url":"https://api.github.com/users/jakirkham/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jakirkham/subscriptions","organizations_url":"https://api.github.com/users/jakirkham/orgs","repos_url":"https://api.github.com/users/jakirkham/repos","events_url":"https://api.github.com/users/jakirkham/events{/privacy}","received_events_url":"https://api.github.com/users/jakirkham/received_events","type":"User","site_admin":false},"repo":{"id":31235042,"name":"dask","full_name":"jakirkham/dask","owner":{"login":"jakirkham","id":3019665,"avatar_url":"https://avatars1.githubusercontent.com/u/3019665?v=4","gravatar_id":"","url":"https://api.github.com/users/jakirkham","html_url":"https://github.com/jakirkham","followers_url":"https://api.github.com/users/jakirkham/followers","following_url":"https://api.github.com/users/jakirkham/following{/other_user}","gists_url":"https://api.github.com/users/jakirkham/gists{/gist_id}","starred_url":"https://api.github.com/users/jakirkham/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jakirkham/subscriptions","organizations_url":"https://api.github.com/users/jakirkham/orgs","repos_url":"https://api.github.com/users/jakirkham/repos","events_url":"https://api.github.com/users/jakirkham/events{/privacy}","received_events_url":"https://api.github.com/users/jakirkham/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/jakirkham/dask","description":"Minimal task scheduling abstraction","fork":true,"url":"https://api.github.com/repos/jakirkham/dask","forks_url":"https://api.github.com/repos/jakirkham/dask/forks","keys_url":"https://api.github.com/repos/jakirkham/dask/keys{/key_id}","collaborators_url":"https://api.github.com/repos/jakirkham/dask/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/jakirkham/dask/teams","hooks_url":"https://api.github.com/repos/jakirkham/dask/hooks","issue_events_url":"https://api.github.com/repos/jakirkham/dask/issues/events{/number}","events_url":"https://api.github.com/repos/jakirkham/dask/events","assignees_url":"https://api.github.com/repos/jakirkham/dask/assignees{/user}","branches_url":"https://api.github.com/repos/jakirkham/dask/branches{/branch}","tags_url":"https://api.github.com/repos/jakirkham/dask/tags","blobs_url":"https://api.github.com/repos/jakirkham/dask/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/jakirkham/dask/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/jakirkham/dask/git/refs{/sha}","trees_url":"https://api.github.com/repos/jakirkham/dask/git/trees{/sha}","statuses_url":"https://api.github.com/repos/jakirkham/dask/statuses/{sha}","languages_url":"https://api.github.com/repos/jakirkham/dask/languages","stargazers_url":"https://api.github.com/repos/jakirkham/dask/stargazers","contributors_url":"https://api.github.com/repos/jakirkham/dask/contributors","subscribers_url":"https://api.github.com/repos/jakirkham/dask/subscribers","subscription_url":"https://api.github.com/repos/jakirkham/dask/subscription","commits_url":"https://api.github.com/repos/jakirkham/dask/commits{/sha}","git_commits_url":"https://api.github.com/repos/jakirkham/dask/git/commits{/sha}","comments_url":"https://api.github.com/repos/jakirkham/dask/comments{/number}","issue_comment_url":"https://api.github.com/repos/jakirkham/dask/issues/comments{/number}","contents_url":"https://api.github.com/repos/jakirkham/dask/contents/{+path}","compare_url":"https://api.github.com/repos/jakirkham/dask/compare/{base}...{head}","merges_url":"https://api.github.com/repos/jakirkham/dask/merges","archive_url":"https://api.github.com/repos/jakirkham/dask/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/jakirkham/dask/downloads","issues_url":"https://api.github.com/repos/jakirkham/dask/issues{/number}","pulls_url":"https://api.github.com/repos/jakirkham/dask/pulls{/number}","milestones_url":"https://api.github.com/repos/jakirkham/dask/milestones{/number}","notifications_url":"https://api.github.com/repos/jakirkham/dask/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/jakirkham/dask/labels{/name}","releases_url":"https://api.github.com/repos/jakirkham/dask/releases{/id}","deployments_url":"https://api.github.com/repos/jakirkham/dask/deployments","created_at":"2015-02-23T23:11:16Z","updated_at":"2016-02-11T18:22:48Z","pushed_at":"2018-02-27T18:21:24Z","git_url":"git://github.com/jakirkham/dask.git","ssh_url":"git@github.com:jakirkham/dask.git","clone_url":"https://github.com/jakirkham/dask.git","svn_url":"https://github.com/jakirkham/dask","homepage":"","size":16248,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"other","name":"Other","spdx_id":null,"url":null},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"dask:master","ref":"master","sha":"7f69508882ea9f783124f6cd28e31f666423e884","user":{"login":"dask","id":17131925,"avatar_url":"https://avatars1.githubusercontent.com/u/17131925?v=4","gravatar_id":"","url":"https://api.github.com/users/dask","html_url":"https://github.com/dask","followers_url":"https://api.github.com/users/dask/followers","following_url":"https://api.github.com/users/dask/following{/other_user}","gists_url":"https://api.github.com/users/dask/gists{/gist_id}","starred_url":"https://api.github.com/users/dask/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dask/subscriptions","organizations_url":"https://api.github.com/users/dask/orgs","repos_url":"https://api.github.com/users/dask/repos","events_url":"https://api.github.com/users/dask/events{/privacy}","received_events_url":"https://api.github.com/users/dask/received_events","type":"Organization","site_admin":false},"repo":{"id":28782747,"name":"dask","full_name":"dask/dask","owner":{"login":"dask","id":17131925,"avatar_url":"https://avatars1.githubusercontent.com/u/17131925?v=4","gravatar_id":"","url":"https://api.github.com/users/dask","html_url":"https://github.com/dask","followers_url":"https://api.github.com/users/dask/followers","following_url":"https://api.github.com/users/dask/following{/other_user}","gists_url":"https://api.github.com/users/dask/gists{/gist_id}","starred_url":"https://api.github.com/users/dask/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dask/subscriptions","organizations_url":"https://api.github.com/users/dask/orgs","repos_url":"https://api.github.com/users/dask/repos","events_url":"https://api.github.com/users/dask/events{/privacy}","received_events_url":"https://api.github.com/users/dask/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/dask/dask","description":"Parallel computing with task scheduling","fork":false,"url":"https://api.github.com/repos/dask/dask","forks_url":"https://api.github.com/repos/dask/dask/forks","keys_url":"https://api.github.com/repos/dask/dask/keys{/key_id}","collaborators_url":"https://api.github.com/repos/dask/dask/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/dask/dask/teams","hooks_url":"https://api.github.com/repos/dask/dask/hooks","issue_events_url":"https://api.github.com/repos/dask/dask/issues/events{/number}","events_url":"https://api.github.com/repos/dask/dask/events","assignees_url":"https://api.github.com/repos/dask/dask/assignees{/user}","branches_url":"https://api.github.com/repos/dask/dask/branches{/branch}","tags_url":"https://api.github.com/repos/dask/dask/tags","blobs_url":"https://api.github.com/repos/dask/dask/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/dask/dask/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/dask/dask/git/refs{/sha}","trees_url":"https://api.github.com/repos/dask/dask/git/trees{/sha}","statuses_url":"https://api.github.com/repos/dask/dask/statuses/{sha}","languages_url":"https://api.github.com/repos/dask/dask/languages","stargazers_url":"https://api.github.com/repos/dask/dask/stargazers","contributors_url":"https://api.github.com/repos/dask/dask/contributors","subscribers_url":"https://api.github.com/repos/dask/dask/subscribers","subscription_url":"https://api.github.com/repos/dask/dask/subscription","commits_url":"https://api.github.com/repos/dask/dask/commits{/sha}","git_commits_url":"https://api.github.com/repos/dask/dask/git/commits{/sha}","comments_url":"https://api.github.com/repos/dask/dask/comments{/number}","issue_comment_url":"https://api.github.com/repos/dask/dask/issues/comments{/number}","contents_url":"https://api.github.com/repos/dask/dask/contents/{+path}","compare_url":"https://api.github.com/repos/dask/dask/compare/{base}...{head}","merges_url":"https://api.github.com/repos/dask/dask/merges","archive_url":"https://api.github.com/repos/dask/dask/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/dask/dask/downloads","issues_url":"https://api.github.com/repos/dask/dask/issues{/number}","pulls_url":"https://api.github.com/repos/dask/dask/pulls{/number}","milestones_url":"https://api.github.com/repos/dask/dask/milestones{/number}","notifications_url":"https://api.github.com/repos/dask/dask/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/dask/dask/labels{/name}","releases_url":"https://api.github.com/repos/dask/dask/releases{/id}","deployments_url":"https://api.github.com/repos/dask/dask/deployments","created_at":"2015-01-04T18:50:00Z","updated_at":"2018-03-01T19:40:41Z","pushed_at":"2018-03-01T19:58:27Z","git_url":"git://github.com/dask/dask.git","ssh_url":"git@github.com:dask/dask.git","clone_url":"https://github.com/dask/dask.git","svn_url":"https://github.com/dask/dask","homepage":"http://dask.pydata.org","size":16139,"stargazers_count":2491,"watchers_count":2491,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":421,"mirror_url":null,"archived":false,"open_issues_count":343,"license":{"key":"other","name":"Other","spdx_id":null,"url":null},"forks":421,"open_issues":343,"watchers":2491,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/dask/dask/pulls/3210"},"html":{"href":"https://github.com/dask/dask/pull/3210"},"issue":{"href":"https://api.github.com/repos/dask/dask/issues/3210"},"comments":{"href":"https://api.github.com/repos/dask/dask/issues/3210/comments"},"review_comments":{"href":"https://api.github.com/repos/dask/dask/pulls/3210/comments"},"review_comment":{"href":"https://api.github.com/repos/dask/dask/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/dask/dask/pulls/3210/commits"},"statuses":{"href":"https://api.github.com/repos/dask/dask/statuses/a84cf8cc4fcdf5346fb97e43f1a1749fa19a908c"}},"author_association":"MEMBER"}}
{"actor":{"display_login":"jakirkham"}}