{"action":"created","comment":{"url":"https://api.github.com/repos/dask/dask/pulls/comments/168811057","pull_request_review_id":97244775,"id":168811057,"diff_hunk":"@@ -889,65 +889,61 @@ def store(sources, targets, lock=True, regions=None, compute=True,\n     sources_dsk = sharedict.merge(*[e.__dask_graph__() for e in sources])\n     sources_dsk = Array.__dask_optimize__(\n         sources_dsk,\n-        [e.__dask_keys__() for e in sources]\n+        list(core.flatten([e.__dask_keys__() for e in sources]))\n     )\n \n-    tgt_dsks = []\n-    store_keys = []\n-    store_dsks = []\n-    if return_stored:\n-        load_names = []\n-        load_dsks = []\n-    for tgt, src, reg in zip(targets, sources, regions):\n-        # if out is a delayed object update dictionary accordingly\n+    # Optimize all targets together\n+    targets_keys = []\n+    targets_dsk = []\n+    for e in targets:\n         try:\n-            each_tgt_dsk = {}\n-            each_tgt_dsk.update(tgt.dask)\n-            tgt = tgt.key\n+            targets_keys.extend(e.__dask_keys__())\n+            targets_dsk.append(e.__dask_graph__())\n         except AttributeError:\n-            each_tgt_dsk = {}\n+            pass\n+\n+    targets_dsk = sharedict.merge(*targets_dsk)\n+    targets_dsk = Delayed.__dask_optimize__(targets_dsk, targets_keys)","path":"dask/array/core.py","position":46,"original_position":46,"commit_id":"566ec928d3282ea899efd5dcc983c63c06b463cd","original_commit_id":"e931d08d922429a57650fc0297748afc7837fb4c","user":{"login":"jakirkham","id":3019665,"avatar_url":"https://avatars1.githubusercontent.com/u/3019665?v=4","gravatar_id":"","url":"https://api.github.com/users/jakirkham","html_url":"https://github.com/jakirkham","followers_url":"https://api.github.com/users/jakirkham/followers","following_url":"https://api.github.com/users/jakirkham/following{/other_user}","gists_url":"https://api.github.com/users/jakirkham/gists{/gist_id}","starred_url":"https://api.github.com/users/jakirkham/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jakirkham/subscriptions","organizations_url":"https://api.github.com/users/jakirkham/orgs","repos_url":"https://api.github.com/users/jakirkham/repos","events_url":"https://api.github.com/users/jakirkham/events{/privacy}","received_events_url":"https://api.github.com/users/jakirkham/received_events","type":"User","site_admin":false},"body":"Probably should also add that the main use case for `Delayed` targets was to allow one to create a storage location only once it was needed. So expect in practice these are quite shallow and don't share anything with the Dask Arrays sources. IOW probably not worth worrying about unless someone brings up a use case.","created_at":"2018-02-16T16:53:11Z","updated_at":"2018-02-16T16:53:11Z","html_url":"https://github.com/dask/dask/pull/3153#discussion_r168811057","pull_request_url":"https://api.github.com/repos/dask/dask/pulls/3153","author_association":"MEMBER","_links":{"self":{"href":"https://api.github.com/repos/dask/dask/pulls/comments/168811057"},"html":{"href":"https://github.com/dask/dask/pull/3153#discussion_r168811057"},"pull_request":{"href":"https://api.github.com/repos/dask/dask/pulls/3153"}},"in_reply_to_id":167701475},"pull_request":{"url":"https://api.github.com/repos/dask/dask/pulls/3153","id":168301488,"html_url":"https://github.com/dask/dask/pull/3153","diff_url":"https://github.com/dask/dask/pull/3153.diff","patch_url":"https://github.com/dask/dask/pull/3153.patch","issue_url":"https://api.github.com/repos/dask/dask/issues/3153","number":3153,"state":"open","locked":false,"title":"Minor optimizations to Dask Array's store","user":{"login":"jakirkham","id":3019665,"avatar_url":"https://avatars1.githubusercontent.com/u/3019665?v=4","gravatar_id":"","url":"https://api.github.com/users/jakirkham","html_url":"https://github.com/jakirkham","followers_url":"https://api.github.com/users/jakirkham/followers","following_url":"https://api.github.com/users/jakirkham/following{/other_user}","gists_url":"https://api.github.com/users/jakirkham/gists{/gist_id}","starred_url":"https://api.github.com/users/jakirkham/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jakirkham/subscriptions","organizations_url":"https://api.github.com/users/jakirkham/orgs","repos_url":"https://api.github.com/users/jakirkham/repos","events_url":"https://api.github.com/users/jakirkham/events{/privacy}","received_events_url":"https://api.github.com/users/jakirkham/received_events","type":"User","site_admin":false},"body":"Performs some optimizations for `Delayed` targets. Namely skips collecting empty dictionaries if no `Delayed` target is present. Also optimized the `Delayed` target portion of the Dask as this was not being done before (already done as part of the Dask Array optimizations). In cases not using `Delayed` targets, this should be a no-op.\r\n\r\nIf `return_stored` and `compute` are both `True`, builds the `load_chunk` Dask after calling `persist` on the `store_chunk` Dask. This comes with the nice benefit of inlining results from `persist` directly into the `load_chunk` Dask. Thus simplifying the graph used for the Dask Arrays constructed in this case. Also simplifies the `for`-loop over sources and targets.\r\n\r\nIn the case of `return_stored=True` and `compute=False`, fuses the `store_chunk` and `load_chunk` calls into one `load_store_chunk` function per @jcrist's [suggestion]( https://github.com/dask/dask/pull/3082#issuecomment-360571394 ). This allows the lock (if any) to be held for the duration of both the store and load steps. Thus this should avoid contention between store and load steps on chunks.\r\n\r\nMerges Dasks corresponding to chunks of particular types (e.g. `Delayed` targets, `store_chunk` calls, `load_chunk` calls) in a few cases before merging with other Dask types.\r\n\r\nAlso simplifies `store_chunk` and `load_chunk` by running `fuse_slice` for the provided `region` before constructing the Dask. This simplifies the logic in these functions and avoids the overhead (however small) of offloading this work to these functions.\r\n\r\nAdmittedly of the optimizations done here, similar optimizations are performed when submitting Dask Arrays for computation. However as `store` represents a final step in many ways, it seems reasonable to perform some of these optimizations here before returning to the user.","created_at":"2018-02-09T17:42:49Z","updated_at":"2018-02-16T16:53:11Z","closed_at":null,"merged_at":null,"merge_commit_sha":"3b314b59b14d5180790a44419f6928f5cc128a98","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/dask/dask/pulls/3153/commits","review_comments_url":"https://api.github.com/repos/dask/dask/pulls/3153/comments","review_comment_url":"https://api.github.com/repos/dask/dask/pulls/comments{/number}","comments_url":"https://api.github.com/repos/dask/dask/issues/3153/comments","statuses_url":"https://api.github.com/repos/dask/dask/statuses/566ec928d3282ea899efd5dcc983c63c06b463cd","head":{"label":"jakirkham:minor_store_opts","ref":"minor_store_opts","sha":"566ec928d3282ea899efd5dcc983c63c06b463cd","user":{"login":"jakirkham","id":3019665,"avatar_url":"https://avatars1.githubusercontent.com/u/3019665?v=4","gravatar_id":"","url":"https://api.github.com/users/jakirkham","html_url":"https://github.com/jakirkham","followers_url":"https://api.github.com/users/jakirkham/followers","following_url":"https://api.github.com/users/jakirkham/following{/other_user}","gists_url":"https://api.github.com/users/jakirkham/gists{/gist_id}","starred_url":"https://api.github.com/users/jakirkham/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jakirkham/subscriptions","organizations_url":"https://api.github.com/users/jakirkham/orgs","repos_url":"https://api.github.com/users/jakirkham/repos","events_url":"https://api.github.com/users/jakirkham/events{/privacy}","received_events_url":"https://api.github.com/users/jakirkham/received_events","type":"User","site_admin":false},"repo":{"id":31235042,"name":"dask","full_name":"jakirkham/dask","owner":{"login":"jakirkham","id":3019665,"avatar_url":"https://avatars1.githubusercontent.com/u/3019665?v=4","gravatar_id":"","url":"https://api.github.com/users/jakirkham","html_url":"https://github.com/jakirkham","followers_url":"https://api.github.com/users/jakirkham/followers","following_url":"https://api.github.com/users/jakirkham/following{/other_user}","gists_url":"https://api.github.com/users/jakirkham/gists{/gist_id}","starred_url":"https://api.github.com/users/jakirkham/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jakirkham/subscriptions","organizations_url":"https://api.github.com/users/jakirkham/orgs","repos_url":"https://api.github.com/users/jakirkham/repos","events_url":"https://api.github.com/users/jakirkham/events{/privacy}","received_events_url":"https://api.github.com/users/jakirkham/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/jakirkham/dask","description":"Minimal task scheduling abstraction","fork":true,"url":"https://api.github.com/repos/jakirkham/dask","forks_url":"https://api.github.com/repos/jakirkham/dask/forks","keys_url":"https://api.github.com/repos/jakirkham/dask/keys{/key_id}","collaborators_url":"https://api.github.com/repos/jakirkham/dask/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/jakirkham/dask/teams","hooks_url":"https://api.github.com/repos/jakirkham/dask/hooks","issue_events_url":"https://api.github.com/repos/jakirkham/dask/issues/events{/number}","events_url":"https://api.github.com/repos/jakirkham/dask/events","assignees_url":"https://api.github.com/repos/jakirkham/dask/assignees{/user}","branches_url":"https://api.github.com/repos/jakirkham/dask/branches{/branch}","tags_url":"https://api.github.com/repos/jakirkham/dask/tags","blobs_url":"https://api.github.com/repos/jakirkham/dask/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/jakirkham/dask/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/jakirkham/dask/git/refs{/sha}","trees_url":"https://api.github.com/repos/jakirkham/dask/git/trees{/sha}","statuses_url":"https://api.github.com/repos/jakirkham/dask/statuses/{sha}","languages_url":"https://api.github.com/repos/jakirkham/dask/languages","stargazers_url":"https://api.github.com/repos/jakirkham/dask/stargazers","contributors_url":"https://api.github.com/repos/jakirkham/dask/contributors","subscribers_url":"https://api.github.com/repos/jakirkham/dask/subscribers","subscription_url":"https://api.github.com/repos/jakirkham/dask/subscription","commits_url":"https://api.github.com/repos/jakirkham/dask/commits{/sha}","git_commits_url":"https://api.github.com/repos/jakirkham/dask/git/commits{/sha}","comments_url":"https://api.github.com/repos/jakirkham/dask/comments{/number}","issue_comment_url":"https://api.github.com/repos/jakirkham/dask/issues/comments{/number}","contents_url":"https://api.github.com/repos/jakirkham/dask/contents/{+path}","compare_url":"https://api.github.com/repos/jakirkham/dask/compare/{base}...{head}","merges_url":"https://api.github.com/repos/jakirkham/dask/merges","archive_url":"https://api.github.com/repos/jakirkham/dask/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/jakirkham/dask/downloads","issues_url":"https://api.github.com/repos/jakirkham/dask/issues{/number}","pulls_url":"https://api.github.com/repos/jakirkham/dask/pulls{/number}","milestones_url":"https://api.github.com/repos/jakirkham/dask/milestones{/number}","notifications_url":"https://api.github.com/repos/jakirkham/dask/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/jakirkham/dask/labels{/name}","releases_url":"https://api.github.com/repos/jakirkham/dask/releases{/id}","deployments_url":"https://api.github.com/repos/jakirkham/dask/deployments","created_at":"2015-02-23T23:11:16Z","updated_at":"2016-02-11T18:22:48Z","pushed_at":"2018-02-16T16:20:15Z","git_url":"git://github.com/jakirkham/dask.git","ssh_url":"git@github.com:jakirkham/dask.git","clone_url":"https://github.com/jakirkham/dask.git","svn_url":"https://github.com/jakirkham/dask","homepage":"","size":16089,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"other","name":"Other","spdx_id":null,"url":null},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"dask:master","ref":"master","sha":"1893ebf938164d6dd80c347c3a31313dcd6b41a7","user":{"login":"dask","id":17131925,"avatar_url":"https://avatars1.githubusercontent.com/u/17131925?v=4","gravatar_id":"","url":"https://api.github.com/users/dask","html_url":"https://github.com/dask","followers_url":"https://api.github.com/users/dask/followers","following_url":"https://api.github.com/users/dask/following{/other_user}","gists_url":"https://api.github.com/users/dask/gists{/gist_id}","starred_url":"https://api.github.com/users/dask/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dask/subscriptions","organizations_url":"https://api.github.com/users/dask/orgs","repos_url":"https://api.github.com/users/dask/repos","events_url":"https://api.github.com/users/dask/events{/privacy}","received_events_url":"https://api.github.com/users/dask/received_events","type":"Organization","site_admin":false},"repo":{"id":28782747,"name":"dask","full_name":"dask/dask","owner":{"login":"dask","id":17131925,"avatar_url":"https://avatars1.githubusercontent.com/u/17131925?v=4","gravatar_id":"","url":"https://api.github.com/users/dask","html_url":"https://github.com/dask","followers_url":"https://api.github.com/users/dask/followers","following_url":"https://api.github.com/users/dask/following{/other_user}","gists_url":"https://api.github.com/users/dask/gists{/gist_id}","starred_url":"https://api.github.com/users/dask/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dask/subscriptions","organizations_url":"https://api.github.com/users/dask/orgs","repos_url":"https://api.github.com/users/dask/repos","events_url":"https://api.github.com/users/dask/events{/privacy}","received_events_url":"https://api.github.com/users/dask/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/dask/dask","description":"Parallel computing with task scheduling","fork":false,"url":"https://api.github.com/repos/dask/dask","forks_url":"https://api.github.com/repos/dask/dask/forks","keys_url":"https://api.github.com/repos/dask/dask/keys{/key_id}","collaborators_url":"https://api.github.com/repos/dask/dask/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/dask/dask/teams","hooks_url":"https://api.github.com/repos/dask/dask/hooks","issue_events_url":"https://api.github.com/repos/dask/dask/issues/events{/number}","events_url":"https://api.github.com/repos/dask/dask/events","assignees_url":"https://api.github.com/repos/dask/dask/assignees{/user}","branches_url":"https://api.github.com/repos/dask/dask/branches{/branch}","tags_url":"https://api.github.com/repos/dask/dask/tags","blobs_url":"https://api.github.com/repos/dask/dask/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/dask/dask/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/dask/dask/git/refs{/sha}","trees_url":"https://api.github.com/repos/dask/dask/git/trees{/sha}","statuses_url":"https://api.github.com/repos/dask/dask/statuses/{sha}","languages_url":"https://api.github.com/repos/dask/dask/languages","stargazers_url":"https://api.github.com/repos/dask/dask/stargazers","contributors_url":"https://api.github.com/repos/dask/dask/contributors","subscribers_url":"https://api.github.com/repos/dask/dask/subscribers","subscription_url":"https://api.github.com/repos/dask/dask/subscription","commits_url":"https://api.github.com/repos/dask/dask/commits{/sha}","git_commits_url":"https://api.github.com/repos/dask/dask/git/commits{/sha}","comments_url":"https://api.github.com/repos/dask/dask/comments{/number}","issue_comment_url":"https://api.github.com/repos/dask/dask/issues/comments{/number}","contents_url":"https://api.github.com/repos/dask/dask/contents/{+path}","compare_url":"https://api.github.com/repos/dask/dask/compare/{base}...{head}","merges_url":"https://api.github.com/repos/dask/dask/merges","archive_url":"https://api.github.com/repos/dask/dask/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/dask/dask/downloads","issues_url":"https://api.github.com/repos/dask/dask/issues{/number}","pulls_url":"https://api.github.com/repos/dask/dask/pulls{/number}","milestones_url":"https://api.github.com/repos/dask/dask/milestones{/number}","notifications_url":"https://api.github.com/repos/dask/dask/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/dask/dask/labels{/name}","releases_url":"https://api.github.com/repos/dask/dask/releases{/id}","deployments_url":"https://api.github.com/repos/dask/dask/deployments","created_at":"2015-01-04T18:50:00Z","updated_at":"2018-02-16T14:48:43Z","pushed_at":"2018-02-16T16:20:18Z","git_url":"git://github.com/dask/dask.git","ssh_url":"git@github.com:dask/dask.git","clone_url":"https://github.com/dask/dask.git","svn_url":"https://github.com/dask/dask","homepage":"http://dask.pydata.org","size":16068,"stargazers_count":2432,"watchers_count":2432,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":412,"mirror_url":null,"archived":false,"open_issues_count":318,"license":{"key":"other","name":"Other","spdx_id":null,"url":null},"forks":412,"open_issues":318,"watchers":2432,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/dask/dask/pulls/3153"},"html":{"href":"https://github.com/dask/dask/pull/3153"},"issue":{"href":"https://api.github.com/repos/dask/dask/issues/3153"},"comments":{"href":"https://api.github.com/repos/dask/dask/issues/3153/comments"},"review_comments":{"href":"https://api.github.com/repos/dask/dask/pulls/3153/comments"},"review_comment":{"href":"https://api.github.com/repos/dask/dask/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/dask/dask/pulls/3153/commits"},"statuses":{"href":"https://api.github.com/repos/dask/dask/statuses/566ec928d3282ea899efd5dcc983c63c06b463cd"}},"author_association":"MEMBER"}}
{"actor":{"display_login":"jakirkham"}}