{"action":"created","issue":{"url":"https://api.github.com/repos/dask/dask/issues/3671","repository_url":"https://api.github.com/repos/dask/dask","labels_url":"https://api.github.com/repos/dask/dask/issues/3671/labels{/name}","comments_url":"https://api.github.com/repos/dask/dask/issues/3671/comments","events_url":"https://api.github.com/repos/dask/dask/issues/3671/events","html_url":"https://github.com/dask/dask/issues/3671","id":335904087,"node_id":"MDU6SXNzdWUzMzU5MDQwODc=","number":3671,"title":"dask array map_overlap large memory use","user":{"login":"magnunor","id":1690979,"node_id":"MDQ6VXNlcjE2OTA5Nzk=","avatar_url":"https://avatars1.githubusercontent.com/u/1690979?v=4","gravatar_id":"","url":"https://api.github.com/users/magnunor","html_url":"https://github.com/magnunor","followers_url":"https://api.github.com/users/magnunor/followers","following_url":"https://api.github.com/users/magnunor/following{/other_user}","gists_url":"https://api.github.com/users/magnunor/gists{/gist_id}","starred_url":"https://api.github.com/users/magnunor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/magnunor/subscriptions","organizations_url":"https://api.github.com/users/magnunor/orgs","repos_url":"https://api.github.com/users/magnunor/repos","events_url":"https://api.github.com/users/magnunor/events{/privacy}","received_events_url":"https://api.github.com/users/magnunor/received_events","type":"User","site_admin":false},"labels":[{"id":242862305,"node_id":"MDU6TGFiZWwyNDI4NjIzMDU=","url":"https://api.github.com/repos/dask/dask/labels/array","name":"array","color":"006b75","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-06-26T16:38:56Z","updated_at":"2018-06-26T20:31:48Z","closed_at":null,"author_association":"NONE","body":"I have some very large datasets containing stacks of images, and I'm trying to do some type of processing on each image. Having both the stack and the images across several chunks is very useful for the processing I'm doing.\r\n\r\n`map_overlap` is therefore the obvious choice, but I've been having some memory issues when using it. Doing the equivalent processing using `map_blocks` uses about 2.1 GB of memory during the processing, while `map_overlap` uses more than 30 GB (I had to cancel the computation before running out of memory).\r\n\r\n`map_blocks` version:\r\n\r\n```python\r\nimport numpy as np\r\nimport dask.array as da\r\nfrom dask.diagnostics import ProgressBar\r\n\r\ndef remove_dead_pixels(data):\r\n    dead_pixels = data == 0\r\n    output_data = np.roll(data, shift=1, axis=-2) * dead_pixels\r\n    return output_data\r\n\r\ndata = da.random.randint(0, 100000, size=(500, 500, 300, 300), chunks=(10, 10, 30, 30))\r\n\r\ntemp_data = da.map_blocks(\r\n        remove_dead_pixels, data, dtype=data.dtype, chunks=data.chunks)\r\n\r\ntemp_data_mean = temp_data.mean(axis=(0, 1))\r\n\r\nwith ProgressBar():\r\n    temp_data_mean_computed = temp_data_mean.compute()\r\n\r\n# Uses about 2.1 GB memory\r\n```\r\n\r\n`map_overlap` version:\r\n\r\n```python\r\nimport numpy as np\r\nimport dask.array as da\r\nfrom dask.diagnostics import ProgressBar\r\n\r\ndef remove_dead_pixels(data):\r\n    dead_pixels = data == 0\r\n    output_data = np.roll(data, shift=1, axis=-2) * dead_pixels\r\n    return output_data\r\n\r\ndata = da.random.randint(0, 100000, size=(500, 500, 300, 300), chunks=(10, 10, 30, 30))\r\n\r\ntemp_data = da.map_overlap(\r\n        func=remove_dead_pixels, x=data, depth=(0, 0, 1, 1),\r\n        dtype=data.dtype, chunks=data.chunks)\r\n\r\ntemp_data_mean = temp_data.mean(axis=(0, 1))\r\n\r\nwith ProgressBar():\r\n    temp_data_mean_computed = temp_data_mean.compute()\r\n\r\n# Uses more than 30+ GB memory\r\n```\r\n\r\nDask version: current master branch\r\nNumpy version: 1.14.5\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/dask/dask/issues/comments/400451888","html_url":"https://github.com/dask/dask/issues/3671#issuecomment-400451888","issue_url":"https://api.github.com/repos/dask/dask/issues/3671","id":400451888,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMDQ1MTg4OA==","user":{"login":"jakirkham","id":3019665,"node_id":"MDQ6VXNlcjMwMTk2NjU=","avatar_url":"https://avatars1.githubusercontent.com/u/3019665?v=4","gravatar_id":"","url":"https://api.github.com/users/jakirkham","html_url":"https://github.com/jakirkham","followers_url":"https://api.github.com/users/jakirkham/followers","following_url":"https://api.github.com/users/jakirkham/following{/other_user}","gists_url":"https://api.github.com/users/jakirkham/gists{/gist_id}","starred_url":"https://api.github.com/users/jakirkham/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jakirkham/subscriptions","organizations_url":"https://api.github.com/users/jakirkham/orgs","repos_url":"https://api.github.com/users/jakirkham/repos","events_url":"https://api.github.com/users/jakirkham/events{/privacy}","received_events_url":"https://api.github.com/users/jakirkham/received_events","type":"User","site_admin":false},"created_at":"2018-06-26T20:31:48Z","updated_at":"2018-06-26T20:31:48Z","author_association":"MEMBER","body":"Could you please clarify what commit of Dask you are using?"}}
{"actor":{"display_login":"jakirkham"}}