{"action":"created","issue":{"url":"https://api.github.com/repos/dask/dask/issues/3419","repository_url":"https://api.github.com/repos/dask/dask","labels_url":"https://api.github.com/repos/dask/dask/issues/3419/labels{/name}","comments_url":"https://api.github.com/repos/dask/dask/issues/3419/comments","events_url":"https://api.github.com/repos/dask/dask/issues/3419/events","html_url":"https://github.com/dask/dask/issues/3419","id":315705893,"number":3419,"title":"Scaling of memory for svd_compressed","user":{"login":"jmarkow","id":76247,"avatar_url":"https://avatars3.githubusercontent.com/u/76247?v=4","gravatar_id":"","url":"https://api.github.com/users/jmarkow","html_url":"https://github.com/jmarkow","followers_url":"https://api.github.com/users/jmarkow/followers","following_url":"https://api.github.com/users/jmarkow/following{/other_user}","gists_url":"https://api.github.com/users/jmarkow/gists{/gist_id}","starred_url":"https://api.github.com/users/jmarkow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jmarkow/subscriptions","organizations_url":"https://api.github.com/users/jmarkow/orgs","repos_url":"https://api.github.com/users/jmarkow/repos","events_url":"https://api.github.com/users/jmarkow/events{/privacy}","received_events_url":"https://api.github.com/users/jmarkow/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2018-04-19T02:30:53Z","updated_at":"2018-04-19T16:40:20Z","closed_at":null,"author_association":"NONE","body":"I'm curious about the memory usage of `linalg.svd`.  In the following example I find that memory scales with the size of the array, not the blocksize,\r\n\r\n```\r\nimport dask.array as da\r\nimport dask.array.linalg\r\nar =  da.random.random((600000,6400), chunks=(1000,6400))\r\nu, s, v= da.linalg.svd_compressed(ar[:500], 100, 0)\r\n%memit s, v = dask.compute(s, v)\r\nu, s, v= da.linalg.svd_compressed(ar[:500000], 100, 0)\r\n%memit s, v = dask.compute(s, v)\r\n```\r\nI see the following memory usage,\r\n\r\n```\r\npeak memory: 357.35 MiB, increment: 53.86 MiB\r\npeak memory: 25273.55 MiB, increment: 24951.71 MiB\r\n```\r\n\r\nBut the documentation for the underlying tsqr algorithm [states](https://github.com/dask/dask/blob/master/dask/array/linalg.py#L45),\r\n\r\n```\r\nIf blocks are of size ``(n, k)`` then this algorithm has memory use that\r\nscales as ``n**2 * k * nthreads``.\r\n```\r\n\r\nI see similar usage patterns running tsqr alone on the data.  "},"comment":{"url":"https://api.github.com/repos/dask/dask/issues/comments/382802303","html_url":"https://github.com/dask/dask/issues/3419#issuecomment-382802303","issue_url":"https://api.github.com/repos/dask/dask/issues/3419","id":382802303,"user":{"login":"jmarkow","id":76247,"avatar_url":"https://avatars3.githubusercontent.com/u/76247?v=4","gravatar_id":"","url":"https://api.github.com/users/jmarkow","html_url":"https://github.com/jmarkow","followers_url":"https://api.github.com/users/jmarkow/followers","following_url":"https://api.github.com/users/jmarkow/following{/other_user}","gists_url":"https://api.github.com/users/jmarkow/gists{/gist_id}","starred_url":"https://api.github.com/users/jmarkow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jmarkow/subscriptions","organizations_url":"https://api.github.com/users/jmarkow/orgs","repos_url":"https://api.github.com/users/jmarkow/repos","events_url":"https://api.github.com/users/jmarkow/events{/privacy}","received_events_url":"https://api.github.com/users/jmarkow/received_events","type":"User","site_admin":false},"created_at":"2018-04-19T16:40:20Z","updated_at":"2018-04-19T16:40:20Z","author_association":"NONE","body":"Fair enough. \r\n\r\nWhat if we used a `naive` covariance estimate?  It's quite straightforward and chunks can be passed in instead of `datums` per the Wikipedia parlance.  What would be the best way to implement with Dask?\r\n\r\nhttps://en.wikipedia.org/wiki/Algorithms_for_calculating_variance\r\n\r\nSee the pseudocode at the top.  "}}
{"actor":{"display_login":"jmarkow"}}