{"action":"opened","issue":{"url":"https://api.github.com/repos/numpy/numpy/issues/10852","repository_url":"https://api.github.com/repos/numpy/numpy","labels_url":"https://api.github.com/repos/numpy/numpy/issues/10852/labels{/name}","comments_url":"https://api.github.com/repos/numpy/numpy/issues/10852/comments","events_url":"https://api.github.com/repos/numpy/numpy/issues/10852/events","html_url":"https://github.com/numpy/numpy/issues/10852","id":311803743,"number":10852,"title":"Possible optimization when multiplying memory-mapped array by mostly zeros","user":{"login":"zietzm","id":7230303,"avatar_url":"https://avatars1.githubusercontent.com/u/7230303?v=4","gravatar_id":"","url":"https://api.github.com/users/zietzm","html_url":"https://github.com/zietzm","followers_url":"https://api.github.com/users/zietzm/followers","following_url":"https://api.github.com/users/zietzm/following{/other_user}","gists_url":"https://api.github.com/users/zietzm/gists{/gist_id}","starred_url":"https://api.github.com/users/zietzm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zietzm/subscriptions","organizations_url":"https://api.github.com/users/zietzm/orgs","repos_url":"https://api.github.com/users/zietzm/repos","events_url":"https://api.github.com/users/zietzm/events{/privacy}","received_events_url":"https://api.github.com/users/zietzm/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-04-05T23:34:35Z","updated_at":"2018-04-05T23:34:35Z","closed_at":null,"author_association":"NONE","body":"We are saving a large number of arrays to disk (in .npy format), which need to be loaded as quickly as possible to be multiplied by a vector (array). The arrays on disk vary in size but can be relatively large (~ several GB), and the vectors are mostly zeros.\r\n\r\nWhen loading an array or matrix with memmap for the purpose of multiplying by another array, [we found](https://github.com/greenelab/hetmech/blob/6d135267ed4ec02fec3a3d976ea8751524c49be0/explore/matrix-io/02.sparse-and-dense.ipynb) that subsetting the array for nonzero rows of the vector can be significantly faster. To do so, we can subset both the in-memory vector and on-disk array for nonzero rows and then multiply. \r\n\r\nMy question is, does or could numpy do similar optimization? That is, when an array is \"loaded\" as a memmap from a disk file and multiplied by a mostly-zero array, does/could numpy subset both for only the nonzero rows to increase speed?"}}
{"actor":{"display_login":"zietzm"}}