{"action":"created","issue":{"url":"https://api.github.com/repos/dask/dask/issues/174","repository_url":"https://api.github.com/repos/dask/dask","labels_url":"https://api.github.com/repos/dask/dask/issues/174/labels{/name}","comments_url":"https://api.github.com/repos/dask/dask/issues/174/comments","events_url":"https://api.github.com/repos/dask/dask/issues/174/events","html_url":"https://github.com/dask/dask/issues/174","id":71356312,"number":174,"title":"Investigate sparse matrices","user":{"login":"mrocklin","id":306380,"avatar_url":"https://avatars.githubusercontent.com/u/306380?v=3","gravatar_id":"","url":"https://api.github.com/users/mrocklin","html_url":"https://github.com/mrocklin","followers_url":"https://api.github.com/users/mrocklin/followers","following_url":"https://api.github.com/users/mrocklin/following{/other_user}","gists_url":"https://api.github.com/users/mrocklin/gists{/gist_id}","starred_url":"https://api.github.com/users/mrocklin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrocklin/subscriptions","organizations_url":"https://api.github.com/users/mrocklin/orgs","repos_url":"https://api.github.com/users/mrocklin/repos","events_url":"https://api.github.com/users/mrocklin/events{/privacy}","received_events_url":"https://api.github.com/users/mrocklin/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-04-27T17:53:52Z","updated_at":"2016-04-28T11:58:15Z","closed_at":"2016-04-26T20:13:24Z","body":"Some questions for anyone who happens along:\r\n\r\n1.  For what applications do people use large sparse matrices?\r\n2.  How do they store these matrices?\r\n3.  What operations do they need?\r\n4.  What do blocked algorithms tend to look like in this case?\r\n\r\nI can imagine a few solutions but I'm likely to build the wrong thing without hearing from people first.  This might require a little research."},"comment":{"url":"https://api.github.com/repos/dask/dask/issues/comments/215401590","html_url":"https://github.com/dask/dask/issues/174#issuecomment-215401590","issue_url":"https://api.github.com/repos/dask/dask/issues/174","id":215401590,"user":{"login":"mrcabbage972","id":11806181,"avatar_url":"https://avatars.githubusercontent.com/u/11806181?v=3","gravatar_id":"","url":"https://api.github.com/users/mrcabbage972","html_url":"https://github.com/mrcabbage972","followers_url":"https://api.github.com/users/mrcabbage972/followers","following_url":"https://api.github.com/users/mrcabbage972/following{/other_user}","gists_url":"https://api.github.com/users/mrcabbage972/gists{/gist_id}","starred_url":"https://api.github.com/users/mrcabbage972/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrcabbage972/subscriptions","organizations_url":"https://api.github.com/users/mrcabbage972/orgs","repos_url":"https://api.github.com/users/mrcabbage972/repos","events_url":"https://api.github.com/users/mrcabbage972/events{/privacy}","received_events_url":"https://api.github.com/users/mrcabbage972/received_events","type":"User","site_admin":false},"created_at":"2016-04-28T11:58:15Z","updated_at":"2016-04-28T11:58:15Z","body":"An important use case of sparse matrices in machine learning is [one-hot encoding](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html). This is the standard way of using categorical variables in ML models and it produces matrices that can be very sparse. Implementations of one-hot encoding that outputs sparse matrices exist in Pandas and Spark ML/MLLib. I would be glad to see this feature supported in Dask as well - then the ML crowd could use Dask for problems that are too large to fit in memory but are not large enough to justify the overhead of managing a Spark cluster."}}