{"action":"created","issue":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/8126","repository_url":"https://api.github.com/repos/scikit-learn/scikit-learn","labels_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/8126/labels{/name}","comments_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/8126/comments","events_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/8126/events","html_url":"https://github.com/scikit-learn/scikit-learn/issues/8126","id":197846221,"number":8126,"title":"Calculating recall takes too much time","user":{"login":"ndomar","id":3229567,"avatar_url":"https://avatars.githubusercontent.com/u/3229567?v=3","gravatar_id":"","url":"https://api.github.com/users/ndomar","html_url":"https://github.com/ndomar","followers_url":"https://api.github.com/users/ndomar/followers","following_url":"https://api.github.com/users/ndomar/following{/other_user}","gists_url":"https://api.github.com/users/ndomar/gists{/gist_id}","starred_url":"https://api.github.com/users/ndomar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ndomar/subscriptions","organizations_url":"https://api.github.com/users/ndomar/orgs","repos_url":"https://api.github.com/users/ndomar/repos","events_url":"https://api.github.com/users/ndomar/events{/privacy}","received_events_url":"https://api.github.com/users/ndomar/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-12-28T12:31:01Z","updated_at":"2016-12-29T02:36:27Z","closed_at":null,"body":"<!--\r\nIf your issue is a usage question, submit it here instead:\r\n- StackOverflow with the scikit-learn tag: http://stackoverflow.com/questions/tagged/scikit-learn\r\n- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn\r\nFor more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions\r\n-->\r\n\r\n<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->\r\n\r\n#### Description\r\n<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->\r\nIn some cases, it is useful to look only at the recall score without looking at the precision, fscore, and support. However whenever the recall_score method is called it will calculate the other 3 metrics anyways. Which makes the recall_score method consume much more time that it should. Sometimes the method takes about 10 seconds to return values while when I calculate only the recall on my own on the same data it takes about 0.7 seconds, returning the exact same value.\r\n\r\n\r\n\r\n#### Steps/Code to Reproduce\r\n<!--\r\nExample:\r\n```\r\nfrom sklearn.feature_extraction.text import CountVectorizer\r\nfrom sklearn.decomposition import LatentDirichletAllocation\r\n\r\ndocs = [\"Help I have a bug\" for i in range(1000)]\r\n\r\nvectorizer = CountVectorizer(input=docs, analyzer='word')\r\nlda_features = vectorizer.fit_transform(docs)\r\n\r\nlda_model = LatentDirichletAllocation(\r\n    n_topics=10,\r\n    learning_method='online',\r\n    evaluate_every=10,\r\n    n_jobs=4,\r\n)\r\nmodel = lda_model.fit(lda_features)\r\n```\r\nIf the code is too long, feel free to put it in a public gist and link\r\nit in the issue: https://gist.github.com\r\n-->\r\n```\r\nfrom sklearn.metrics import recall_score\r\n\r\nrecall = recall_score(actual, predictions, average='micro')\r\n```\r\n\r\n\r\n#### Versions\r\n<!--\r\nPlease run the following snippet and paste the output below.\r\nimport platform; print(platform.platform())\r\nimport sys; print(\"Python\", sys.version)\r\nimport numpy; print(\"NumPy\", numpy.__version__)\r\nimport scipy; print(\"SciPy\", scipy.__version__)\r\nimport sklearn; print(\"Scikit-Learn\", sklearn.__version__)\r\n-->\r\nDarwin-15.5.0-x86_64-i386-64bit\r\nPython 3.5.2 |Anaconda custom (x86_64)| (default, Jul  2 2016, 17:52:12) \r\n[GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)]\r\nNumPy 1.11.1\r\nSciPy 0.18.1\r\n\r\n\r\n<!-- Thanks for contributing! -->\r\n"},"comment":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/comments/269573459","html_url":"https://github.com/scikit-learn/scikit-learn/issues/8126#issuecomment-269573459","issue_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/8126","id":269573459,"user":{"login":"jnothman","id":78827,"avatar_url":"https://avatars.githubusercontent.com/u/78827?v=3","gravatar_id":"","url":"https://api.github.com/users/jnothman","html_url":"https://github.com/jnothman","followers_url":"https://api.github.com/users/jnothman/followers","following_url":"https://api.github.com/users/jnothman/following{/other_user}","gists_url":"https://api.github.com/users/jnothman/gists{/gist_id}","starred_url":"https://api.github.com/users/jnothman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jnothman/subscriptions","organizations_url":"https://api.github.com/users/jnothman/orgs","repos_url":"https://api.github.com/users/jnothman/repos","events_url":"https://api.github.com/users/jnothman/events{/privacy}","received_events_url":"https://api.github.com/users/jnothman/received_events","type":"User","site_admin":false},"created_at":"2016-12-29T02:36:27Z","updated_at":"2016-12-29T02:36:27Z","body":"Clearly, our code is not optimised for micro-averaging with a sparse large-scale multilabel problem using dense arrays. But the problem is certainly not in the calculation of P and F1. See benchmarks below, where your approach is faster in all cases (though there is some overhead in scikit-learn to handle various types of input), but is little faster than calculating all of P, R and F, and not substantially faster when using a sparse data structure.\r\n\r\nPlease update your issue description, and feel free to offer a patch.\r\n\r\n```python\r\ndef recall(y_true, y_pred):\r\n    n_true = y_true.sum()\r\n    nonzeros = y_pred.nonzero()\r\n    tp = y_pred[nonzeros].sum()\r\n    return tp / n_true\r\n\r\n\r\ndef prfs(y_true, y_pred):\r\n    n_true = y_true.sum()\r\n    n_pred = y_true.sum()\r\n    nonzeros = y_pred.nonzero()\r\n    tp = y_pred[nonzeros].sum()\r\n    return tp / n_true, tp / n_pred, 2 * tp / (n_true + n_pred), y_true.sum(axis=0)\r\n\r\n\r\nfrom scipy import sparse\r\nfrom sklearn.metrics import recall_score\r\n\r\n\r\nfor density in [.01, .1, .25, .5, .75, .9]:\r\n    y_true = np.random.rand(5551, 16890) < density\r\n    y_pred = np.random.rand(5551, 16890) < density\r\n    y_true_sparse = sparse.csr_matrix(y_true)\r\n    y_pred_sparse = sparse.csr_matrix(y_pred)\r\n\r\n    print('density =', density)\r\n    print('custom recall - dense')\r\n    %timeit recall(y_true, y_pred)\r\n    print('custom prfs - dense')\r\n    %timeit prfs(y_true, y_pred)\r\n    print('skl prfs - dense')\r\n    %timeit recall_score(y_true, y_pred, average='micro')\r\n    print('custom recall - sparse')\r\n    %timeit recall(y_true_sparse, y_pred_sparse)\r\n    print('custom prfs - sparse')\r\n    %timeit prfs(y_true_sparse, y_pred_sparse)\r\n    print('skl prfs - sparse')\r\n    %timeit recall_score(y_true_sparse, y_pred_sparse, average='micro')\r\n```\r\n\r\nResutlts look like:\r\n```\r\n              DENSE                      SPARSE\r\n        cust rec,cust prfs,sklprfs,cust rec,cust prfs,sklprfs,\r\n      [[  0.426,   0.649,   4.29 ,   0.041,   0.05 ,   0.084],\r\n       [  0.687,   0.902,   5.82 ,   0.577,   0.665,   0.855],\r\n       [  1.14 ,   1.35 ,   8.2  ,   1.55 ,   1.83 ,   2.23 ],\r\n       [  1.9  ,   2.07 ,  12.4  ,   3.38 ,   3.93 ,   4.72 ],\r\n       [  2.06 ,   2.26 ,  14.7  ,   5.16 ,   5.99 ,   7.18 ],\r\n       [  2.14 ,   2.37 ,  16.3  ,   6.2  ,   7.25 ,   8.55 ]]\r\n```\r\n\r\n"}}
{"actor":{"display_login":"jnothman"}}