{"action":"created","issue":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/7388","repository_url":"https://api.github.com/repos/scikit-learn/scikit-learn","labels_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/7388/labels{/name}","comments_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/7388/comments","events_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/7388/events","html_url":"https://github.com/scikit-learn/scikit-learn/pull/7388","id":176272358,"number":7388,"title":"[WIP] ENH Allow `cross_val_score`, `GridSearchCV` and `RandomizedSearchCV` to evaluate on multiple metrics","user":{"login":"raghavrv","id":9487348,"avatar_url":"https://avatars.githubusercontent.com/u/9487348?v=3","gravatar_id":"","url":"https://api.github.com/users/raghavrv","html_url":"https://github.com/raghavrv","followers_url":"https://api.github.com/users/raghavrv/followers","following_url":"https://api.github.com/users/raghavrv/following{/other_user}","gists_url":"https://api.github.com/users/raghavrv/gists{/gist_id}","starred_url":"https://api.github.com/users/raghavrv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/raghavrv/subscriptions","organizations_url":"https://api.github.com/users/raghavrv/orgs","repos_url":"https://api.github.com/users/raghavrv/repos","events_url":"https://api.github.com/users/raghavrv/events{/privacy}","received_events_url":"https://api.github.com/users/raghavrv/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":48,"created_at":"2016-09-11T23:39:29Z","updated_at":"2016-10-08T23:50:08Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/7388","html_url":"https://github.com/scikit-learn/scikit-learn/pull/7388","diff_url":"https://github.com/scikit-learn/scikit-learn/pull/7388.diff","patch_url":"https://github.com/scikit-learn/scikit-learn/pull/7388.patch"},"body":"Supercedes #2759 \r\nFixes #1837\r\n\r\n<hr>\r\n\r\n#### TODO\r\n\r\n- [x] Add utils function `check_multimetric_scoring` for validation of multimetric `scoring` param\r\n  - [x] Tests\r\n- [x] Support multiple metrics for `cross_val_score`\r\n  - [x] Tests\r\n- [x] Support multiple metrics for `validation_curve`\r\n  - [ ] Tests\r\n- [x] Support multiple metrics for `learning_curve` (Do we need it?)\r\n  - [ ] Tests\r\n- [x] Support multiple metrics for `GridSearchCV` and `RandomizedSearchCV`\r\n  - [x] Tests\r\n- [x] [Example on `GridSearchCV` and `cross_val_score`](https://github.com/raghavrv/scikit-learn/blob/multimetric_cross_val_score/examples/model_selection/multiple_metric_evaluation.py)\r\n- [ ] Example with plot for `validation_curve` (After finalizing API)?\r\n\r\n<hr>\r\n\r\n**Currently, in master**\r\n\r\n* `scoring` can only be a single string ('precision' etc) or a single callable (`make_scorer(precision_score)`, custom_scorer).\r\n\r\n**In this PR**\r\n\r\n* `scoring` can now be a list/tuple like `('precision', 'accuracy'...)` or a dict like `{'precision': make_scorer(precision_score), 'accuracy': make_scorer(accuracy), 'custom': custom_scorer}`\r\n* If (and only if) the `scoring` is of multimetric type, the return of `cross_val_score` / `learning_curve` / `validation_curve` will be dict mapping `scorer_names` to their corresponding `train_scores` or `test_scores`.\r\n* `GridSearchCV`'s attributes `best_index_`, `best_params_`, `best_score_` will be a dict if (and only if) `scoring` is a list/tuple/dict with the above format.\r\n* `GridSearchCV`'s `cv_result_` attribute will consist of keys ending with scorer names for multiple metrics...\r\n\r\n**Example Usage** (Links refer to the branch and not the commit, hence will be automatically updated)\r\n\r\n1. [Example on `GridSearchCV` and `cross_val_score`](https://github.com/raghavrv/scikit-learn/blob/multimetric_cross_val_score/examples/model_selection/multiple_metric_evaluation.py)\r\n2. [Example on `cross_val_score`](https://github.com/scikit-learn/scikit-learn/pull/7388#issuecomment-247565362)\r\n\r\n[`cross_val_score` docstring](https://github.com/scikit-learn/scikit-learn/pull/7388/files#diff-60033c11a662f460e1567effd5faa6f0R137)... (Link on the diff. May go stale)\r\n\r\n<hr>\r\n\r\n**Open Questions**\r\n\r\n0. Do we need the `cross_validate` function which returns something similar to the `cv_results_` attribute? If yes, do we still want `cross_val_score` to support multiple metrics?\r\n1. Do we need this for `permutation_test_score`?\r\n2. Do we need this for `learning_curve`?\r\n3. If we don't want to return dict for multiple metric and list/array for single metric, can we generate a named tuple and return that instead of the dict? (That can be indexed like a list of lists and also by the scorer name like an attribute but with restrictions in the naming of scorer)\r\n\r\n```python\r\nfrom collections import namedtuple\r\n\r\nscorer_names = sorted(('accuracy', 'precision'))\r\naccuracy_score = [0.5, 1, 0.9]\r\nprecision_score = [0.8, 0.9, 1]\r\n\r\n# As named tuple\r\nresult_tuple = namedtuple('cv_results_', sorted(scorer_names))\r\nres = result_tuple(accuracy_score, precision_score)\r\nprint(res.accuracy)\r\nprint(res[1])  # precision\r\n\r\n# As dict (allows more flexibility in scorer names)\r\nscorer_names = sorted(('accuracy.1 (foo)', 'precision'))\r\nres = dict(zip(scorer_names, (accuracy_score, precision_score)))\r\nprint(res['accuracy.1 (foo)'])\r\nprint(res['precision'])\r\n```\r\n\r\n*cc*: @jnothman @amueller @vene"},"comment":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/comments/252454868","html_url":"https://github.com/scikit-learn/scikit-learn/pull/7388#issuecomment-252454868","issue_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/7388","id":252454868,"user":{"login":"amueller","id":449558,"avatar_url":"https://avatars.githubusercontent.com/u/449558?v=3","gravatar_id":"","url":"https://api.github.com/users/amueller","html_url":"https://github.com/amueller","followers_url":"https://api.github.com/users/amueller/followers","following_url":"https://api.github.com/users/amueller/following{/other_user}","gists_url":"https://api.github.com/users/amueller/gists{/gist_id}","starred_url":"https://api.github.com/users/amueller/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amueller/subscriptions","organizations_url":"https://api.github.com/users/amueller/orgs","repos_url":"https://api.github.com/users/amueller/repos","events_url":"https://api.github.com/users/amueller/events{/privacy}","received_events_url":"https://api.github.com/users/amueller/received_events","type":"User","site_admin":false},"created_at":"2016-10-08T23:50:08Z","updated_at":"2016-10-08T23:50:08Z","body":"I was thinking more about something that makes a sensible DataFrame, though that is hard for 2d things like confusion matrices."}}
{"actor":{"display_login":"amueller"}}