{"action":"created","issue":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/10823","repository_url":"https://api.github.com/repos/scikit-learn/scikit-learn","labels_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/10823/labels{/name}","comments_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/10823/comments","events_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/10823/events","html_url":"https://github.com/scikit-learn/scikit-learn/issues/10823","id":306109024,"number":10823,"title":"Multi-metric scoring with pipelines repeats transform for each metric prediction","user":{"login":"alvinthai","id":19196039,"avatar_url":"https://avatars3.githubusercontent.com/u/19196039?v=4","gravatar_id":"","url":"https://api.github.com/users/alvinthai","html_url":"https://github.com/alvinthai","followers_url":"https://api.github.com/users/alvinthai/followers","following_url":"https://api.github.com/users/alvinthai/following{/other_user}","gists_url":"https://api.github.com/users/alvinthai/gists{/gist_id}","starred_url":"https://api.github.com/users/alvinthai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alvinthai/subscriptions","organizations_url":"https://api.github.com/users/alvinthai/orgs","repos_url":"https://api.github.com/users/alvinthai/repos","events_url":"https://api.github.com/users/alvinthai/events{/privacy}","received_events_url":"https://api.github.com/users/alvinthai/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-03-16T23:59:59Z","updated_at":"2018-05-22T21:33:08Z","closed_at":null,"author_association":"NONE","body":"<!--\r\nIf your issue is a usage question, submit it here instead:\r\n- StackOverflow with the scikit-learn tag: http://stackoverflow.com/questions/tagged/scikit-learn\r\n- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn\r\nFor more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions\r\n-->\r\n\r\n<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->\r\n\r\n#### Description\r\n<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->\r\n\r\nThis is related to issue #10802, multi-metric scoring is especially slow in the case of pipeline estimators. As @jimmywan points out, each scorer from the ``scoring`` dict is called because predictions are repeated.\r\n\r\nThe ``predict`` call in a pipeline calls the transformation every time a prediction is made. Since multi-metric scoring calls the ``predict`` function of the pipeline, the number of transform calls before the refit equal:\r\n\r\n> ``cv`` * 1 +  ``cv`` * (1 + ``return_train_score``) * ``len(scoring)`` \r\n\r\nThis total number THEN gets multiplied by the size of the parameter grid in a search.\r\n\r\nIt should be unnecessary to repeat the transform calls ``len(scoring)`` times, it can be expensive to repeat the exact same transformation on ``X_test`` and ``X_train`` each time ``predict`` or ``predict_proba`` is called. \r\n\r\nFor the case where ``return_train_score`` is not False, the original fit step already covers the initial ``X_train`` transformation, so there is a multiple of ``cv`` * ``len(scoring)`` extra calls to transform ``X_train`` added under the current implementation.\r\n\r\nMy suggestion would be to perform a fit_transform step under L475 for [_fit_and_score](https://github.com/scikit-learn/scikit-learn/blob/95c76318a6ef9e6733b974e8834c195ee9483b25/sklearn/model_selection/_validation.py#L360), and pass the transformed X_test and X_train datasets, along with the pipeline final estimator, to the scorers in L519 and L522 when pipeline estimators are encountered."},"comment":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/comments/391148012","html_url":"https://github.com/scikit-learn/scikit-learn/issues/10823#issuecomment-391148012","issue_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/10823","id":391148012,"user":{"login":"amueller","id":449558,"avatar_url":"https://avatars3.githubusercontent.com/u/449558?v=4","gravatar_id":"","url":"https://api.github.com/users/amueller","html_url":"https://github.com/amueller","followers_url":"https://api.github.com/users/amueller/followers","following_url":"https://api.github.com/users/amueller/following{/other_user}","gists_url":"https://api.github.com/users/amueller/gists{/gist_id}","starred_url":"https://api.github.com/users/amueller/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amueller/subscriptions","organizations_url":"https://api.github.com/users/amueller/orgs","repos_url":"https://api.github.com/users/amueller/repos","events_url":"https://api.github.com/users/amueller/events{/privacy}","received_events_url":"https://api.github.com/users/amueller/received_events","type":"User","site_admin":false},"created_at":"2018-05-22T21:33:08Z","updated_at":"2018-05-22T21:33:08Z","author_association":"MEMBER","body":"Pipeline.memory doesn't memoize transform? Should it? optionally?"}}
{"actor":{"display_login":"amueller"}}