{"action":"created","issue":{"url":"https://api.github.com/repos/dask/dask-ml/issues/370","repository_url":"https://api.github.com/repos/dask/dask-ml","labels_url":"https://api.github.com/repos/dask/dask-ml/issues/370/labels{/name}","comments_url":"https://api.github.com/repos/dask/dask-ml/issues/370/comments","events_url":"https://api.github.com/repos/dask/dask-ml/issues/370/events","html_url":"https://github.com/dask/dask-ml/pull/370","id":361549391,"node_id":"MDExOlB1bGxSZXF1ZXN0MjE2NDkzNzgz","number":370,"title":"WIP: API: Incremental search naming changes","user":{"login":"stsievert","id":1320475,"node_id":"MDQ6VXNlcjEzMjA0NzU=","avatar_url":"https://avatars1.githubusercontent.com/u/1320475?v=4","gravatar_id":"","url":"https://api.github.com/users/stsievert","html_url":"https://github.com/stsievert","followers_url":"https://api.github.com/users/stsievert/followers","following_url":"https://api.github.com/users/stsievert/following{/other_user}","gists_url":"https://api.github.com/users/stsievert/gists{/gist_id}","starred_url":"https://api.github.com/users/stsievert/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stsievert/subscriptions","organizations_url":"https://api.github.com/users/stsievert/orgs","repos_url":"https://api.github.com/users/stsievert/repos","events_url":"https://api.github.com/users/stsievert/events{/privacy}","received_events_url":"https://api.github.com/users/stsievert/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2018-09-19T02:07:44Z","updated_at":"2018-09-20T23:31:33Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/dask/dask-ml/pulls/370","html_url":"https://github.com/dask/dask-ml/pull/370","diff_url":"https://github.com/dask/dask-ml/pull/370.diff","patch_url":"https://github.com/dask/dask-ml/pull/370.patch"},"body":"This adds \"CV\" to `BaseIncrementalSearch` and it's children. `BaseIncrementalSearchCV` does do some (very basic) cross validation.\r\n\r\nThis PR\r\n\r\n* rename `IncrementalSearch` to `DecaySearch`\r\n* renames `*Search` to `*SearchCV`\r\n* modifies the docs to use \"adaptive\" instead of \"incremental\"\r\n* renames `history_results_` to `history_`\r\n* selects best model by highest score at last partial fit call\r\n* adds `cv_results_` to `BaseAdaptiveSearchCV`.\r\n\r\n\r\n~~I also tried to come up with a better name for `IncrementalSearchCV`: it seems like a very general name (but I didn't think about it too hard).~~"},"comment":{"url":"https://api.github.com/repos/dask/dask-ml/issues/comments/423366706","html_url":"https://github.com/dask/dask-ml/pull/370#issuecomment-423366706","issue_url":"https://api.github.com/repos/dask/dask-ml/issues/370","id":423366706,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMzM2NjcwNg==","user":{"login":"stsievert","id":1320475,"node_id":"MDQ6VXNlcjEzMjA0NzU=","avatar_url":"https://avatars1.githubusercontent.com/u/1320475?v=4","gravatar_id":"","url":"https://api.github.com/users/stsievert","html_url":"https://github.com/stsievert","followers_url":"https://api.github.com/users/stsievert/followers","following_url":"https://api.github.com/users/stsievert/following{/other_user}","gists_url":"https://api.github.com/users/stsievert/gists{/gist_id}","starred_url":"https://api.github.com/users/stsievert/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stsievert/subscriptions","organizations_url":"https://api.github.com/users/stsievert/orgs","repos_url":"https://api.github.com/users/stsievert/repos","events_url":"https://api.github.com/users/stsievert/events{/privacy}","received_events_url":"https://api.github.com/users/stsievert/received_events","type":"User","site_admin":false},"created_at":"2018-09-20T23:31:33Z","updated_at":"2018-09-20T23:31:33Z","author_association":"CONTRIBUTOR","body":"\r\n> So going back to me not wanting CV suffix because we don't support the cv parameter. Do you have thoughts on stsievert? Is `Search` expressive enough on its own that we're doing hyperparmeter optimization\r\n\r\nI do not think `Search` is expressive enough; to me `*Search` indicates \"train and test on the same data\" and `*SearchCV` indicates \"train and test on different data\". This class does perform (basic) cross validation because it creates separate datasets to test and train on. This also underpins the naming of `cv_results_`.\r\n\r\nThe `cv` parameter specifies the amount of cross validation for a particular cross validation scheme, `KFold` (and does not specify whether or whether not to use cross validation). This class could also the vary the amount of cross validation with the something like\r\n\r\n``` python\r\nsearch = AdaptiveSearchCV()\r\ndatasets = [train_test_split(X, y) for _ in range(cv)]\r\nscores = [search.fit(X1, y1).score(X2, y2)\r\n          for X1, X2, y1, y2 in datasets]\r\n```\r\n\r\nthough I don't think this should be implemented right now (one train/test set is used in practice).\r\n\r\n> idle question: do we know that partial_fit_score is monotonically increasing for a given model through history? I don't think we can assume that later = higher (not saying that you do. Haven't looked at the code yet).\r\n\r\nCorrect – monotonically increasing scores is a very bad assumption (though that's almost the case when `patience=1` and `tol=0`).\r\n\r\nI think we should choose `best_estimator_` based off the maximum final score, not the maximum score after any `partial_fit_call`. Two cases where reporting the maximum score on any `partial_fit` breaks down:\r\n\r\n* maybe an estimator receives a high score on a particular `partial_fit` call because the same data is fed to `partial_fit` and `score`.\r\n* maybe a particular estimator overfits the data. At one point, this model could have had the best score for any model. However, it could converges to the lowest possible (finite) score.\r\n\r\nThe protection against `score` decreasing too much is to stop training on plateau, which is specified by the `patience` and `tol` parameters (though allowing some decrease of `score` may be required to reach a better solution).\r\n\r\nSome of this will be fixed by https://github.com/dask/dask-ml/pull/370#discussion_r219328288, but not all of it.\r\n\r\n>> selected the best model by highest score at last partial fit call\r\n>\r\n> I think that was already being done.\r\n\r\n`best_model_id` was selected to be the model returned by `_fit` without looking at any `score`s https://github.com/dask/dask-ml/pull/370/files#diff-0a54ded74c8013caf0588e9a5c879e99L487.\r\n"}}
{"actor":{"display_login":"stsievert"}}