{"comment":{"url":"https://api.github.com/repos/jcrist/dask-learn/comments/17980763","html_url":"https://github.com/jcrist/dask-learn/commit/77483bb1e87b41c61b33a5b8033579597a0789dd#commitcomment-17980763","id":17980763,"user":{"login":"jcrist","id":2783717,"avatar_url":"https://avatars.githubusercontent.com/u/2783717?v=3","gravatar_id":"","url":"https://api.github.com/users/jcrist","html_url":"https://github.com/jcrist","followers_url":"https://api.github.com/users/jcrist/followers","following_url":"https://api.github.com/users/jcrist/following{/other_user}","gists_url":"https://api.github.com/users/jcrist/gists{/gist_id}","starred_url":"https://api.github.com/users/jcrist/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jcrist/subscriptions","organizations_url":"https://api.github.com/users/jcrist/orgs","repos_url":"https://api.github.com/users/jcrist/repos","events_url":"https://api.github.com/users/jcrist/events{/privacy}","received_events_url":"https://api.github.com/users/jcrist/received_events","type":"User","site_admin":false},"position":162,"line":162,"path":"dklearn/cross_validation.py","commit_id":"77483bb1e87b41c61b33a5b8033579597a0789dd","created_at":"2016-06-23T02:36:00Z","updated_at":"2016-06-23T02:36:00Z","body":"The scikit-learn api yields out indices to select from data, we yield out collections that are already selected. Data dependencies and the resulting graphs are virtually identical, but this is easier to do for more cross-validation strategies.\r\n\r\n> How is this used in practice?\r\n\r\nNew api point for https://github.com/jcrist/dask-learn/blob/master/dklearn/grid_search.py#L122.\r\n\r\nWill common use involve multiple reads of the same data?\r\n\r\nWithout shuffling (chunks are in same order for all cv sets), it should still result in only a single pass over the data with minimal data kept for long periods in the scheduler cache (only test subsets are depended on later, and that's unavoidable. Test sets should also be small ideally). Shuffling poses a problem, because one cv set could have chunk 1000 first, and another could have it last - the scheduler would keep it around for the entire time (which is why shuffling isn't implemented)."}}
{"actor":{"display_login":"jcrist"}}