{"action":"created","issue":{"url":"https://api.github.com/repos/EpistasisLab/tpot/issues/727","repository_url":"https://api.github.com/repos/EpistasisLab/tpot","labels_url":"https://api.github.com/repos/EpistasisLab/tpot/issues/727/labels{/name}","comments_url":"https://api.github.com/repos/EpistasisLab/tpot/issues/727/comments","events_url":"https://api.github.com/repos/EpistasisLab/tpot/issues/727/events","html_url":"https://github.com/EpistasisLab/tpot/issues/727","id":340283698,"node_id":"MDU6SXNzdWUzNDAyODM2OTg=","number":727,"title":"Does cv option do anything when given a custom generator?","user":{"login":"sohrabtowfighi","id":7122606,"node_id":"MDQ6VXNlcjcxMjI2MDY=","avatar_url":"https://avatars3.githubusercontent.com/u/7122606?v=4","gravatar_id":"","url":"https://api.github.com/users/sohrabtowfighi","html_url":"https://github.com/sohrabtowfighi","followers_url":"https://api.github.com/users/sohrabtowfighi/followers","following_url":"https://api.github.com/users/sohrabtowfighi/following{/other_user}","gists_url":"https://api.github.com/users/sohrabtowfighi/gists{/gist_id}","starred_url":"https://api.github.com/users/sohrabtowfighi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sohrabtowfighi/subscriptions","organizations_url":"https://api.github.com/users/sohrabtowfighi/orgs","repos_url":"https://api.github.com/users/sohrabtowfighi/repos","events_url":"https://api.github.com/users/sohrabtowfighi/events{/privacy}","received_events_url":"https://api.github.com/users/sohrabtowfighi/received_events","type":"User","site_admin":false},"labels":[{"id":281818167,"node_id":"MDU6TGFiZWwyODE4MTgxNjc=","url":"https://api.github.com/repos/EpistasisLab/tpot/labels/question","name":"question","color":"cc317c","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-07-11T15:04:15Z","updated_at":"2018-07-13T14:38:39Z","closed_at":null,"author_association":"NONE","body":"I am examining the results of my output which make me question whether TPOT is actually using the type of cross validation I specified using the 'cv' parameter. Please hear me out.\r\n\r\nI am running a classification problem with the TPOTClassifier and I have set its 'cv' option to 'cv=LeaveOneOut()' so as to run leave one out cross validation internally. During the execution, my terminal read 'Generation 15 - Current best internal CV score: 0.7281553398058253'. I have set TPOT to periodically generate output and the output file reads' \r\n    \r\n    training_features, testing_features, training_target, testing_target = \\\r\n    train_test_split(features, tpot_data['target'].values, random_state=42)\r\n\r\n    # Score on the training set was:0.7281553398058253\r\n    exported_pipeline = make_pipeline(\r\n        FeatureAgglomeration(affinity=\"l1\", linkage=\"average\"),\r\n        DecisionTreeClassifier(criterion=\"gini\", max_depth=10, min_samples_leaf=5, min_samples_split=11)\r\n    )\r\n\r\nPlease notice that the output pipeline uses a train_test_split with default values, not a LeaveOneOut splitter. I am not familiar with TPOT internals so this issue is probably naive. Is it expected behaviour for the pipeline file generated by TPOT to use a train_test_split object, even though I specified cv=LeaveOneOut()?"},"comment":{"url":"https://api.github.com/repos/EpistasisLab/tpot/issues/comments/404852856","html_url":"https://github.com/EpistasisLab/tpot/issues/727#issuecomment-404852856","issue_url":"https://api.github.com/repos/EpistasisLab/tpot/issues/727","id":404852856,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNDg1Mjg1Ng==","user":{"login":"rhiever","id":1719223,"node_id":"MDQ6VXNlcjE3MTkyMjM=","avatar_url":"https://avatars0.githubusercontent.com/u/1719223?v=4","gravatar_id":"","url":"https://api.github.com/users/rhiever","html_url":"https://github.com/rhiever","followers_url":"https://api.github.com/users/rhiever/followers","following_url":"https://api.github.com/users/rhiever/following{/other_user}","gists_url":"https://api.github.com/users/rhiever/gists{/gist_id}","starred_url":"https://api.github.com/users/rhiever/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rhiever/subscriptions","organizations_url":"https://api.github.com/users/rhiever/orgs","repos_url":"https://api.github.com/users/rhiever/repos","events_url":"https://api.github.com/users/rhiever/events{/privacy}","received_events_url":"https://api.github.com/users/rhiever/received_events","type":"User","site_admin":false},"created_at":"2018-07-13T14:38:39Z","updated_at":"2018-07-13T14:38:39Z","author_association":"COLLABORATOR","body":"During the optimization process, TPOT passes whatever is in the `cv` parameter directly to the `cross_val_score` `cv` parameter. Thus, the 'training' score is from whatever cross-validation scheme you passed to the `cv` parameter.\r\n\r\nYou're correct that the generated Python code from the `export` function is based on a fixed template that uses `train_test_split` for the initial train/test split on the data. We don't have the `export` function generate code with a cross-validation scheme, so there is no configuration for that. You could add a cross-validation scheme yourself by, e.g.,\r\n\r\n```Python\r\ntraining_features, testing_features, training_target, testing_target = \\\r\ntrain_test_split(features, tpot_data['target'].values, random_state=42)\r\n\r\n# Score on the training set was:0.7281553398058253\r\nexported_pipeline = make_pipeline(\r\n    FeatureAgglomeration(affinity=\"l1\", linkage=\"average\"),\r\n    DecisionTreeClassifier(criterion=\"gini\", max_depth=10, min_samples_leaf=5, min_samples_split=11)\r\n)\r\n\r\ncross_val_score(exported_pipeline, training_features, training_target, cv=LeaveOneOut())\r\n```\r\n\r\nEssentially, the generated Python code is meant only to be a starting point for you to integrate the pipeline into your own ML workflow. The `exported_pipeline` (and related imports) are the important part."}}
{"actor":{"display_login":"rhiever"}}