title: Using decision tree regression and cross-validation in sklearn
body: <p>I am a novice in statistical methods so please excuse any naivety. I am having a problem understanding the execution of cross validation when using Decision tree regression from sklearn (e.g. DecisionTreeRegressor and RandomForestRegressor). My dataset varies from having multiple predictors (y = single dependent variable; X = multiple independent variables) to having a single predictor and consists of enough cases (> 10k). The following explanation applies for all cases. </p>
  
  <p>When fitting and scoring the regressors with the standard methods:</p>
  
  <pre><code>dt = DecisionTreeRegressor()
  rf = RandomForestRegressor()
  
  dt.fit(X,y)
  rf.fit(X,y)
  
  dt_score = dt.score(X,y)
  rf_score = rf.score(X,y)
  </code></pre>
  
  <p>The dt_score and rf_score returns promising R-squared values (> 0.7), however I am aware of the over-fitting properties of the DT and to lesser extent the RF. Therefore I tried to score the regressors with cross-validation (10 fold) to get a more true representation of the accuracy:</p>
  
  <pre><code>dt = DecisionTreeRegressor()
  rf = RandomForestRegressor()
  
  dt.fit(X,y)
  rf.fit(X,y)
  
  dt_scores = cross_val_score(dt, X, y, cv = 10)
  rf_scores = cross_val_score(rf, X, y, cv = 10) 
  
  dt_score = round(sum(dt_scores )/len(dt_scores ), 3)
  rf_score = round(sum(rf_scores )/len(rf_scores ), 3)
  </code></pre>
  
  <p>The results of this cross validation always returns negative values. I assume they are R squared values according to the sklearn guidlines: <strong><em>By default, the score computed at each CV iteration is the score method of the estimator</em></strong>  (the score method of both the regressors is R squared). The explanation given from the guidelines for the basic KFold cross validation is: <strong><em>Each fold is then used once as a validation while the k - 1 remaining folds form the training set.</em></strong></p>
  
  <p>How I understand this, when using 10 old cv, is: my dataset is split into 10 equal parts, for each part the remaining 9 parts are used for training (I am not sure if this is a fit operation or a score operation) and the remaining part is used for validation (not sure what is done for validation). These regressors are a complete "black box" for me, so I have no idea on how a tree is used for regression and where the cross validation gets its R square values from. </p>
  
  <p>So to summarize, I am struggling to understand how the cross validation can decrease the accuracy (R squared) so dramatically? Am I using the cross validation right for a regressor? Does it make sense to use cross validation for a decision tree regressor? Should I be using another cross-validation method?</p>
  
  <p>Thank you     </p>