title: Can I use caret RFE result for subsequent random forest with CV?
body: <p>I've been googling and reading alot about my issue but couldn't find a clear answer.</p>
  
  <p>In order to prevent data leakage, I use caret RFE for feature elimination:</p>
  
  <pre><code>rfFuncs$summary &lt;- twoClassSummary
  #---- rfe control
  rfe_ctrl &lt;- rfeControl(functions = rfFuncs,
                         method = "repeatedcv",
                         number = 5, # folds
                         repeats = 3, # iterations
                         verbose = TRUE,
                         returnResamp = "all",
                         saveDetails = TRUE)
  
  #---- rfe
  set.seed(164)
  rfe_profile &lt;- rfe(df_train[,features_name], 
                    df_train[,target_name],
                    sizes = subsets,
                    rfeControl = rfe_ctrl,
                    metric = "ROC",
                    scale = TRUE)
  
  #---- select final features
  features_rfe &lt;- predictors(rfe_profile)[1:30]
  </code></pre>
  
  <p>This works well and tells me how many predictors I should use. Let's say I get the result that 60 features works best. However, according to plots and further analysis (e.g. <code>pickSizeTolerance</code>) I know I can use the first 30 features and still having good performance of predictions.</p>
  
  <p>Now, can I use these 30 features (=<code>features_rfe</code>) in a subsequent model training step (see code below) since I don't touch the test set until after this step? Or would I introduce data leakage with this step because the model gets trained with pre-selected best features?</p>
  
  <pre><code>#---- random forest control
  rf_ctrl &lt;- trainControl(method = "cv",
                          number = 5, # folds
                          verbose = TRUE,
                          returnResamp = "all",
                          classProbs = TRUE,
                          summaryFunction = twoClassSummary)
  
  #---- random forest grid search
  rf_grid &lt;-  expand.grid(mtry = c(1, 5, 10, 15, 20))
  
  #---- random forest
  set.seed(10)
  rf_caret_train &lt;- train(df_train[,features_rfe], 
                  df_train[,target_name], 
                  method = "rf", 
                  trControl = rf_ctrl,
                  tuneGrid = rf_grid,
                  metric = "ROC",
                  verbose = TRUE)
  #---- make predictions
  rf_caret_test_raw &lt;- predict(rf_caret_train, df_test[,features_rfe], type='raw', scale=TRUE)
  </code></pre>
  
  <p>I hope you cann help me.</p>
  
  <p>Best,
  Troji</p>