title: Caret - different results using train(), predict() and resamples()
body: <p>I'm using the Caret package to analyse various models and I'm assessing the results using:</p>
  
  <ul>
  <li>print() [print the results of train()], </li>
  <li>predict(), and</li>
  <li>resamples().</li>
  </ul>
  
  <p>Why are these results in the following example different?</p>
  
  <p>I'm interested in sensitivity (true positives).
  Why is J48_fit assessed as having a sensitivity of .71, then .81, then .71 again</p>
  
  <p>The same happens when I run other models - the sensitivity changes depending on the assessment.</p>
  
  <p>NB: I have included two models here so as to illustrate the resamples() function, which has to take two models as input, but my main question is about the differences between the results depending on what method one uses. </p>
  
  <p>In other words, what is the difference between the outcome of train() (C5.0_fit/J48_fit), predict() and resamples()?
  What is going on 'behind the scenes' and which result should I trust?</p>
  
  <p>EXAMPLE:</p>
  
  <pre><code>library(C50)
  data(churn)
  
  Seed &lt;- 10
  
  # Set train options
  set.seed(Seed)
  Train_options &lt;- trainControl(method = "cv", number = 10,
                                classProbs = TRUE,
                                summaryFunction = twoClassSummary)
  
  # C5.0 model:
  set.seed(Seed)
  C5.0_fit &lt;- train(churn~., data=churnTrain, method="C5.0", metric="ROC",
                   trControl=Train_options)
  
  # J48 model:
  set.seed(Seed)
  J48_fit &lt;- train(churn~., data=churnTrain, method="J48", metric="ROC",
                   trControl=Train_options)
  # Get results by printing the outcome
  print(J48_fit)
  
  #                      ROC Sens Spec
  # Best (sensitivity): 0.87 0.71 0.98  
  
  # Get results using predict()
  set.seed(Seed)
  J48_fit_predict &lt;- predict(J48_fit, churnTrain)
  confusionMatrix(J48_fit_predict, churnTrain$churn)
  #             Reference
  # Prediction  yes   no
  #       yes  389    14
  #       no    94  2836
  # Sens : 0.81          
  # Spec : 0.99
  
  # Get results by comparing algorithms with resamples()
  set.seed(Seed)
  results &lt;- resamples(list(C5.0_fit=C5.0_fit, J48_fit=J48_fit))
  summary(results)
  # ROC         mean
  # C5.0_fit    0.92  
  # J48_fit     0.87
  # Sens        mean
  # C5.0_fit    0.76  
  # J48_fit     0.71
  # Spec        mean
  # C5.0_fit    0.99  
  # J48_fit     0.98
  </code></pre>
  
  <p>By the way, here is a function for getting all three results together:</p>
  
  <pre><code>Get_results &lt;- function(...){
  
    Args &lt;- list(...)
    Model_names &lt;- as.list(sapply(substitute({...})[-1], deparse))
  
    message("Model names:")
    print(Model_names)
  
    # Function for getting max sensitivity
    Max_sens &lt;- function(df, colname = "results"){
      df &lt;- df[[colname]]
      new_df &lt;- df[which.max(df$Sens), ]
      x &lt;- sapply(new_df, is.numeric)
      new_df[, x] &lt;- round(new_df[, x], 2)
      new_df
    }
  
    # Find max Sens for each model
    message("Max sensitivity from model printout:")
    Max_sens_out &lt;- lapply(Args, Max_sens)
    names(Max_sens_out) &lt;- Model_names
    print(Max_sens_out)
  
    # Find predict() result for each model
    message("Results using predict():")
    set.seed(Seed)
    Predict_out &lt;- lapply(Args, function(x) predict(x, churnTrain))
    Predict_results &lt;- lapply(Predict_out, function(x) confusionMatrix(x, churnTrain$churn))
    names(Predict_results) &lt;- Model_names
    print(Predict_results)
  
    # Find resamples() results for each model
  
    message("Results using resamples():")
    set.seed(Seed)
    results &lt;- resamples(list(...),modelNames = Model_names)
    # names(results) &lt;- Model_names
    summary(results)
  
  }
  
  # Test
  Get_results(C5.0_fit, J48_fit)
  </code></pre>
  
  <p>Many thanks!</p>