title: Inconsistent "best tune" and "Resampling results across tuning parameters" caret R package
body: <p>I am trying to create a model using Caret with a tune grid</p>
  
  <p>svmGrid &lt;-  expand.grid(C = c(0.0001,0.001,0.01,0.1,1,10,20,30,40,50,100))</p>
  
  <p>and then once again with a subset of this grid:</p>
  
  <p>svmGrid &lt;-  expand.grid(C = c(0.0001,0.001,0.01,0.1,1,10,20,30,40,50))</p>
  
  <p>The problem is that I get different "best tune" and "resampling results across tuning parameters", although the C parameter value that was chosen for the first tune grid, is also appear in the second tune grid.</p>
  
  <p>I also encounter these discrepancies when using different options for the sampling parameter and also when using different summaryFunction options in <strong>trainControl()</strong> </p>
  
  <p>Needless to say, since different best model is selected every time, it effects the prediction results on a test sets.</p>
  
  <p>Any one has a clue why is it happening?</p>
  
  <p><strong>Reproducible data set:</strong></p>
  
  <pre><code>library(caret)
  library(doMC)
  registerDoMC(cores = 8)
  
  set.seed(2969)
  imbal_train &lt;- twoClassSim(100, intercept = -20, linearVars = 20)
  imbal_test  &lt;- twoClassSim(100, intercept = -20, linearVars = 20)
  table(imbal_train$Class)
  </code></pre>
  
  <p><strong>Run using the first tune grid</strong> </p>
  
  <pre><code>svmGrid &lt;-  expand.grid(C = c(0.0001,0.001,0.01,0.1,1,10,20,30,40,50,100))
  
  up_fitControl = trainControl(method = "cv", number = 10 , savePredictions = TRUE, allowParallel = TRUE, sampling = "up", seeds = NA)
  
  
  set.seed(5627)
  up_inside &lt;- train(Class ~ ., data = imbal_train,
                     method = "svmLinear",
                     trControl = up_fitControl,
                     tuneGrid = svmGrid,
                     scale = FALSE)
  
  up_inside
  </code></pre>
  
  <p><strong>First run output:</strong></p>
  
  <pre><code>&gt; up_inside
  Support Vector Machines with Linear Kernel 
  
  100 samples
   25 predictors
    2 classes: 'Class1', 'Class2' 
  
  No pre-processing
  Resampling: Cross-Validated (10 fold) 
  Summary of sample sizes: 90, 91, 90, 90, 89, 90, ... 
  Addtional sampling using up-sampling
  
  Resampling results across tuning parameters:
  
    C      Accuracy   Kappa         Accuracy SD  Kappa SD 
    1e-04  0.7734343   0.252201364  0.1227632    0.3198165
    1e-03  0.8225253   0.396439198  0.1245455    0.3626456
    1e-02  0.7595960   0.116150973  0.1431780    0.3046825
    1e-01  0.7686869   0.051430454  0.1167093    0.2712062
    1e+00  0.7695960  -0.004261294  0.1162279    0.2190151
    1e+01  0.7093939   0.111852756  0.2030250    0.3810059
    2e+01  0.7195960   0.040458804  0.1932690    0.2580560
    3e+01  0.7195960   0.040458804  0.1932690    0.2580560
    4e+01  0.7195960   0.040458804  0.1932690    0.2580560
    5e+01  0.7195960   0.040458804  0.1932690    0.2580560
    1e+02  0.7195960   0.040458804  0.1932690    0.2580560
  
  Accuracy was used to select the optimal model using  the largest value.
  The final value used for the model was C = 0.001. 
  </code></pre>
  
  <p><strong>Run using the second tune grid</strong> </p>
  
  <pre><code>svmGrid &lt;-  expand.grid(C = c(0.0001,0.001,0.01,0.1,1,10,20,30,40,50))
  
  up_fitControl = trainControl(method = "cv", number = 10 , savePredictions = TRUE, allowParallel = TRUE, sampling = "up", seeds = NA)
  
  
  set.seed(5627)
  up_inside &lt;- train(Class ~ ., data = imbal_train,
                     method = "svmLinear",
                     trControl = up_fitControl,
                     tuneGrid = svmGrid,
                     scale = FALSE)
  
  up_inside
  </code></pre>
  
  <p><strong>Second run output:</strong></p>
  
  <pre><code>&gt; up_inside
  Support Vector Machines with Linear Kernel 
  
  100 samples
   25 predictors
    2 classes: 'Class1', 'Class2' 
  
  No pre-processing
  Resampling: Cross-Validated (10 fold) 
  Summary of sample sizes: 90, 91, 90, 90, 89, 90, ... 
  Addtional sampling using up-sampling
  
  Resampling results across tuning parameters:
  
    C      Accuracy   Kappa         Accuracy SD  Kappa SD 
    1e-04  0.8125253   0.392165694  0.13043060   0.3694786
    1e-03  0.8114141   0.375569633  0.12291273   0.3549978
    1e-02  0.7995960   0.205413345  0.06734882   0.2662161
    1e-01  0.7495960   0.017139266  0.09742161   0.2270128
    1e+00  0.7695960  -0.004261294  0.11622791   0.2190151
    1e+01  0.7093939   0.111852756  0.20302503   0.3810059
    2e+01  0.7195960   0.040458804  0.19326904   0.2580560
    3e+01  0.7195960   0.040458804  0.19326904   0.2580560
    4e+01  0.7195960   0.040458804  0.19326904   0.2580560
    5e+01  0.7195960   0.040458804  0.19326904   0.2580560
  
  Accuracy was used to select the optimal model using  the largest value.
  The final value used for the model was C = 1e-04.
  </code></pre>