title: Custom parameter tuning for KNN in caret
body: <p>I have a k nearest neighbors implementation that let me compute in a single pass predictions for multiple values of k and for multiple subset of training and test data (e.g. all the folds in the K-fold cross validation, AKA resampling metrics). My implementation can also leverage multiple cores. </p>
  
  <p>I would like to interface my method to be used with the caret package. I can easily build custom method for the train function. But this will result in multiple calls to the model fit (one for each parameter and fold combinations).</p>
  
  <p>As far as I know, I can't indicate tuning strategies when using trainControl. The code source of train mention something about "seq" model fitting :</p>
  
  <pre><code>## There are two types of methods to build the models: "basic" means that each tuning parameter
  ## combination requires it's own model fit and "seq" where a single model fit can be used to
  ## get predictions for multiple tuning parameters.
  </code></pre>
  
  <p>But I can't see any way to actually use that with custom models.</p>
  
  <p>Any clue on how to approach this ? </p>
  
  <p>More generally, suppose that you have a model class where you can estimate prediction errors across multiple parameters using a single model fit (e.g. ala <a href="http://notesofastatisticswatcher.wordpress.com/2012/12/18/linear-regression-loocv-trick/" rel="nofollow">Linear Regression LOOCV Trick</a> but for multiple parameter values too), how would you interface it in caret? </p>
  
  <p>Here's some example code to set up a (empty) custom model in caret: </p>
  
  <pre><code># Custom caret
  library(caret)
  learning_data = data.frame(y=sample(c("one","two","three"),200,replace=T))
  learning_data = cbind(learning_data,matrix(runif(3*200),ncol=3))
  testRatio=0.75
  inTrain &lt;- createDataPartition(learning_data$y, p = testRatio, list = FALSE)
  trainExpr &lt;- learning_data[inTrain,]
  testExpr &lt;- learning_data[-inTrain,]
  
  trainClass &lt;- trainExpr$y
  testClass &lt;- testExpr$y
  
  trainExpr$y&lt;-NULL
  testExpr$y&lt;-NULL
  cv_opts = trainControl(method="cv", number=4,verboseIter=T)
  
  my_knn &lt;- function(data,weight,parameter,levels,last,...){
          print("training")
          # print(dim(data))
          # str(parameter)
          # list(fit=rdist(data$,data))
          list(fit=NA)
  }
  my_knn_pred &lt;- function(object,newdata){
      print("testing")
      # str(object)
      # print(dim(newdata))
      return("one")
  }
  
  sortFunc &lt;- function(x)  x[order(x$k),]
  # Values of K to test
  knn_opts = data.frame(.k=c(seq(7,11, 2))) #odd to avoid ties
  custom_tr = trainControl(method="cv", number=4,verboseIter=T,   custom=list(parameters=knn_opts,model=my_knn,prediction=my_knn_pred,probability=NULL,sort=sortFunc))
  
  # This will result in 12 calls, 6 to my_knn, 6 to my_knn_pred, one per combination of fold and parameter value
  custom_knn_performances &lt;- train(x = trainExpr, y = trainClass,method = "custom",trControl=custom_tr,tuneGrid=knn_opts)
  </code></pre>
  
  <p>I would like to control the training procedure so as to generate predictions for all folds and parameter values in a single call. </p>