title: glmnet / glmnetUtils: Repeated cross-validation
body: <p>I am trying to run repeated 10-fold CV (<code>alpha</code> and <code>lambda</code>) using <code>glmnet</code> / <code>glmnetUtils</code>. My proposed workflow is to:</p>
  
  <p>a) fit a proposed model at 11 values of <code>alpha</code>,</p>
  
  <p>b) run the process X (in this case, 10) times,</p>
  
  <p>c) average the results, and</p>
  
  <p>d) fit a final model with the best combination of <code>alpha</code> and <code>lambda</code> (<code>s = "lambda.1se"</code>).</p>
  
  <p>To address a-c, I used the code below; however, the results from the 10 iterations are exactly the same.</p>
  
  <pre><code>library(glmnet)
  library(glmnetUtils)
  library(doParallel)
  
  data(BinomialExample)
  
  
  # Create alpha sequence; fix folds
  
  alpha &lt;- seq(.5, 1, .05)
  
  set.seed(1)
  folds &lt;- sample(1:10, size = length(y), replace = TRUE)
  
  
  # Determine optimal combination of alpha and lambda; extract lowest CV error and associated lambda at each alpha
  
  extractGlmnetInfo &lt;- function(object)
  {
    # Find lambdas
    lambda1se &lt;- object$lambda.1se
  
    # Determine where lambdas fall in path
    which1se &lt;- which(object$lambda == lambda1se)
  
    # Create data frame with selected lambdas and corresponding error
    data.frame(lambda.1se = lambda1se, cv.1se = object$cvm[which1se])
  }
  
  
  #Run glmnet
  
  cl &lt;- makeCluster(detectCores())
  registerDoParallel(cl)
  
  enet &lt;- foreach(i = 1:10,
                  .inorder = FALSE,
                  .multicombine = TRUE,
                  .packages = "glmnetUtils") %dopar%
    {
      cv &lt;- cva.glmnet(x, y,
                       foldid = folds,
                       alpha = alpha,
                       family = "binomial",
                       parallel = TRUE)
      }
  
  stopCluster(cl)
  
  
  # Extract smallest CV error and lambda at each alpha for each iteration of 10-fold CV
  # Calculate means (across iterations) of lowest CV error and associated lambdas for each alpha
  
  cv.rep1 &lt;- ldply(enet[[1]]$modlist, extractGlmnetInfo)
  cv.rep2 &lt;- ldply(enet[[2]]$modlist, extractGlmnetInfo)
  cv.rep3 &lt;- ldply(enet[[3]]$modlist, extractGlmnetInfo)
  cv.rep4 &lt;- ldply(enet[[4]]$modlist, extractGlmnetInfo)
  cv.rep5 &lt;- ldply(enet[[5]]$modlist, extractGlmnetInfo)
  cv.rep6 &lt;- ldply(enet[[6]]$modlist, extractGlmnetInfo)
  cv.rep7 &lt;- ldply(enet[[7]]$modlist, extractGlmnetInfo)
  cv.rep8 &lt;- ldply(enet[[8]]$modlist, extractGlmnetInfo)
  cv.rep9 &lt;- ldply(enet[[9]]$modlist, extractGlmnetInfo)
  cv.rep10 &lt;- ldply(enet[[10]]$modlist, extractGlmnetInfo)
  
  cv.rep &lt;- bind_rows(cv.rep1, cv.rep2, cv.rep3, cv.rep4, cv.rep5, cv.rep6, cv.rep7, cv.rep8, cv.rep9, cv.rep10)
  
  cv.rep &lt;- data.frame(cbind(alpha, cv.rep))
  </code></pre>
  
  <p><strong>Questions</strong></p>
  
  <ol>
  <li><p>My understanding is that the folds should be fixed when cross-validating over <code>alpha</code>. Therefore, should I <code>set.seed()</code> multiple times to generate different <code>folds</code> for each iteration and run each iteration separately, rather than looping over them? For example:</p>
  
  <pre><code># Set folds for first iteration
  
  set.seed(1)
  folds1 &lt;- sample(1:10, size = length(y), replace = TRUE)
  
  
  # Run first iteration
  
  enet1 &lt;- cva.glmnet(x, y,
                  foldid = folds1,
                  alpha = alpha,
                  family = "binomial")
  
  
  # Set folds for second iteration
  
  set.seed(2)
  folds2 &lt;- sample(1:10, size = length(y), replace = TRUE)
  
  
  # Run second iteration
  
  enet2 &lt;- cva.glmnet(x, y,
                  foldid = folds2,
                  alpha = alpha,
                  family = "binomial")
  </code></pre></li>
  <li><p>Or is there a way to fix the <code>folds</code> and loop over the iterations, thereby making use of parallel processing?</p></li>
  <li><p>Re: the option presented in 1., how do I determine which configuration of <code>folds</code> I should use to fit the final model using the optimal combination of <code>alpha</code> and <code>lambda</code>? Is the decision arbitrary?</p></li>
  </ol>
  
  <p><strong>NB.</strong> I am not using <code>caret</code> for this specific task.</p>