title: R package (`caret`?) for nested time series cross validation
body: <p>I'm using <code>caret</code>'s <code>timeslice</code> method to do step-ahead cross validation on time series data. I was surprised to find that:</p>
  <ul>
  <li>the 'best' hyperparameters chosen by <code>caret</code> are those with the best average performance across all train/test splits, and similarly the reported performance is the average across all train/test splits based on these hyperparam values, and</li>
  <li><code>caret</code> trains a final model using all of the data available - which makes sense when <code>fixedWindow = TRUE</code> but perhaps not otherwise.</li>
  </ul>
  <p>My data are non-stationary, so I'd like hyperparameter tuning, performance reporting and final model training to be based on recent data so that:</p>
  <ul>
  <li>optimal hyperparameter values can change as underlying relationships change</li>
  <li>reported performance can account for the fact that the best hyperparam values may change across splits, and</li>
  <li>recent changes in underlying relationships are best captured when training the final model.</li>
  </ul>
  <p>I'm wondering if the best approach for my non-stationary data would follow an approach something like:</p>
  <ol>
  <li>split each training fold into a training subset and validation subset - using the validation subset to pick hyperparam values within each training fold.</li>
  <li>train on the entire training fold using the hyperparam values selected in (1)</li>
  <li>report performance based on whatever hyperparameter values were selected in (1), even though these may change from fold to fold</li>
  <li>The final model is trained, and hyperparameter values selected, based on steps (1) and (2) using the most recent data only.</li>
  </ol>
  <p>Having typed this up I've realised that I'm just describing nested CV for time series. My questions:</p>
  <ol>
  <li>is this best practice for training time series models when data are non-stationary?</li>
  <li>can <code>caret</code>, or another R package, do this?</li>
  </ol>