title: CatBoost: Are we overfitting?
body: <p>Our team is currently using CatBoost to develop credit scoring models, and our current process is to...</p>
  <ol>
  <li>Sort the data chronologically for out-of-time sampling, and split it into train, valid, and test sets</li>
  <li>Perform feature engineering</li>
  <li>Perform feature selection and hyperparameter tuning (mainly learning rate) on train, using valid as an eval set for early stopping</li>
  <li>Perform hyperparameter tuning on the combination of train and valid, using test as an eval set for early stopping</li>
  <li>Evaluate the results of Step #4 using standard metics (RMSE, ROC AUC, etc.)</li>
  </ol>
  <p>However, I am concerned that we may be overfitting to the test set in Step #4.</p>
  <p>In Step #4, should we just be refitting the model on train and valid without tuning (i.e., using the selected features and hyperparameters from Step #3)?</p>
  <p>The motivation for having a Step #4 at all is to train the models on more recent data due to our out-of-time sampling scheme.</p>