Suppose we have a data set $X$ that is split as $X_{\text{train}}$, $X_{\text{val}}$ and $X_{\text{test}}$ and the outcome variable is binary. Let's say we train three different models (logistic regression, random forest, and a support vector machine) using $X_{\text{train}}$. We then get predictions for $X_{\text{val}}$ using each of the three models.

In stacking, is it correct to say that we train a logistic regression model on a data set of dimension $|X_{\text{val}}| \times 3$ with the predicted values and actual values of the validation set? This logistic regression model is then used to predict outcomes for data in $X_{\text{test}}$?

  • $\begingroup$ By stacking, do you mean ensemble learning? $\endgroup$ – nbro Apr 28 '19 at 12:25
  • $\begingroup$ Do you mind if I edit the title of your question to be a bit more general so it is more clear the context of the post? $\endgroup$ – Hanzy Apr 28 '19 at 13:59

The predictions of each of your initial models will become a feature to feed the meta learner. If you use $n$ initial models, then for each example you will feed the meta learner $n$ features, each feature being a prediction from one of the initial models.

Note that this doesn’t mean the size of your dataset increases. Instead, each member of $X_{val}$ will be represented by $n$ features, with the $k$th feature being equal to the initial prediction of the $k$th initial predictor.

This medium link recommends a package that can help you build a pipeline.

Note: in a sense if you are caching the intermediate predictions of each initial predictor, you’ll end up with $n$ times as many data points as there are entries in $X_{val}$. But these are simply each going to be a feature of another data point, so the dataset hasn’t really increased in size.

| improve this answer | |

Your Answer

