The bootstrap method has seen a great diffusion in the last years, I also use it a lot, especially because the reasoning behind is quite intuitive.

But that's one thing I don't understand. Why Efron chose to perform resample with replace instead of simply subsampling by randomly including or excluding single observations?

I think that random subsampling has one very good quality, that is represent ideally the real life situation in which the observations we have in our study are a subset of an hypothetical population. I don't see the advantage of having multiplied observations during resampling. In a real context no observation is similar to another, especially for complex multivariate situations.

  • 2
    $\begingroup$ resampling with resampling is done because that is the right thing to do, given the model. The model behind the bootstrap is to use nonparametric maximum likelihood to estimate the cumulative distribution function, then sampling independent observations from the estimated cumulative distribution function. Think about it---algoritmically, that is obtained by sampling by replacement from the original sample. $\endgroup$ – kjetil b halvorsen Sep 7 '15 at 14:10

One way to understand this choice is to think of the sample at hand as being the best representation you have of the underlying population. You may not have the whole population to sample from any more, but you do have this particular representation of the population. A truly random re-sample from this representation of the population means that you must sample with replacement, otherwise your later sampling would depend on the results of your initial sampling. The presence of a repeated case in a particular bootstrap sample represents members of the underlying population that have characteristics close to those of that particular repeated case. Leave-one-out or leave-several-out approaches, as you suggest, can also be used but that's cross validation rather than bootstrapping.

I think this pretty much just puts into other words the comment from @kjetil_b_halvorsen

  • $\begingroup$ I understand the point. Making individual observations in a bootstrap sample independent from each other. In literature do exist methods based on subsampling, see Politis, Romano, Wolf. The use a fixed subset m of n, chosen without replacement. How do they avoid the pitfall you said before? In their case again I don't understand why they use a fixed size subsample instead of random subsample. $\endgroup$ – Bakaburg Sep 7 '15 at 15:08
  • 1
    $\begingroup$ Subsampling methods are trying to accomplish something different from the bootstrap. Those methods are seeking to select random subsets from the data sample rather than trying to emulate a new random sample from the underlying population. It's not that one or the other is wrong; they are different approaches that have particular strengths and weaknesses. $\endgroup$ – EdM Sep 7 '15 at 15:14
  • $\begingroup$ So maybe I should ask a new question regarding the difference between the two method in inference statistics. thanks! $\endgroup$ – Bakaburg Sep 7 '15 at 15:44
  • $\begingroup$ @Bakaburg see this question for a superb introduction into the literature on bootstrapping versus cross-validation (which is a particular type of subsampling). $\endgroup$ – EdM Sep 7 '15 at 16:24
  • $\begingroup$ @Bakaburg The bootstrap method is simulating the repeated independent drawing of random samples of size n (not a subset smaller than n) from a larger population. This means it's conceivable a random sample would contain a large number of extreme small or large values from the parent population which are often under-represented in our original sample. As EdM pointed out, resampling w/ replacement allows a single sample observation to "represent" multiple observations in the population which have similar values - it's a way to obtain a smooth approximation of the population distribution. $\endgroup$ – RobertF Sep 4 '18 at 16:18

Your Answer

