In this youtube video about bootstrap resampling, the creator states that when the number of bootstrap processes is small, the distribution of the parameter being estimated can no longer be thought to have a normal distribution (see from the 8:50 mark).

Since the parameter's distribution is not normal, the standard deviation can't be obtained via the standard formula:

enter image description here

The author thus states that one needs to use the CDF to calculate where the 68% confidence interval values are and obtain from there an estimation of the standard deviation.

My question is: where can I get a source for this workaround for obtaining the standard deviation when the number of times the bootstrap process is repeated is small?

I'm interested in this because I'm using bootstrap with replacement to estimate the uncertainty of a parameter and I can't possibly repeat it thousands of times (not even dozens actually) because it would be impossibly time consuming.

  • $\begingroup$ In many case the whole point of using the bootstrap is that the sampling distribution of the statistic is unknown, so in general I would not expect that distribution to be normal (Gaussian) even with a large number of bootstrap replications. $\endgroup$ – Maarten Buis Feb 5 '14 at 16:28
  • $\begingroup$ @MaartenBuis so are you saying one should always use the CDF method to calculate the uncertainty? $\endgroup$ – Gabriel Feb 5 '14 at 16:53
  • 1
    $\begingroup$ "always" is a big word, but looking at the percentiles (with possible adjustments) for computing confidence intervals fits more naturally within the logic of the bootstrap. $\endgroup$ – Maarten Buis Feb 5 '14 at 17:17
  • $\begingroup$ from what I heard, isn't the bootstrap generally supposed to be done with replacement though? $\endgroup$ – user38457 Feb 5 '14 at 19:11

A few reactions that are too long for a comment:

Don't ask us for sources of what somebody claimed on youtube. Ask the creator of the video. The idea of using the normal quantiles to estimate the scale of the distribution obviously comes from robust statistics, and my vague recollection is that it may be due to Tukey. Note that with 20 bootstrap replicates, the 68% confidence interval is going to be far more lousy that the estimate of $s$ that you would get from the standard formula.

The formula for $s^2$ provides an unbiased estimator no matter whether the underlying distribution is normal or not (but it has to have the second moment, of course). What the normality affects is whether that $s^2$ (scaled by $\sigma^2$) will have a $\chi^2$ distribution (and hence the ratios with $s$ in the denominator will have a true $t$-distribution), but unless something really crazy is going on, the $t$-distributions are remarkably robust to non-normality.

To improve performance of the bootstrap with small number of replicates, one can consider balanced schemes that allows one to produce exactly the same estimates of variance that the complete perfect bootstrap would. Most people are just used to Monte Carlo versions of the bootstrap, so their hair may stand up on their heads when they would hear about deterministic bootstrap, but one should not forget that the distant goal is the complete bootstrap, and whatever gets us closer to it is helpful.


Your Answer

