Comments on: Bootstrapping and Subsampling: Part I

By: Comunidades tribais são mais violentas? O quão próxima é a distribuição normal? O papel do BNDES. | Análise Real

Tue, 05 Feb 2013 01:14:40 +0000

[...] Sobre bootstraping I e [...]

By: normaldeviate

normaldeviate — Tue, 29 Jan 2013 16:02:00 +0000

the original

By: Bootstrapping and Subsampling: Part II « Normal Deviate

Bootstrapping and Subsampling: Part II « Normal Deviate — Sun, 27 Jan 2013 14:02:09 +0000

[...] « Bootstrapping and Subsampling: Part I [...]

By: Martin Azizyan

Martin Azizyan — Wed, 23 Jan 2013 10:46:12 +0000

rj444, I believe the “original data” in the statement “drawing a sample of size n from Pn is the same as drawing n points with replacement from the original data” refers to the finite sample we were given, not the true distribution. This is not a statistical claim. It follows from the definitions of the empirical distribution and of sampling with replacement.

I think that’s what you were asking, but I’m sorry if I misunderstood the subject of your confusion.

By: Christian Hennig

Christian Hennig — Mon, 21 Jan 2013 23:11:57 +0000

n is not always large, though.

By: normaldeviate

normaldeviate — Mon, 21 Jan 2013 19:32:58 +0000

P_n is a very accurate estimate of P
This follows from standard empirical process theory
For example, in one-dimension, P( ||P_n – P||_infty > epsilon) < 2e^{-2n epsilon^2}

The issue is the behavior of T(P).

By: rj444

rj444 — Mon, 21 Jan 2013 19:28:25 +0000

Yes, I was confused by the comment – “A moment’s reflection should convince you that drawing a sample of size n from Pn is the same as drawing n points with replacement from the original data”. The adequacy of the Pn approximation is exactly what I have trouble convincing myself of even after a moment’s reflection, especially when n is relatively small and T(P) is relatively complex.

I find it interesting that Larry says “an example of a bootstrap failure is in the problem of estimating phylogenetic trees”. In practice this is one of the areas where boostrap is used most often.

By: Keith O'Rourke

Keith O'Rourke — Mon, 21 Jan 2013 18:12:10 +0000

I prefer to think of the bootstrap as sampling paths from the product space Pn(A)^n WITH replacement which is less efficient than without replacement, but that very quickly decreases with increasing “n”.

Makes it teachable as simple survey sampling, which worked well with undergrad class once. But the BCA correction stuff and why it fails for complex problems seemed way too hard to get accross.

Remember Efron wrote a paper in the early 2000 complaining most statisticians actual do bootstrapping incorrectly.

That included me, in that when the percentile intervals were _simmilar_ to the BCA intervals in applications, I (mis)thought they had no advantage (i.e. I forgot to think about the distribution of intervals over repeated applications).

By: Christian Hennig

Christian Hennig — Mon, 21 Jan 2013 15:35:28 +0000

I like bootstrap and use it occasionally. However, I think that an important issue with (nonparametric) bootstrap is that some features of P_n are essentially different from P (or our typical idea of P). P_n is basically discrete. This has implications. For example, in clustering, samples from P_n tend to produce larger between-cluster separation than the original sample, because points that could spoil separation can be taken away, but never added. Producing multiple points at the same location can lead to artifacts in clustering, but also in covariance-matrix estimation or group-wise covariance matrices in classification (implosion of eigenvalues). Samples from P_n (with or without multiple points) can have a smaller but not larger convex hull then the original samples etc. So I’d say that one needs to be very careful with bootstrapping statistics that can somehow be affected by discreteness, the convex hull and other problematic features somebody else comes up with.

By: Geoff Gordon

Geoff Gordon — Mon, 21 Jan 2013 15:32:25 +0000

I think one reason the bootstrap isn’t more popular is what Larry mentioned: “it is most useful in complex situations, but these are often the situations where the theory breaks down”. (There’s not much point in going to a lot of effort to get an error bound if the error bound is probably wrong.) Another is that the bootstrap requires re-running your estimation procedure a very large number of times, and often in ML we’re lucky if we can run the estimation procedure once.