And people (like me for example) do frequentist analyses

of massively complicated problems all the time.

But you and I will never agree on this so I suggest

we just agree to disagree.

Best wishes for a New year

]]>In a less extreme case this may not be so obvious and the only way Frequentist would detect the error is if they did a power calculation. Laplace avoided all this completely with a simple Bayesian calculation that never needs any special notice of the power. That’s why he wasn’t tripped up.

Things are much more complicated today. A while back Christian Robert posted a lecture about using Bayesian models for weather prediction. These models are extraordinarily complicated and combine staggering amounts of measurements and physics. It’s hard to imagine applying the arsenal of Frequentists methods to do something similar, but suppose for the sake of argument that there was an equivalent Frequentist model.

If you had a Frequentist version of this massive weather model, you’d have to do an impossible number of power calculations to ensure you weren’t falling into a more complication version of the trap described above! So I don’t see how using methods that would have already tripped up Laplace’s simple examples would help in any way whatsoever.

]]>on 50,000 genes or 1,000,000,000 galaxies

as in the Sloan survey. ]]>

Just out of curiosity does anyone know how much time Laplace spent controlling the power of his tests and ensuring that he used powerful tests? Surely at the early date using a clearly bogus philosophy of statistics (Essai philosophique sur les probabilités 1814) he must have been tripped up by this all the time unless he spent a great deal of trouble avoiding that pitfall.

]]>I for one did actually realize the magnitude of the problem for selective publishing and reporting of RCTs (given the flak I initially encountered early in my work on meta-analysis methods for RCTs) and I have been continually disappointed at the slow pace of the adoption of any remedies for RCTs (e.g. trial registration).

I have been tracking this for some time http://andrewgelman.com/2012/02/meta-analysis-game-theory-and-incentives-to-do-replicable-research/

The only effective use of evidence for health care that I am aware of is in regulatory agencies where published studies are strictly interpreted as supportive (i.e. hear say) and only pre-agreed upon studies are taken seriously and sometimes even audited.

Unfortunately, remedies for observational studies are not so clear (as they are much harder to design/anticipate).

]]>