Comments on: Most Findings Are False

By: The Perils of Hypothesis Testing … Again « Normal Deviate

The Perils of Hypothesis Testing … Again « Normal Deviate — Sun, 28 Apr 2013 02:17:04 +0000

[...] few months ago I posted about John Ioannidis’ article called “Why Most Published Research Findings Are [...]

By: Single factor asset pricing model with leverage shocks « Prof. Jayanth R. Varma’s Financial Markets Blog

Mon, 21 Jan 2013 06:11:16 +0000

[...] be false even if they are published in a top notch peer reviewed journal (see for example here or here). (I do recall the incident a couple of years ago when the Chen-Zhang q-factor papers became [...]

By: Linkage (technical) « An Ergodic Walk

Linkage (technical) « An Ergodic Walk — Thu, 03 Jan 2013 17:30:00 +0000

[...] seen a talk recently by John Ioannidis on how medical research is (often) bunk, this finer corrective by Larry Wasserman was nice to [...]

By: normaldeviate

normaldeviate — Mon, 31 Dec 2012 18:27:26 +0000

The Cox example of two measurements does not make CI’s nonsense.
And people (like me for example) do frequentist analyses
of massively complicated problems all the time.
But you and I will never agree on this so I suggest
we just agree to disagree.

Best wishes for a New year

By: Entsophy

Entsophy — Mon, 31 Dec 2012 18:19:25 +0000

I don’t think that’s it. If you’re combing measurements from different observatories with different instruments, weird things can happen which don’t show up in the usual text book “equal variance” case. The easiest way to see this is to consider one measurement which is far more accurate than the others (small variance). C.I.’s will become complete nonsense, while the naive Bayes answer will do exactly the right thing and automatically base the answer on the accurate data point. (incidentally, this is a real world example of that xkcd cartoon you railed against: one extremely accurate measurement effectively makes the others about as relevant as the two dice, but CI’s will nevertheless be dominated by them)

In a less extreme case this may not be so obvious and the only way Frequentist would detect the error is if they did a power calculation. Laplace avoided all this completely with a simple Bayesian calculation that never needs any special notice of the power. That’s why he wasn’t tripped up.

Things are much more complicated today. A while back Christian Robert posted a lecture about using Bayesian models for weather prediction. These models are extraordinarily complicated and combine staggering amounts of measurements and physics. It’s hard to imagine applying the arsenal of Frequentists methods to do something similar, but suppose for the sake of argument that there was an equivalent Frequentist model.

If you had a Frequentist version of this massive weather model, you’d have to do an impossible number of power calculations to ensure you weren’t falling into a more complication version of the trap described above! So I don’t see how using methods that would have already tripped up Laplace’s simple examples would help in any way whatsoever.

By: normaldeviate

normaldeviate — Mon, 31 Dec 2012 17:38:44 +0000

no I am just saying that statistics is much harder than in Laplace’s day

By: Entsophy

Entsophy — Mon, 31 Dec 2012 17:18:57 +0000

Are you saying that lots of measurements are a problem for statistics and liable to trip statisticians up?

By: normaldeviate

normaldeviate — Mon, 31 Dec 2012 16:00:42 +0000

Laplace didn’t have to deal with measurements
on 50,000 genes or 1,000,000,000 galaxies
as in the Sloan survey.

By: Entsophy

Entsophy — Mon, 31 Dec 2012 15:44:35 +0000

Yes Laplace used these kinds of ideas to detect signals in noise and applied them to Astronomy several hundred years ago. Oddly his published astronomical discoveries seemed to suffer nothing like the 50% error rate that we see today.

Just out of curiosity does anyone know how much time Laplace spent controlling the power of his tests and ensuring that he used powerful tests? Surely at the early date using a clearly bogus philosophy of statistics (Essai philosophique sur les probabilités 1814) he must have been tripped up by this all the time unless he spent a great deal of trouble avoiding that pitfall.

By: Keith O'Rourke

Keith O'Rourke — Mon, 31 Dec 2012 14:13:18 +0000

Rob, I agree we have a very big problem and I was surprised at the magnitude of the variation in results given arguably sensible variations in analysis methods.

I for one did actually realize the magnitude of the problem for selective publishing and reporting of RCTs (given the flak I initially encountered early in my work on meta-analysis methods for RCTs) and I have been continually disappointed at the slow pace of the adoption of any remedies for RCTs (e.g. trial registration).

I have been tracking this for some time http://andrewgelman.com/2012/02/meta-analysis-game-theory-and-incentives-to-do-replicable-research/

The only effective use of evidence for health care that I am aware of is in regulatory agencies where published studies are strictly interpreted as supportive (i.e. hear say) and only pre-agreed upon studies are taken seriously and sometimes even audited.

Unfortunately, remedies for observational studies are not so clear (as they are much harder to design/anticipate).