## anti xkcd

anti xkcd

Some of you may have noticed that the recent installment of the always entertaining web comic, xkcd,
had a statistical theme with a decidedly anti-frequentist flavor: see here.

In the interest of balance, here is my
(admittedly crude) attempt at an xkcd style comic.
Right back at you Randall Munroe!

1. Posted November 9, 2012 at 10:58 pm | Permalink

That’s pretty good actually! Very much in the XKCD style 😉

2. Posted November 10, 2012 at 2:33 am | Permalink

But if the “experiments” are independent and you conduct N of them so N bayesian 95% “conf. intervals” are created, the probability of none of them contain the “true value” is (1-0.95)^N=0.05^N which asymptotically is zero for N->inf

Isn’t it so? 🙂

• Posted November 10, 2012 at 9:38 am | Permalink

No! That’s the point. Bayesian intervals have no coverage guarantees.
A Bayesian 95 percent interval does not cover with frequency probability .95.

• Posted November 10, 2012 at 11:04 am | Permalink

Oh, you mean that in the Bayesian settings the posteriors are always conditioned on the data so that there are no frequency-based guarantees (irrespective of the sample)? thanks, Stelios

• Markku K.
Posted November 13, 2012 at 3:39 am | Permalink

I think in a sense they have coverage guarantees: 95 % credibility interval means by the definition that the true value belongs to this set by 95 % probability. Or rather, when the true value somehow realizes, it will come from this set by 95 % probability. This probability is based on your best knowledge. The fact that you expect the true value to come out from somewhere else reveals that you do not believe in your analysis yourself: either you believe that the data are biased (in which case you have a misspecified sampling model), or you believe that the true value should, somehow, be something else (in which case you have a wrong prior). I.e. if you don’t trust your interval, that shows that you have done something wrong; of course I acknowledge that it may by very difficult or indeed impossible to do everything right.

• Posted November 13, 2012 at 8:27 am | Permalink

I will do a whole post on this since, from the comments, it seems

LW

• bayesrules
Posted November 13, 2012 at 8:41 am | Permalink

Larry’s right. A 95% credible interval does not necessarily have 95% coverage. It could have a lot more, or it could have a lot less. In the case of the Berger-Mossman article, since Mossman was planning to publish in a journal and field where coverage is considered important, they designed their objective Bayes procedure so that it would have approximately 95% coverage. But that was by design; it doesn’t have to be that way.

3. Posted November 10, 2012 at 4:06 am | Permalink

Eh, I cannot really vocalize it, but I find myself preferring the xkcd strip…

4. gwern
Posted November 10, 2012 at 12:59 pm | Permalink

Frequentist particle mass 95% CIs don’t contain the true mass 95% of the time either: http://hanson.gmu.edu/temp/Henrion-Fischhoff-AmJPhysics-86.pdf

• Posted November 10, 2012 at 1:11 pm | Permalink

then it’s not a confidence interval

that’s the definition

• gwern
Posted November 10, 2012 at 1:13 pm | Permalink

Thank you for demonstrating my point.

5. Posted November 10, 2012 at 2:09 pm | Permalink

Excellent cartoon! And I notice that it actually relates to Bayesian methods supported by at least some/many subjectivists. Still, while a necessary condition, I would deny that good long-run coverage is all that is required for a satisfactory frequentist inference (I know you agree).

• Posted November 10, 2012 at 2:34 pm | Permalink

Thanks. Yes I do indeed agree.

6. Christian Hennig
Posted November 11, 2012 at 6:40 pm | Permalink

Well, according to de Finetti there is no such thing as a “true parameter”. So de Finetti would ask how the first character could ever find out that the true parameter is elsewhere… (and why he’d pay money to a Bayesian if he has a method to find the truth.)

• Posted November 11, 2012 at 7:00 pm | Permalink

That’s true. But DeFinetti is a bit too “post-modern” for me.
Call me old fashioned, but I think truth exists.

7. Christian Hennig
Posted November 11, 2012 at 7:23 pm | Permalink

De Finetti would agree that truth exists when it comes to observables.

• Posted November 11, 2012 at 7:34 pm | Permalink

ok
for DeFinetti’s sake, replace “confidence interval” with
“prediction interval” in the comic.

8. Rhiannon Weaver
Posted November 12, 2012 at 7:40 am | Permalink

I’ve been trying to figure out what the fallacy is in the xkcd comic, aside maybe from only giving the frequentist one data point. The only thing I can think is that it’s a poorly designed experiment. I mean, frequentists use conditional/hierarchical models and have no problems with inference in light of false positives/false negatives, or empirical population proportions, etc. But I guess part of the point is that you have to inject some form of repeatability before limiting relative frequency makes sense.

• Paul
Posted November 12, 2012 at 2:50 pm | Permalink

The fallacy is that the null hypothesis tested was not “the sun has not gone nova” but rather “the sun has not gone nova and two dice have not come up sixes.” So the correct conclusion after rejecting this null hypothesis is “the sun has gone nova or two dice have come up sixes”, not “the sun has gone nova.”

• JohnDoe
Posted November 12, 2012 at 9:57 pm | Permalink

Paul, that’s not right. One can test the hypothesis that the sun has gone nova this way if one likes, but the test has no power beyond the Type I error rate, which makes it a crappy basis for inference. By any common-sense measure, this approach is of course very very bad, but by the usual (frequentist) standards that introductory texts actually explicitly set for tests (i.e. do they control the Type I error rate) it passes. So, the only fallacious thinking is that control of the Type I error rate, alone, is sufficient to make for an adequate statistical procedure.

NB the same ‘test’ could be used in situations where repeatability is possible, e.g. in a clinical trial of a new drug.

• Paul
Posted November 13, 2012 at 12:01 pm | Permalink

JohnDoe — I think we are mostly in agreement. (?)

• Kaplan
Posted November 13, 2012 at 12:02 pm | Permalink

JohnDoe, I’m not sure what you say about the power is right either! Suppose that getting a “Yes” from the machine is the criterion for rejecting “H0: the sun has not gone nova”. If H0 is true then H0 is incorrectly rejected with probability 1/36 (the size of the test). On the other hand, if the alternative hypothesis “H1: the sun has gone nova” is true then H0 is rejected with probability 35/36. i.e. the frequentist test is powerful and on these grounds seems quite reasonable.

• JohnDoe
Posted November 14, 2012 at 1:35 am | Permalink

@Paul; your hypothesis includes statements about dice. This is not what the cartoon is illustrating.

@Kaplan; the test only ever rejects H0 with probability 1/36. Its power is therefore 1/36, not 35/36.

• Kaplan
Posted November 14, 2012 at 4:32 am | Permalink

@JohnDoe: that’s not so. The cartoon says “Then, it rolls two dice. If they both come up six then it lies to us. Otherwise it tells us truth”. So, if the sun has indeed gone nova then it correctly rejects H0 w.p. 35/36.

Had the cartoon said “Then, it rolls two dice. If they both come up six then the machine says ‘YES’, otherwise it says ‘NO’ ” then I would agree with you. But, as worded in the cartoon, since the machine tells the truth w.p. 35/36, a test that rejects H0 when the machine says ‘YES’ has size 1/36 and power 35/36 – quite reasonable?!

• JohnDoe
Posted November 14, 2012 at 12:06 pm | Permalink

@Kaplan. Sorry, I stand corrected.

9. Posted November 12, 2012 at 3:13 pm | Permalink

It’s helpful to move this away from philosophy and make a technical claim. Suppose you use the usual model y=m+error for the mass of neutrino and you collect some data. When you compute the 95% CI with this data one of two things will happen: the interval will contain the true value or it won’t. Consider both cases separately,

95% CI CONTAINS THE TRUE VALUE: using the same data the Bayesian will get exactly the same result with a uniform prior on the whole real line. On the other hand, if the Bayesian uses a uniform prior over an interval [a,b] and the true mass lies in this interval, than the new 95% Bayesian interval will be even smaller than the 95% CI and still contain the true length!

Moreover, it’s not difficult to find such an [a,b] in real life. For example, you could have used [0, “mass of a hydrogen atom”] since it is probably known before any direct measurement that the mass of the neutrino lies in this interval.

95% CI DOESN’T CONTAIN THE TRUE VALUE: In this case the measurement errors are such that they’re misleading and pull the interval away from the true value. When the Bayesian repeats the calculation using a uniform prior on [a,b] they will find that some of the ‘truth’ in this prior will correct some of the ‘falsity’ in the error model and drag the interval back closer to the true value. In some cases this effect will be big enough to drag the 95% Bayesian back onto the true value.

In either case including true prior information gets us an interval that demonstratedly brings us closer to the truth about the mass of the neutrino. So if your goal in life is to be wrong 5% of the time, then 95% CIs are the way to go. If you simply want to know the mass of the neutrino than do the Bayesian calculation.

• Posted November 12, 2012 at 3:52 pm | Permalink

Incidentally, I might add that for a 95% CI to have actual 95% coverage you’d need to know something like “the errors over a very long sequence of trials have a histogram that looks the assumed probability distribution for the errors”

This assumption fails in real physical experiments quite a bit which is why the 95% CI’s don’t actually have 95% coverage very often.

On the hand every point I make in the above comment still holds true regardless of what the long range histogram of errors looks like.

• Paul
Posted November 12, 2012 at 9:59 pm | Permalink

The experiments to measure neutrino mass actually measure mass-squared. And one interesting thing so far is that the 95% confidence intervals extend to negative values.

• Rhiannon Weaver
Posted November 13, 2012 at 9:48 am | Permalink

Then it is a poorly constructed interval likely based on asymptotics that have boundary issues? Are the intervals driving the theory at that point or is it the other way around?

10. bayesrules
Posted November 12, 2012 at 9:03 pm | Permalink

I don’t understand the problem here. Sure, if you just choose a prior out of the blue (pull it out of your proverbial ass, or even construct it based on prior evidence) there’s no reason to expect anything about the frequentist coverage.

But some objective Bayesian procedures produce credible intervals that have good coverage properties, see for example the paper by Berger and Mossman:

http://www.stat.duke.edu/~berger/papers/mossman.pdf

So it is misleading to imply that Bayesian procedures generally have bad coverage (as the cartoon implies). It all depends on how they are conducted, ISTM.

• Anonymous
Posted November 12, 2012 at 11:03 pm | Permalink

If the method doesn’t take into account prior information, why then to use a Bayesian method? (Taking into account that bayesian methods are usually computationally more expansive than frequentist ones)

• bayesrules
Posted November 12, 2012 at 11:07 pm | Permalink

Read the paper. The objective Bayesian method actually outperformed some of the (commonly used) frequentist methods that were investigated.

• bayesrules
Posted November 12, 2012 at 11:11 pm | Permalink

Also, in this example there is nothing particularly “computationally expensive”. I assign this to my students. A program in the computationally inefficient language R will get Berger and Mossman’s results in a few seconds on my laptop.

• Posted November 13, 2012 at 8:22 am | Permalink

Bill

It was really just meant as a joke

But yes there are cases where Bayes intervals alos have frequentist coveage
But not always
And it certainly is not the goal of Bayes

Larry

• Illy Whacker
Posted December 7, 2012 at 8:25 am | Permalink

Bayes has only one goal: to reason consistently under uncertainty. When it comes time to make a prediction, if the loss function concerns frequency properties, then Bayes will make predictions with good frequency properties.

11. Posted November 13, 2012 at 6:58 pm | Permalink

This defines coverage probability very well. But many people do not understand the fact that the 95% confidence does not apply to a single result after the data has been gathered. And even if they are aware of this ‘in theory’, they still have dangerous intuitions. Could you do a blog to explain that fully? I like to consider the following:

Many experiments will be performed on each of the fundamental particles. Estimators are derived which have 95% coverage. Before each experiment is performed, we know that the probability is at least 95% that the interval will contain the true value. (Note the future tense here). These probabilities are independent of each other; in other words, the incorrect results will be spread uniformly throughout the experiments. If you ‘condition’ on anything before the experiment (such as the day of the week on which it is performed), then this independent-and-identically-distributed 95% probability still holds.

*But*, after the experiments are done and you condition on the results, these probabilities break down entirely. We are now using the past tense. I think many people don’t understand how badly they break down – we cannot select an arbitrary experiment’s interval and say the probability is 95% that it was correct. (Also, many people don’t appreciate that this breakdown of probabilities is *not* fatal to the usefulness of confidence intervals.)

Imagine drawing a scatterplot as follows: For each experiment on the fundamental particles we draw a point. The x-axis is the lower bound of the interval and the y-axis is the upper bound. We are able to spend large sums of money to get almost-perfect estimates of the true masses and we can then go back to the original scatterplot and colour the dots red or green depending on whether the interval did contain the true value.

Imagine looking at the scatterplot of red and green points. Will the red and greens dots be uniformly mixed up with each other? *No*. There may well be patterns and clumps in the colours of the dots. Therefore, nobody is entitled to pick one dot from the scatterplot (after the initial experiments, but before the huge expenditure that establishes the true value) and say “The probability is 95% that further experiments will confirm this interval”. To make such a claim is equivalent to saying that, conditioning on the data, the incorrect results are spread uniformly throughout the observed intervals.

In such circumstances, a subject matter expert could look at each of the intervals and make comments on them – “I don’t believe the muon mass is that large and I expect the expensive follow up experiments to confirm that”. The frequentist couldn’t really agree with that comment – but the frequentist could not disagree with that comment either. The frequentist can make no claim on how the incorrect results are distributed – *in particular*, they cannot throw their hands up in the air and say they are uniformly distributed.

As a particular case, let’s take one particular, the muon, and repeat the experiment many times. For each experiment, plot the CI in the form of a scatterplot as described above. Then, we spent lots of money and get the true mass of the muon and colour the dots accordingly. Clearly there will be a pattern in the colours, the green dots will be clumped together and the red dots will surround them.

Is this fair? Is it useful? (And no, I’m not comparing it to the Bayesian approach. I’m just explaining frequentist CIs.)

• Posted November 13, 2012 at 9:38 pm | Permalink

You raise good points.
I have decided I will indeed do a lon post on
coverage, what frequentist results mean, what
Bayesian results mean etc.
Hopefully this weekend

Thanks for the comment.

Larry

• Keith O'Rourke
Posted November 14, 2012 at 10:36 am | Permalink

A lack of understanding of coverage seems to not be that unusual.

12. Posted November 14, 2012 at 12:41 am | Permalink

What if he used a matching prior?

Posted November 14, 2012 at 4:48 pm | Permalink

Interesting to see them in the class today 🙂

14. Posted December 1, 2012 at 12:33 am | Permalink

Larry,

You said that you believe that truths exist. What sort of truths?

When we observe nature (we are included in nature), there are at least two BIG jumps:

1. electrical signals are transformed in images;
2. images are transformed in sounds (such sounds are thereafter transformed in words).

In this process, we create words to represent “things´´ that are just images created by electrical signals. I am very skeptical that a specific word is THE thing itself. A word represent some features of nature that are interpreted by us, human beings. Such words should not have to be related to “real things´´, they just have to increase the chance to perpetuate our species, or at least they must not decrease it.

However, when it comes to statistical modelling…

15. Posted December 1, 2012 at 12:52 am | Permalink

Larry,

You said that you believe that truths exist. What sort of truths?

When we observe nature (we are included in nature), there are at least two BIG jumps:

1. electrical signals are transformed in images;
2. images are transformed in sounds (such sounds are thereafter transformed in words).

In this process, we create words to represent “things´´ that are just images created by electrical signals. I am very skeptical that a specific word is THE thing itself. A word represents some features of nature that are interpreted by us, human beings. Such words do not necessarily have to be related to “real things´´, they just have to increase the chance to perpetuate our species, or at least they must not decrease it.

However, when it comes to statistical modelling…

• Posted December 1, 2012 at 8:38 am | Permalink

Well, I think that hydrogen atoms exist and they have a definite mass.
I think the Hubble constant has a definite value.
I think that the random variable “it will rain tomorrow” will have a
definite known value tomorrow.