Is Bayesian Inference a Religion?

Time for a provocative post.

There is a nice YouTube video with Tony O’Hagan interviewing Dennis Lindley. Of course, Dennis is a legend and his impact on the field of statistics is huge.

At one point, Tony points out that some people liken Bayesian inference to a religion. Dennis claims this is false. Bayesian inference, he correctly points out, starts with some basic axioms and then the rest follows by deduction. This is logic, not religion.

I agree that the mathematics of Bayesian inference is based on sound logic. But, with all due respect, I think Dennis misunderstood the question. When people say that “Bayesian inference is like a religion,” they are not referring to the logic of Bayesian inference. They are referring to how adherents of Bayesian inference behave.

(As an aside, detractors of Bayesian inference do not deny the correctness of the logic. They just don’t think the axioms are relevant for data analysis. For example, no one doubts the axioms of Peano arithmetic. But that doesn’t imply that arithmetic is the foundation of statistical inference. But I digress.)

The vast majority of Bayesians are pragmatic, reasonable people. But there is a sub-group of die-hard Bayesians who do treat Bayesian inference like a religion. By this I mean:

  1. They are very cliquish.
  2. They have a strong emotional attachment to Bayesian inference.
  3. They are overly sensitive to criticism.
  4. They are unwilling to entertain the idea that Bayesian inference might have flaws.
  5. When someone criticizes Bayes, they think that critic just “doesn’t get it.”
  6. They mock people with differing opinions.

To repeat: I am not referring to most Bayesians. I am referring to a small subgroup. And, yes, this subgroup does treat it like a religion. I speak from experience because I went to all the Bayesian conferences for many years, and I watched witty people at the end-of-conference Cabaret, perform plays and songs that merrily mocked frequentists. It was all in fun and I enjoyed it. But you won’t see that at a non-Bayesian conference.

No evidence you can provide would ever make the die-hards doubt their ideas. To them, Sir David Cox, Brad Efron and other giants in our field who have doubts about Bayesian inference, are not taken seriously because they “just don’t get it.”

It is my belief that Dennis agrees with me on this. (If you are reading this Dennis, and you disagree, please let me know.) Here is my evidence. Many years ago, there was a debate about whether to start a Bayesian journal. Dennis argued strongly against it precisely because he feared it would make Bayesian inference appear like a sect. Instead, he argued, Bayesians should just think of themselves as statisticians and send their papers to JASA, the Annals, etc. I think Dennis was 100 percent correct. Dennis lost this fight, and we ended up with the journal Bayesian Analysis.

So is Bayesian inference a religion? For most Bayesians: no. But for the thin-skinned, inflexible die-hards who have attached themselves so strongly to their approach to inference that they make fun of, or get mad at, critics: yes, it is a religion.

About these ads

37 Comments

  1. Alexander Kruel
    Posted September 1, 2013 at 9:09 am | Permalink

    People perform plays and songs that mock frequentists? Well, here is one that mocks a Bayesian :-)

    The only other mockery of Bayesianism that I know of is the following creed:

    There is no Science but Bayes and it is our Method.

    I believe in Probability Theory, the Foundation, the wellspring of knowledge,
    I believe in Bayes, Its only Interpretation, our Method.
    It was discovered by the power of Induction and given form by the Elder Jaynes.
    It suffered from the lack of priors, was complicated, obscure, and forgotten.
    It descended into AI winter. In the third millennium it rose again.
    It ascended into relevance and is seated at the core of our FAI.
    It will be implemented to judge the true and the false.
    I believe in the Sequences,
    Many Worlds, too slow science,
    the solution of metaethics,
    the cryopreservation of the brain,
    and sanity everlasting.
    Phyg.

  2. george
    Posted September 1, 2013 at 3:22 pm | Permalink

    1. There’s another recent interview with Lindley here

    2. Another small subgroup of statisticians (still) get worked up preaching against (as they see it) the sins of Bayesianism. This view used to be a lot more common – when ISBA was founded, being labelled as a Bayesian was a risky career move. I hope you’ll also post something provocative about other types of fanatics.

    3. So long as no-one gets blown up, isn’t it ultimately useful to have *small* groups of witty, zealous people pushing unconventional ideas? Member of the pragmatic majority may not agree with extreme ideas, but they may value a good description of them, as an aid to understanding the compromises their pragmatic position imposes.

    • Posted September 1, 2013 at 3:37 pm | Permalink

      1. Thanks for the link.
      2. I don’t think anyone’s career was ever adversely affected by being a Bayesian. I don’t know of a single example.
      3. Maybe. I am not so sure.

      • george
        Posted September 1, 2013 at 5:35 pm | Permalink

        2. This doesn’t count?

      • Posted September 3, 2013 at 1:14 am | Permalink

        I do believe that being a Bayesian was hazardous to one’s career some years ago. Certainly not today, though, and I have a hunch that today it is hazardous to be a frequentist. Deborah’s Frequentists in Exile blog is there for good reason!

      • rj444
        Posted September 3, 2013 at 12:19 pm | Permalink

        “2. I don’t think anyone’s career was ever adversely affected by being a Bayesian. I don’t know of a single example.”

        There are massive swaths of research that won’t allow you to publish unless you have p-values in your papers, even in study designs where testing is largely inappropriate. I’m not trying to be a p-value policeman here – I think that there are settings where testing is appropriate and settings where it’s not (I agree with some of what Sander Greenland and Andrew Gelman have written on this).

        I’d say even if most statisticians have moderated their views on the legitimacy of Bayesian approaches, these institutional barriers elsewhere in academia and science constitute quite substantial lost career opportunities.

      • george
        Posted September 4, 2013 at 2:03 am | Permalink

        2. (again) Here’s a brief comment from Andrew Gelman on the above, and also a post by him on “religion” as a term of insult.

      • Posted September 8, 2013 at 4:56 pm | Permalink

        Larry:

        Here is an example of some of the ignorant and obnoxious attitudes I had to deal with back in the 1990s. My career turned out fine in retrospect, but these guys did their very very best to derail it. These were people who achieved pure anti-Bayesianness by the simple expedient of looking away whenever any applied Bayesian analysis was done in their vicinity. The only good thing I can say about this was that being treated with complete disrespect gave me a true appreciation for pluralism. It was really horrible to be in that environment.

      • Posted September 8, 2013 at 5:20 pm | Permalink

        Andrew

        Yes, Leo’s skepticism was certainly strong.
        But it wasn’t limited to Bayesian inference.
        It also applied to parametric models and, for that matter,
        anything that was not about prediction.
        He also basically rejected probability!

      • Posted September 9, 2013 at 6:01 am | Permalink

        Larry:

        Breiman’s skepticism was not merely “strong”; it was also misinformed (i.e., ignorant), as I explain in my linked post. Beyond this, he and his friends really did try to derail my career, and the fact that I used Bayesian models to do applied statistics was a big part of it, I think. So now you do know someone whose career was adversely affected by being a Bayesian!

        It’s funny in retrospect because both Breiman’s work and mine have become influential in computer science. But it wasn’t enough for him to be proud of the good work he’d done; he had to go further and put down the work of others, work that he had not bothered to try to understand.

      • Posted September 10, 2013 at 7:55 am | Permalink

        Andrew – “wasn’t enough for him to be proud of the good work he’d done; he had to go further and put down the work of others”

        From http://en.wikipedia.org/wiki/Tall_poppy_syndrome “Status is a relative value, so for someone to rise in status, another person must fall.”

        I have often noticed people who have done good work put down work of other talented (but yet less successful/established) people. I believe it happens more often when the yet less successful people have knowledge/skills that the already successful cannot easily acquire.

  3. Posted September 1, 2013 at 5:43 pm | Permalink

    During my life time, I have heard of many comments about the Statisticians who happen to believe in the Bayesian approach in resolving a problem . Some were quite unfriendly. But Bayesians forming ” Sect ” is kind of a novel remark. For over two centuries Bayes Rule was not even recognized as a part of serious statistical studies By the Classical Statisticians.
    The only thing remotely is related to a “religion” is perhaps its creator, Rev. Thomas Bayes, an amateur Statistician who loved Probability,

    So, pardon my audacity of writing this Limerick on this issue. Just cannot help it.

    ” There lived once a Reverend Thomas Bayes,
    Who laid down the logic, of simple Rules of Bayes
    Combining a Prior probabilty
    With a Conditional probabilty,
    And initiated his ” Sect”, for these future days.”

    To be a Bayesian Statistician, one has to be a Statistician first. One can take a Bayesian out of a Statistician , but cannot take a Statististician out of Bayesian. There can be several approaches to a study. Bayesian is an approach
    appears to be logical to some. It is a personal choice based on logic.

    May be those are fewer in numbers than the other kind?

    Sumedha
    Livermore, CA.

    • Posted September 5, 2013 at 3:07 pm | Permalink

      If we’re going for poems, this sprang to mind a few months ago and I wrote it down. It is missing, of course, a few more verses in the style of Charles Dodgson (satirizing a more boring poem).

      “You are bold, Reverend Thomas,” the young Student said,
      “And your judgment’s exceedingly iffy,
      To say that the chance a coin lands on its head
      Should be anything but fifty-fifty.”

      “Were I meek,” said the Reverend, “a Prior so crude
      I knew nothing before I’d begun,
      For all of your science, all I could conclude
      Is that two heads are better than one!”

      “You are bold,” said the youth, averting his eyes,
      “The reason is probably apparent,
      To be out and about, are you sure its quite wise
      when all of your wardrobe’s transparent?”

      “In my youth,” said the Rev, striking a pose,
      “I’d often forget about coverage,
      Instead of being clothed some times and not
      It’s best to be dressed on average!”

  4. Posted September 1, 2013 at 9:14 pm | Permalink

    I have a question: in what sense is ‘religion’ being used here in any way but to mock certain people the author dislikes?

    Specifically, let’s take the following list:

    They are very cliquish.
    They have a strong emotional attachment to Bayesian inference.
    They are overly sensitive to criticism.
    They are unwilling to entertain the idea that Bayesian inference might have flaws.
    When someone criticizes Bayes, they think that critic just “doesn’t get it.”
    They mock people with differing opinions.

    Now, if the people in question are treating Bayesian inference as a *religion*, and these are the defining attributes which render them guilty, surely an *actual* honest-to-god impossible-to-dispute religion like Roman Catholicism will also satisfy this list? But does it? When someone criticizes the Catholic Church over institutionalized protection of pedophilia, is the response to think ‘that critic just doesn’t get it’? Do churches put on skits mocking Jews for being too blind to follow Jesus? Do they strike one as ‘overly sensitive to criticism’ and flip out if anyone suggests that transubstantiation of the Eucharist’s substance is really pretty dumb? Do they all have a strong emotional attachment to every tenet of Catholic dogma (in the technical sense of the word)? Are they entirely unable to consider the merits of say Calvinism? Having been raised a Catholic, gone to a Catholic highschool and discussed many things with the brothers who ran it, etc etc, I submit that at least in my own experience, the answer to all of these questions are no, and the more expert in Catholicism someone was, the less any of this was true.

    So, if religions do not match the checklist put forth, one has to wonder why the term was chosen or applied at all. If nothing ele, it’s rather misleading.

  5. Posted September 2, 2013 at 7:23 pm | Permalink

    I kind of like the fact that there is now a Bayesian Analysis Journal. It was about time that we had one.
    Thanks to the Project Euclid.

    Sumedha

  6. Posted September 2, 2013 at 8:15 pm | Permalink

    I was wondering what a Bayesian is. For example, how do I know whether I am a Bayesian or not? Is a Bayesian anyone who uses a prior?

    I feel people even do not agree on the definition of being a Bayesian. This makes me very curious. Could you write another post on this?

  7. Posted September 2, 2013 at 9:50 pm | Permalink

    Love your post, in fact I will try using that reblog function to lasso it.
    Frequentist-in-Exile

  8. Posted September 2, 2013 at 9:52 pm | Permalink

    Reblogged this on Error Statistics Philosophy and commented:
    Reblogging a stimulating post from the Normal Deviate!

  9. Posted September 3, 2013 at 12:15 pm | Permalink

    Well it’s true that “frequentists just don’t get it”. There’s a real asymmetry concerning who understands what, due to everyone getting the same formative frequentist introduction to statistics, but Bayes coming much later if at all.

    The cliquishness part was new to me though since I’ve only met three Bayesians in person and they were very mild. That Lindley interview was a real eye opener. As was this bizarre fellow: http://www.thomashoskynsleonard.co.uk/bayesian_boy5.htm#axiomatic

    Choice quote:

    “Dennis also confirms how he encouraged Florence David to leave UCL for California (he’d previously been a bit more explicit to me about this) and, quite remarkably, says that he tried to arrange the early retirement of two of his colleagues at UCL for not being sufficiently Bayesian!! This was around the time that he was influencing a downturn in my career at the University of Warwick. Dennis’s account of his own early retirement does not match what actually happened. According to Adrian Smith, Dennis was encouraged to retire after a fight with the administrators over the skylight in Peter Freeman’s office.”

    This is believable given Lindley’s own words. It’s repulsive as hell. What if one of those cliques denies tenur to a well deserving candidate whose wife was already tenured at that school and they have kids? These people are screwing with other people’s lives over some stupid blah blah. What’s worse is that instead of having a stand-up debate, they’re cowardly trying to win the debate in hiring/tenure committees. Pathetic. I don’t care if Frequentists were worse back in the day, it’s still pathetic. This stuff actually lowered my opinion of academia even further than it already was, which I hadn’t thought possible. Academia is little more than welfare for small men who like to think they’re something special.

    • Posted September 3, 2013 at 12:21 pm | Permalink

      >> “Academia is little more than welfare for small men who like to think they’re something special.”
      Like any group of people, academia contains some bad actors but also lots of good people.
      I wouldn’t generalize too much from a few anecdotes.

      • Posted September 3, 2013 at 12:51 pm | Permalink

        I spent more than half my adult life in academia and at some point the plural of “anecdote” IS “data”. Plus I wasn’t talking about good or bad.

    • Posted September 5, 2013 at 9:29 pm | Permalink

      Why do you misplace the source of the bad behavior? It’s not academia in general. We’re talking about an individual–one who smirked about this, and I can tell you a dozen such stories about Bayesians who showed less than professional behavior when in power over frequentists (hiring, tenure, awards, named chairs, publications, etc.). I’m actually a bit surprised you’re surprised.

      • Posted September 6, 2013 at 9:14 am | Permalink

        Becuase there were a bazillion examples of Frequentists doing worse to each and Bayesians back in the day. If they’re both doing it, then its acamedemia and not some voo-doo spell cast by Bayes a few hundred years ago.

      • Posted September 6, 2013 at 9:56 am | Permalink

        You say “back in the day”, but that’s not the period we’re speaking of. I never said there was a voo-doo spell cast by Bayes a few hundred years ago, but there is some contemporary Bayesian ‘voodoo statistics’ (a term that actually arose in a book I wrote).

      • Posted September 6, 2013 at 9:14 am | Permalink

        I meant “doing worse to each other and Bayesians”

      • Posted September 6, 2013 at 9:30 am | Permalink

        Incidently, I’ve never hired a Bayesian and only hired classically trained frequentists. They’re free to use whatever method in their work they feel most comfortable with. Although in practice they always gum it up somehow and I have to fix it before it kills anyone. Otherwise you’ll get claims like “the size of bombs has no impact on lethality” because they did a regression anlysis that didn’t include the distance between the bomb and the target at the time of the explosion and found that the coefficient of the “size” variable wasn’t statistically significant at the .05 level. I wish I were joking.

      • Posted September 6, 2013 at 10:27 am | Permalink

        Mayo, I refer to “back in the day” because those are the horror stories I happen to be familiar with. There may be plenty of examples today, I don’t know. Gelman may have some more current recollections from his Berkley experience. I was told once to stick exactly to the Frequentist text book when teaching an undergrad statistics class and not to change any parts of it I thought were seriously wrong, but that’s small beans really. I’m so grateful to be out of the swamplands of academia it was no big deal and frankly I didn’t think it made a bit of difference whether those undergrad social scientists understood statistics or not.

        Oh yeah there was a frequentist Economist who insisted on doing some frequentist blah blah even though I had mountain of research papers showing they could skip that step entirely and get on with the project. Although in fairness they may have just been wanting to plus up the number of billable hours they could charge. That’s about it for me personally.

  10. Logan
    Posted September 6, 2013 at 6:30 am | Permalink

    I also know plenty of Fundamentalist Frequentists who are so thin-skinned about criticisms on the likelihood principle, maximum likelihood estimators, profile likelihoods …

    Example:

    John Doe: The Maximum Likelihood Estimator of the parameters of model M does not exist.
    Fundamentalist Frequentist: The model is wrong!

    John Doe: The MLE of the parameters of the distribution F is infinite with positive probability.
    Fundamentalist Frequentist: The distribution is wrong!

    • Posted September 6, 2013 at 7:55 am | Permalink

      I have never heard anyone respond like your John Doe.
      If the MLE does not exist (I assume you mean the likelihood
      is unbounded) then the frequentist response is: then the MLE
      has bad frequentist behavior and should not be used.
      There is nothing special about the MLE in frequentist inference.

  11. Hongyi Zhang
    Posted September 8, 2013 at 10:30 pm | Permalink

    Could I digress a little and ask for your opinion on a (what I believe to be) related question? I find (and many argue) that Bayesian approach — more precisely, using posterior distribution — is especially desirable when making decisions under uncertainty. As outcome of statistical tests would often serve as the basis of decision making (including the rejection of hypotheses), isn’t it more appropriate to output a distribution rather than a summary of the distribution (such as confidence interval)?

  12. Posted September 10, 2013 at 9:33 am | Permalink

    When the FDA allows sponsors to create priors and conduct analysis on that basis, then I suppose we can say that Bayes won.

    • Posted September 10, 2013 at 1:30 pm | Permalink

      But that’s history – read the online guidance from FDA.

      If I recall correctly, the prior is subject to review and the acceptance is likely on a case by case basis.

  13. Posted July 29, 2014 at 9:54 am | Permalink

    thank you for your comments, Entsophy. I am really not that bizarre. For further insights into Bayesian academia, please read A Personal History of Bayesian Statistics by Thomas Hoskyns Leonard (Wiley Interdisciplinary Reviews, 2014 with a link to further chapters on my website. For a recital of my poem ‘Wherefore Dennis?’ please google the Youtube collection ‘The Leith Walk Rhymers’. For further fun and games,, google ‘reverb Dennis Lindley and my interview The Life of a Bayesian Boy’ in Statistics Views. My Bayesian History refers to my correspondence with Deborah Mayo regarding her, and Michael Evans, refutation of Birnbaum’s proof of the Likelihood Principle, Now that was bizarre! Love and best wishes Thomas Hoskyns Leonard

8 Trackbacks

  1. […] Is Bayesian Inference a Religion? (normaldeviate.wordpress.com) […]

  2. […] a minority of statisticians who use Bayesian methods (and he does emphasize it’s a minority), Bayesian inference is like a religion. I think Larry’s choice of words is unfortunate (he says “religion” when really […]

  3. […] I just commented on Larry’s “provocative” post […]

  4. By Bayesian religion | LARS P. SYLL on September 9, 2013 at 4:16 pm

    […] Normal Deviate […]

  5. By 学点数据分析 | Antique on September 10, 2013 at 11:30 pm

    […] 今天最喜欢的网页:贝叶生统计是宗教信仰吗?老调看着还是挺有趣。 […]

  6. By Entsophy on September 12, 2013 at 12:43 pm

    […] This is far from the only embarrassment for SEV, but I won’t run up the scoreboard by mentioning others. At this rate “Error Statistics” will turn everyone into Bayesians and where’s the fun in that? What’s weird though that no one thought to do an elementary check on it. It’s almost as though “SEV” was accepted on religious grounds. […]

  7. By Somewhere else, part 75 | Freakonometrics on September 14, 2013 at 5:55 pm

    […] Bayesian Inference a Religion?” http://normaldeviate.wordpress.com/2013/09/01/… via @cynorrhodon for me, Bayesian statisticans are wizards […]

  8. By Bayesian inference gone awry | LARS P. SYLL on July 24, 2014 at 4:57 am

    […] Larry Wasserman […]

Follow

Get every new post delivered to your Inbox.

Join 945 other followers

%d bloggers like this: