Proof That Theory Matters

Proof That Theory Matters

The title of this post is a slight exaggeration but I do want to discuss an interesting paper by Steve Stigler that provides empirical support for the fact that:

… there is a tendency for influence to flow from theory to applications to a much greater extent than in the reverse direction.

Steve is a professor in the department of statistics at the University of Chicago and is known, among many other things, for his scholarly work on the history of statistics. His father was the Nobel prize winning economist George Stigler.

1. The Paper

The paper is “Citation patterns in the journals of statistics and probability,” which appeared in Statistical Science in 1994 (p 94-108). I think you can get it from the following link.

The article examines citation data between various journals. Obviously the data are now out of date. And there are many practical problems with citation data which Stigler discusses at length in the article.

The part of the paper I want to focus on is where Stigler adopts an economic point if view and treats citations as a form of trade. Citations are viewed as imports and exports. Restricting to eight major journals, Stigler assigns an export score {S} to each journal. The difference of these scores measures the exporting power of one journal to another. Specifically, he defines

\displaystyle  {\rm logodds} (A\ {\rm exports\ to\ B}| {\rm A\ and\ B\ trade})= S_A - S_B.

There are eight journals but only seven free parameters (since the relevant quantities are differences). So, without loss of generality, he takes The Annals of Statistics to be the baseline journal with score {S=0}. This results in the following export scores:

Journal Score
Annals 0.00
Biometrics -1.19
Biometrika -0.35
Communications -3.27
JASA -0.81
JRSS B -0.06
JRSS C -1.30
Technometrics -0.98

To quote from the paper: “The larger the export score, the greater the propensity to export intellectual influence.” In particular, we see that The Annals is the largest exporter.

Later, he puts the journals into three groups: theory (Annals, Biometrika, JRSS B), applied (Biometrics, Technometrics, JRSS C) and mixed (JASA). The result is:

Theory 0.00
Mixed -0.66
Applied -0.99

Again we see a flow from theory to applied. The importance of this finding should not be underestimated. Quoting again from the paper:

Thus, we have striking evidence supporting a fundamental role for basic theory that runs strongly counter to the sometimes voiced claim that basic theory is not relevant to applicable methodology.

Another interesting finding in the paper is that there is very little intellectual trade between statistics journals and probability journals.

2. Conclusion

I am fascinated by this paper. The role of theory versus applied work is sometimes controversial and I believe this is one of the few quantitative studies about this issue. The data on which the study was based are now out of date. It would be great if someone would do an updated analysis. And of course we have many new sources of information such as Google Scholar.

3. Reference

Stigler, S.M. (1994). Citation patterns in the journals of statistics and probability. Statistical Science, 94-108.


  1. Zach
    Posted October 15, 2012 at 8:54 pm | Permalink

    The role of citations of theoretical papers in applied papers is often to create an air of rigorous plausibility after the method was already designed and created. I think applied methods are rarely inspired by theoretical work. Not to say that theoretical work isn’t important for other reasons.

    • Daniel
      Posted October 16, 2012 at 2:15 am | Permalink

      This is obviously false in my most humble opinion. Good examples are Support Vector Machines, Boosting and even Random Forests all of which originated by theoretical motivations. Now in the repertoire of any ‘practitioner’. Ofcourse the practitioners take them for granted, having acquired the techniques by osmosis. I guess this is the ‘other reason’ you mentioned which is infact related deeply to your first point.

      • Arik
        Posted October 16, 2012 at 7:01 pm | Permalink

        I believe these were designed based on computational attractiveness, not statistical theory. Same is true of OLS and LASSO. Theory came later.

        I find that many methods that work well in theory don’t in practice, often for computational reasons or assumptions which don’t fit reality or otherwise.

      • Posted October 16, 2012 at 7:16 pm | Permalink

        That’s true.
        However, the question of where an idea first appeared
        is different from the question of what direction is the
        net flow of ideas.


    • Posted October 16, 2012 at 8:49 am | Permalink

      Perhaps you are right.
      But it would be interesting to quantify this and do a
      rigorous study.


  2. Posted October 15, 2012 at 10:12 pm | Permalink

    I tend to prefer: theoretical ideas that are applicable and applied statistical tool/algorithm which have strong theoretical foundation. In the light of Gödel’s Incompleteness Theorem (which falls into the first category), I feel that theorems are NOT the “proof of the pudding”.

  3. Posted October 16, 2012 at 1:38 am | Permalink

    I’d guess statistical theory has more applied influence than maths theory. Understanding a newish paper on motivic cohomology requires a lot more work on the reader’s part than implementing a new test or something else that’s already prettorihmised and all familiar symbols. In a stats theory paper the reasoning be hard for a non-expert to follow but still able to copy-paste something from the paper.

  4. Alberto
    Posted October 16, 2012 at 4:29 am | Permalink

    The fight “Applications vs. Theory” is ancient. In schools where physics and mathematics are taught together this is an everyday conversation topic. Physicist say that most mathematical theories have roots in physics (they typically give the example of differential equations) and mathematicians say that advances in physics are given by advances in mathematics (e.g. Boltzmann equation, Poincare theorem).

    As long as we do not understand how our thinking works, we should start appreciating contributions in each area and forget about Freudian issues about which one is better which ends up being just a search for an identity.

    • Posted October 16, 2012 at 8:51 am | Permalink

      Agreed. On the other hand, if you are a funding agency allocating finite resources
      you don’t have the luxury of appreciating all contributions equally.

  5. Ricardo Silva
    Posted October 17, 2012 at 10:20 am | Permalink

    Thanks for the link, Larry. Very interesting reading (Stigler has always a lot of interesting things to say, and we had the pleasure of having him as the main guest speaker when our department celebrated its 100th anniversary). A closely related question is: when does theory itself help to develop radically new theories? I’m not talking about steady improvements, which of course it is something that happens all the time, but an theoretical idea which taken into a different context predicts new consequences which were not anticipated by practical work?

%d bloggers like this: