Proof That Theory Matters

The title of this post is a slight exaggeration but I do want to discuss an interesting paper by Steve Stigler that provides empirical support for the fact that:

*… there is a tendency for influence to flow from theory to applications to a much greater extent than in the reverse direction.*

Steve is a professor in the department of statistics at the University of Chicago and is known, among many other things, for his scholarly work on the history of statistics. His father was the Nobel prize winning economist George Stigler.

**1. The Paper **

The paper is “Citation patterns in the journals of statistics and probability,” which appeared in *Statistical Science* in 1994 (p 94-108). I think you can get it from the following link.

The article examines citation data between various journals. Obviously the data are now out of date. And there are many practical problems with citation data which Stigler discusses at length in the article.

The part of the paper I want to focus on is where Stigler adopts an economic point if view and treats citations as a form of trade. Citations are viewed as imports and exports. Restricting to eight major journals, Stigler assigns an export score to each journal. The difference of these scores measures the exporting power of one journal to another. Specifically, he defines

There are eight journals but only seven free parameters (since the relevant quantities are differences). So, without loss of generality, he takes *The Annals of Statistics* to be the baseline journal with score . This results in the following export scores:

Journal | Score |

Annals | 0.00 |

Biometrics | -1.19 |

Biometrika | -0.35 |

Communications | -3.27 |

JASA | -0.81 |

JRSS B | -0.06 |

JRSS C | -1.30 |

Technometrics | -0.98 |

To quote from the paper: “The larger the export score, the greater the propensity to export intellectual influence.” In particular, we see that *The Annals* is the largest exporter.

Later, he puts the journals into three groups: theory (Annals, Biometrika, JRSS B), applied (Biometrics, Technometrics, JRSS C) and mixed (JASA). The result is:

Theory | 0.00 |

Mixed | -0.66 |

Applied | -0.99 |

Again we see a flow from theory to applied. The importance of this finding should not be underestimated. Quoting again from the paper:

*Thus, we have striking evidence supporting a fundamental role for basic theory that runs strongly counter to the sometimes voiced claim that basic theory is not relevant to applicable methodology.*

Another interesting finding in the paper is that there is very little intellectual trade between statistics journals and probability journals.

**2. Conclusion **

I am fascinated by this paper. The role of theory versus applied work is sometimes controversial and I believe this is one of the few quantitative studies about this issue. The data on which the study was based are now out of date. It would be great if someone would do an updated analysis. And of course we have many new sources of information such as Google Scholar.

**3. Reference **

Stigler, S.M. (1994). Citation patterns in the journals of statistics and probability. *Statistical Science*, 94-108.

## 12 Comments

The role of citations of theoretical papers in applied papers is often to create an air of rigorous plausibility after the method was already designed and created. I think applied methods are rarely inspired by theoretical work. Not to say that theoretical work isn’t important for other reasons.

This is obviously false in my most humble opinion. Good examples are Support Vector Machines, Boosting and even Random Forests all of which originated by theoretical motivations. Now in the repertoire of any ‘practitioner’. Ofcourse the practitioners take them for granted, having acquired the techniques by osmosis. I guess this is the ‘other reason’ you mentioned which is infact related deeply to your first point.

I believe these were designed based on computational attractiveness, not statistical theory. Same is true of OLS and LASSO. Theory came later.

I find that many methods that work well in theory don’t in practice, often for computational reasons or assumptions which don’t fit reality or otherwise.

That’s true.

However, the question of where an idea first appeared

is different from the question of what direction is the

net flow of ideas.

LW

Perhaps you are right.

But it would be interesting to quantify this and do a

rigorous study.

LW

I tend to prefer: theoretical ideas that are applicable and applied statistical tool/algorithm which have strong theoretical foundation. In the light of Gödel’s Incompleteness Theorem (which falls into the first category), I feel that theorems are NOT the “proof of the pudding”.

Apologize profusely, don’t want to come across as a pedant. But for the Godel reference: http://www.amazon.com/Godels-Theorem-Incomplete-Guide-Abuse/dp/1568812388/ref=sr_1_1?ie=UTF8&qid=1350409568&sr=8-1&keywords=Godel%27s+theorem+use+and+abuse

I admit that I am not a expert of mathematical logic but interestingly you just proved why we have higher citation for theoretical papers :)..and I agree that is what our natural tendency is..

I’d guess statistical theory has more applied influence than maths theory. Understanding a newish paper on motivic cohomology requires a lot more work on the reader’s part than implementing a new test or something else that’s already prettorihmised and all familiar symbols. In a stats theory paper the

reasoningbe hard for a non-expert to follow but still able to copy-paste something from the paper.The fight “Applications vs. Theory” is ancient. In schools where physics and mathematics are taught together this is an everyday conversation topic. Physicist say that most mathematical theories have roots in physics (they typically give the example of differential equations) and mathematicians say that advances in physics are given by advances in mathematics (e.g. Boltzmann equation, Poincare theorem).

As long as we do not understand how our thinking works, we should start appreciating contributions in each area and forget about Freudian issues about which one is better which ends up being just a search for an identity.

Agreed. On the other hand, if you are a funding agency allocating finite resources

you don’t have the luxury of appreciating all contributions equally.

LW

Thanks for the link, Larry. Very interesting reading (Stigler has always a lot of interesting things to say, and we had the pleasure of having him as the main guest speaker when our department celebrated its 100th anniversary). A closely related question is: when does theory itself help to develop radically new theories? I’m not talking about steady improvements, which of course it is something that happens all the time, but an theoretical idea which taken into a different context predicts new consequences which were not anticipated by practical work?