Perfunctory Examples and the Role of Applications: A Rant

Since I am working on revising of a paper (always an annoying task) this seems like a good time for rant. In particular, I want to complain about the perfunctory example.

1. Perfunctory Examples

Imagine this: you submit a paper to a journal (or a conference). The reviews are positive. But at least one reviewer feels compelled to add: “please add a real example” or “please add more simulations.”

Now, if adding an example or another simulation would truly make the paper better, then fine. But it’s my experience that the “add a real example” reaction is more of a reflex action. The reviewer has not asked himself or herself: does this paper really need an example (or another simulation)?

The result of these requests for examples is this: our journals are full of papers with perfunctory examples and simulations that add nothing to the paper. Worse, researchers waste weeks of precious time satisfying the arbitrary whims of a reviewer.

Let me repeat: I do think examples and simulations can sometimes make a paper better. But often not. It is just a waste of time. There just seems to be a basic, unquestioned assumption that a paper with a “real example” is publishable while a paper without one is not.

2. The Role of Applications

Now I want to clarify that I do think applications play a critical role in statistics and machine learning. In my opinion, theory is interesting when it is motivated, even indirectly, by real applications. (Just as theoretical physics is ultimately based on trying to understand the world we live in.)

Kiri Wagstaff has a nice paper called Machine Learning That Matters at ICML. Her abstract begins Much of current machine learning (ML) research has lost its connection to problems of import to the larger world of science and society. This is very interesting to me because many years ago (perhaps in the 1980’s?) similar discussions took place in the field of statistics. I think there was a general consensus that statistics was putting too much emphasis on theory for its own sake and needed to get back to its roots.

The problem is this: while I do think that good theory is ultimately rooted in applications, the link between applications and theory can be subtle. Solving applied problems leads to the development of new methods. Then people start to ask questions about the behavior of these methods. This leads to new theory. But the chain linking theory to applications can be a long, complex, very non-linear route. Simply adding an example into a paper is not going to illuminate the connection.

3. Conclusion

To me, theory is interesting if it explains, creates or casts some light on methodology. Judging whether theory is interesting is subtle and, yes, also a matter of taste. But there is a temptation to reduce it to: is there an example in the paper?

How can we make sure that statistics and machine learning stay anchored in the real world without simply requiring that people add some bogus example to their papers? Kiri Wagstaff offers some ideas, including six impact challenges. I’d like to know what other people think about it.

One possible answer is: it doesn’t matter. People should work on, and publish, whatever they want. On average, good theory will get noticed and used; less useful theory will attract less attention. In other words, let the field find its own direction. We shouldn’t try to direct it. (This happens to be my opinion but I suspect many will disagree.)

In the meantime, let’s have a moratorium on reviewers asking for more examples and simulations, just for the sake of it.

And to borrow from Dennis Miller, the master of rants, I’ll conclude with:

That’s just my opinion, I could be wrong.

—Larry Wasserman


  1. Corey
    Posted July 25, 2012 at 10:50 am | Permalink

    I think this blog post would be improved if you gave an example of a paper (perhaps a paper of your own, or a fictional example would do too) that was improved by the reviewer’s request for an example, and an example of the converse, with a brief explanation of the contrast.

    I’m not just being a jackass, I really do think this (although I do enjoy the irony of making this comment). I also recognize that there’s a time trade-off that could militate against supplying examples for a simple blogpost.

    • Posted July 25, 2012 at 11:51 am | Permalink

      An interesting idea but I couldn’t do it without using real papers
      and then could end up offending someone.

  2. Posted July 25, 2012 at 10:48 pm | Permalink

    I agree strongly with this rant. The role of an example is to clarify the method — that is, it should be illustrative. The role of an experiment is to test an hypothesis. But we have lots of published papers that include “experimental results” where there was no hypothesis at risk of being testing. And at the same time, the “experiments” do nothing to clarify the method being presented because they typically just show us performance numbers rather than providing insight into the method or its limitations.

  3. Ron Coleman
    Posted July 26, 2012 at 6:18 am | Permalink

    Moreover, these requests motivate that many of the “applications” shown in small journals are nothing else than an illustration. This is, there is no real/practical/empirical interest in applying their models/methods on that particular data set. Small papers could survive without these kind of “applications” and would save lots of pages.

  4. Christian Hennig
    Posted July 26, 2012 at 11:38 am | Permalink

    I have seen lots of superfluous examples in papers. I have also asked for examples as a reviewer myself. I think this was always in cases where some implicit or explicit model assumptions were made that I expected to be violated in a critical way in pretty much every real situation and where I wanted to know whether the authors could come up with a situation where anyone could accept the assumption as reasonable. The most superfluous examples are probably those where it would be interesting to ask such questions, but where the examples are not used to address them (and wouldn’t work if one tried).

  5. Clark
    Posted July 27, 2012 at 4:49 pm | Permalink

    Sometimes you just have to tell a reviewer (politely), “No”. There are many tactful & reasonable ways to do this. I have done this on several occasions, and never seen it impact the acceptance of the paper by the journal; there are other reviewers opinions involved, plus an editor. On the other hand, if you get multiple reviewers suggesting the same thing, then it is probably a good idea to do it.

  6. Posted July 31, 2012 at 1:57 pm | Permalink

    I’m sympathetic to your point. However, as a consumer of a lot of papers, I’d point out one advantage of even fairly trivial examples: verification. If the paper concerns a method that I’d like to try implementing, it’s helpful to have a simple example so that I can be sure that I’ve actually copied all the math correctly and can replicate the process. Actual code in supplemental material is of course much better than a narrative for this purpose though.

    • Posted August 1, 2012 at 12:22 pm | Permalink

      I think that’s an important point for many papers. Even actual code usually needs a data set or worked example to follow.

      An additional concern is the audience – I edit a journal that publishes mathematical and statistical work for a particular field and we usually require an example from a related discipline (but I agree that it isn’t always necessary).

%d bloggers like this: