Peer review failure

I recently received a request to review an article submission from the editor of a proceedings volume. The experience eventually led me to ask the question: what is peer review?

I wasn’t an expert in the paper’s subject matter, but after expressing this reservation and receiving encouragement in return, I agreed to read it. The paper was a pleasant read, and in the end I wrote six pages of non-compulsory stylistic recommendations, a few short questions, and a couple of typographic corrections. I sent it in, and that’s when I received a puzzling reply:

I have to ask you to send it again [...] could you please state that the results are correct?

I answered that I could not possibly say with certainty that the results were correct, since for instance, the paper relies on other papers which I haven’t read. Moreover, I don’t think I could ever be sure that any complex statement is 100% correct. In any case, if I had found an error, I obviously would have mentioned this in my report. So, as a compromise, I offered to state (somewhat redundantly) that I did not notice any errors. Here was his response:

That’s good enough although I usually think that a referee report includes truth verification based on the assumption that results from other papers that are used are correct.

This view highly questionable, and not the least important reason for this is that it’s self-contradictory. Indeed, if we are merely assuming that the other papers are correct, then clearly their referees haven’t lived up to this standard. Moreover, from an objective standpoint this view is false: the existence of a single incorrect published paper already proves that! But even viewed as a qualitative or hoped-for statement, it has implications which are troublesome to me. For one thing, it implies that incorrect articles are worthless. But surely there are counterexamples to this; it’s just embarrassing for journals obsessed with “standards”. What’s more, it places the blame (and there must be blame) for incorrect articles squarely on an ambiguous cloud of anonymity. It lets mathematicians stand up and joke “well, the reviewer didn’t notice it!” as if they’ve gotten away with something.

They haven’t gotten away with anything. As long as the error is found, even if this comes after publication, then everything is in its right place. In fact, the publication-as-perfect culture is a real detriment here, because it makes it almost impossible to let readers know there is a correction.

Why not regard peer review as a chance to help mathematicians improve their exposition? Most errors come out in this process anyway. And for that matter, why not let it be open and transparent? The more voices are able to participate in the discussion, the more successful it will be.

To drive my point home, I refer as I always to to Wikipedia, which as of this date states:

Publication of incorrect results does not in itself indicate a peer review failure.[citation needed]

What an appealing line! I like it because the old guard will certainly object: “Citation needed?” To which I would counter: “Even if we find one, who will verify it’s correct?”

This entry was posted in Uncategorized and tagged . Bookmark the permalink. Post a comment or leave a trackback: Trackback URL.


  1. Posted October 7, 2011 at 8:42 pm | Permalink

    I once heard that Solovay had the following style as a referee. He’d start reading a submitted paper until reaching a point of uncertainty. If that uncertainty could not be clarified with ease, he’d return the paper to the editor, asking for clarification of that particular point. Then, this point would have been clarified by the authors, and a corrected version would be submitted. Of course, he’d continue reading until reaching the next point of uncertainty, and then returning the paper for clarifications again and again.
    Not only that this method helps validating the correctness of the reported results, it also motivates the authors to submit clearer papers in the first place (otherwise, it’ll take years for their papers to be published).

    I tend to agree with the editor who is seeking to only publish “correct” papers, but believe that the authors are the ones to be held responsible for this.

    (Disclaimer: I never verified this story with Solovay.)

    • sam
      Posted October 8, 2011 at 10:27 am | Permalink

      Haha, thanks for the story! I certainly agree with the sentiment that authors should check their papers very carefully before submitting them.

      (Personally, I always arXiv and post my papers 30 seconds before submission, and I think this means I’ll be sufficiently embarrassed if the quality is poor.)

  2. Posted October 8, 2011 at 4:15 pm | Permalink

    Sam, that’s a great post. Changing peer review one editor at a time.

    Had you considered asking for direct (anonymous or not) contact with the author? Similar to the Solovay story Assaf tells, I think immediate exchange is a great idea. Especially, in your case where you were not too familiar with the topic, there’s an obvious benefit for both author and referee: allowing the author to convince the referee of the correctness could allow the referee an opportunity to learn about a specialty they have little experience with.

    I like the idea to be challenged to convince one random person that my results are correct. As you know I don’t think it works anymore and I think we do not add to this method. After all, I spend a considerable amount of time telling others about my results. And those people can, to varying degree, vouch for the correctness of my results: maybe they’ve read my preprint, maybe they saw (a recording of) a talk of mine about the topic, maybe I approached them for discussion, references and other support — etc. Above all, they can add to assessing my artisanship as a researcher.

    To widen the extend the notion of “refereeing” could add bit of wisdom of crowds to overcome the situation you so elegantly described with that phenomenal wikipedia quote.

    • sam
      Posted October 8, 2011 at 5:45 pm | Permalink


      Yes, I always consider contacting the author directly. I asked an experienced colleague of mine whether this is frowned upon, and he replied “it’s a free country!”. The only reason that I haven’t done it yet is there has been no pressing need.

      But if I were stuck on some statement, so that I would have no choice but to “reject” the paper (whatever that means), then I would certainly rather contact the author than send a hasty recommendation to the editor.

2 Trackbacks

Post a Comment

Your email is never published nor shared. Required fields are marked *


+ 2 = five

You may use these HTML tags and attributes <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>