March 26, 2002
Follow me on Twitter

Anonymous reviewers

Elin Sjursen asks (March 25), "Why do scientific conferences like Hypertext '02 have anonymous reviewers?"

The most important duty of a conference reviewer is to preserve the integrity of the literature -- to ensure that reported data are sound and that their interpretation is as thorough and as correct as possible. If the data are wrong, they must not be published. People rely on the scientific literature. You have to assume that someone might design a bridge or a spacecraft that relies on this data; if the data are wrong, people will die.

Worse, if we let any consideration override this imperative, if we knowingly allow questionable results or false conclusions to appear in the literature, then nobody can trust the literature. That betrays science and welcomes the return of the dark ages.

But reviewers are people, and there's often a significant power imbalance between the author of a paper and the reviewer. The reviewers with the keenest eyes and freshest perceptions are often graduate students or post-docs; dare they offend the senior people who control jobs, salaries, and honors? To whom they may owe personal and professional debts?

Some conferences hide author identities from reviewers to prevent reviewers from being swayed by fame or reputation. In theory, this makes sense. In hypertext research, I think this is problematical. Because it's a small field, readers would often know the author's identity even though the paper was notionally anonymous. You'd exchange one kind of bias for another. And the new bias would be noisy -- occasionally, reviewers would be absolutely certain they knew the author's identity, but would get it wrong anyway. It's easier to watch for bias if the cards are on the table. (On rare occasions, reviewers do waive anonymity. Usually, this happens when a reviewer identifies a mistake and can offer unique assistance in remedying it.)

The combination of signed papers and anonymous reviewers protects new members of the research community. As reviewers, they can speak firmly and clearly, without fear of political repercussions. As authors, they gain reputation from their submissions -- even their unsuccessful submissions. It may in fact be better to write a controversial paper that's ultimately rejected than to write a tame, middle-of-the-pack paper that is accepted. The controversial paper is going to be read, reread, scrutinized, and argued over by the whole program committee; that's about as good an audience as you can hope for in hypertext research.

I'm nursing some wounds myself here -- one of my HT02 papers went down in flames this year. On the whole, I think I'm happier not knowing the identities of the reviewers.