Anecdotes are Metascience
In Irving Langmuir's 1953 talk Pathological Science, published in 1989, he tells the story of visiting another professor's lab hoping to see claims about electrons defying Bohr's model of the atom firsthand. The experiment required counting electron strikes in a dark room over a period of a few minutes, and depending on the voltage applied to an apparatus, the counts followed an unexpected pattern. Langmuir observed the counting process, which appeared to be subjective, then tricked the professor into thinking the voltage was high when it was actually low. The professor nonetheless "saw" the phenomenon. Langmuir revealed the deception, and yet this, and a long confrontation, didn't convince the professor he was deceiving himself.
The paper, which is worth reading for the other anecdotes, and the elaborate details of self-deception, is an early example of Red Teaming, metascience, and of scientific anecdote.
Did any of the events happen? We don't know for sure other than Langmuir's word and the fact that the phenomenon was never reproduced. Does the anecdote mean anything about science? Is it metascientific? The answer to both, I believe, is yes. Despite the fact that this is a dangerous practice of putting one person's word against another's, it is scientific and necessary.
Metascience is the study of the practice of science and it collects data about scientific outputs like papers. If the way that a paper was produced invalidates using it as evidence, then evidence about the production of the paper is metascience. Duplicate image sleuthing and fraud detection are metascience. So is whistleblowing. So is anecdote.
Anecdote, at least, forces the reader to think through how fragile certain evidence is. If it is as fragile as one voltage in one lab, then that is a problem apart from only having Langmuir's word. At least science can be tested. Even if we'll never know if Langmuir made it up, we know the phenomenon isn't real.
And much of science depends on one person's word already. Many multi-authored papers have this problem. Francesca Gino's alleged fraud was not known to her coauthors, nor was Jonathan Pruitt's, nor Jan Hendrik Schön's. The results and the data were only known to one person and it was their word that kept the finding alive.
Furthermore, there are fewer and fewer people involved as you go from primary research, to metascience, to evidence about metascience. Sometimes things that deal with the fate of all science are hashed out on obscure blogs.
The more that science is based on data that wasn't shared and hypotheses that weren't preregistered, and is performed by a class of people who often behave like one big team, the more anecdote will count as science. The more that science is "gameable," and outcomes depend on single points of failure, the more that anecdote will count.
Two of my future posts are long email discussions I've had with various scientific professionals at The Lancet, and Community for Rigor. They are anecdotal. As in the introduction, I am judging how important these posts are by how important they would have been to me years ago had I never worked in science, or spent time reading the metascience reading list. They are "Red Teaming" of processes with single points of failure. In the case of The Lancet, the process is COPE guidelines around reported errors. In the case of Community for Rigor, it is their call for feedback on annual conference recordings. If we don't test these processes, we can only guess that they work, and science has guessed wrong before.
I won't have any editorializing in the posts themselves, so some preamble:
1. Anecdotes aren't proof of anything.
2. The subjects are not "bad apples," if they've done anything wrong — or even unscientific — at all, which I can't prove. They are one of many, and choices in a scientific career, like the choice to publish or not, are often about keeping one's job and the fruits of many years of work. I think sympathy is appropriate and I hope they can appreciate my position too.
My next post, “Science reform: Do we need a plan B?” comes out Monday, May 19th, 2025.
“A correspondence with The Lancet about an apparent subtraction error” comes out Monday, May 26th