Cognitive dissonance and a truly impossible assumption
The many unopened boxes of psychology
One reaction to news that another keystone psychology finding has been severely limited by researcher dishonesty would be to wonder if there’s anything left. Is there anything to psychology at all? Another would be the opposite, that we need to stop throwing out whole fields, that “the replication crisis brain” is a totalizing epistemic black hole.
Another would be to doubt the veracity of the news. It makes sense to doubt that a single independent researcher without apparent credentials could overturn 70 years of a scientific discipline by opening an unsealed archive called “box 4.”
Surprisingly, psychologists don’t seem to be doubting the source. The source is The Journal of the History of the Behavioral Sciences and independent researcher Thomas Kelly who claims that the seminal When Prophecy Fails from the 1950s is bunk.
When Prophecy Fails is about cult leaders and cognitive dissonance from which they suffered. Kelly is daring to say a cult wasn’t as wrong about something and researchers who infiltrated it were more so. Surely he could be doubted. The debate is the efforts of a major scientific discipline over 70 years versus a guy with a box. Yet even psychology’s defenders don’t want to bet against the box.
If you’ve ever had vast swathes of your life decided by psychology, or watched some of the great questions of the day weighed by psychologists, or have sympathy for people with skin in the game so to speak, you might feel angry. Your reaction might not be scientific. You might not demand something be measured.
You would be wrong, though. The thing to measure is the thing itself: how much scientists lie or are consciously biased. Kelly takes the rare path and states it: “the researchers’ notes leave no option but to conclude the misrepresentations were intentional.”
As he suggests, the possibility of intention has a mathematical quality to it now. There doesn’t seem to be any option left but to consider it.
Unfortunately we’re very polite about lying in science. So much so that it took an anonymous researcher reading 2,500 papers to finally say, “Stop assuming good faith” in 2020.
Metaresearchers — that is, people who study other researchers — leave out intentionality except in extreme cases. This year, a paper considered only two possibilities on the topic: innocent bias, and deliberate misconduct like making up data. Deliberate use of the grey area is, and has always been, unmentioned and unmeasured. P-values move on their own. If the agent is involved at all, bias is due to forgetfulness or the unconscious.
None of the psychologists who said unconscious bias is a major factor in the replication crisis have updated their beliefs over 15 years. Yet p-hacking is obviously more conscious after one knows what p-hacking is, otherwise there’d be no point to discussing it. But psychologists still hang on to the unconscious explanation.
At least we can state the very obvious fact that we don’t know if questionable research practices are mostly intentional or not. We at least don’t know because we can’t diagnose at a distance. This goes unstated and it’s trivial to anyone outside.
The easy answer as to why reformers don’t want to make this rude suggestion about their colleagues is that they are biased too and want to preserve their place in academia like anyone else. A harder answer to face is that the measurement of lies is volatile. Once it’s out there, “researchers are lying” can be used to support any conclusion you like. “Researchers are lying” could be the mantra from the lowly extremist to the highest reaches of our politically powerful, including the most moderate meta-researcher in between. We might not want to know what’s in that box.
The more modern take, though, is consistent with the available evidence that nobody is surprised anymore. Maybe we already know what’s in the box.
Responses:
Both Devezer and Kelly responded with brief notes, which I appreciate very much. I changed two misleading words and linked to the full thread from Devezer for context.