In 2011, the replication crisis was started by a paper about ESP. The paper, called Feeling the Future by Daryl Bem, is a normal paper. It uses significance testing, exploratory analyses, closed data, and a fair amount of selective reporting. Where precisely it falls on a rigor spectrum is up for debate but it is a normal paper. Were it not, there would have been no reason to suspect the crisis was any wider than a single kooky hypothesis.
Psychologists around the world were activated by Bem’s paper and criticized it thoroughly. What arose was a realization, or a confession we’ll never know which, that normal papers can come to any conclusion they like.
Researchers then spent 15 years trying to prevent papers like Bem’s and while they’ve come up with some good methodology, these methods are rarely practiced and rarely taught. In the most scrutinized field, psychology, post-publication critique was recently measured at 0%. In a newly-scrutinized field, physical therapy, the rate of replication attempts was found to be 0%. The other metrics aren’t encouraging.
Now another happenstance has occurred not unlike that very embarrassing paper. The president and HHS secretary needed to show progress on autism. They took a recent paper and propagated it as strong evidence that Tylenol causes autism, boosting it into the 99th percentile of discussed papers, and triggering an adversarialist’s dream.
The paper, Prada et al., has now been pored over by right and left-wing commentators, and the results are devastating. I disagree with some critiques that Prada represents “no evidence.” “No evidence” is a lazy and totalizing comment that is harder to refute than it deserves to be. Choosing papers to call “evidence” or “not evidence” is another way to prove anything you like.
Nonetheless, this paper has gotten the scrutiny it deserves and many of the criticisms land. The first one is simple. It is a literature review, a design that is inexpensive, plentiful, and doesn’t make strong statistical claims. Published literature reviews, almost by definition, find what the author intended since they can be discarded at no great cost. One of the common criticisms of the paper is on conflict of interest. This is great, although with literature reviews, the first author to summarize the literature may simply be the most motivated one.
The critiques
American College of Obstetricians and Gynecologists
Association of Women’s Health, Obstetric and Neonatal Nurses
The Daily podcast (New York Times)
Updating list from PubPeer
Updating list from Altmetric
Excellent post. I have not read the critiques besides Sensible Medicine. As an undergraduate I attended a lecture by Daryl and Sandra Bem extolling the virtues of living an “androgynous” lifestyle, purely in a social, not a physical sense. Stuck in my mind for some reason, they had persuasive way of making a connection with the audience.