Priming is subtle suggestions in a paragraph, or a room, or a question that cause nebulous and just as subtle changes in behavior in someone who experiences them.

Why does this research persist despite it being so unlikely to be real? The answer, like many things, is p-values. Luckily, this poll asks approximately the right question so that we can translate the responses into math and say what p-values mean in terms of what the public understands: whether or not something is true.

To do this, we have to fight through a problem known since the 1920s, that p-values don’t guarantee anything, and even if we use what we’re really talking about, which is effect size, the sizes don’t correspond to “care” or “don’t care.” For example, psychology is undecided as to whether we should care about an effect of 0.1, whether it is important, or random. Gone are the days of effects big enough for grand public demonstration, of dino bones and electromagnetic fields. We don’t have a game show called RESIST. THAT. PRIME!

We can’t demonstrate anything with effect sizes, but we can use two numbers that are unambiguous: the ordinary threshold for publication (p < 0.05), and the acceptable proportion of false findings in the literature, and say that priming research is beyond repair.

The proportion of false findings for all of published science was estimated to be 50%. It’s a number very difficult to estimate science-wide, but we can agree that 50% false positives is bad. The ordinary interpretation of publication thresholds is that false positives occur only about 5% of the time.

For the portion of published findings that are false, let’s assume this interpretation:

In a single field with a single distribution of how big the effect probably is, you can control the rate of false discoveries by raising the standards for publication. So how high would the standards have to be in priming to keep the false discovery rate at a reasonable 5% level?

Here’s the distribution implied by the poll, counting only researchers who had an opinion:

By these estimates, no feasible standard would keep priming research at a reasonable level of false discovery. p-values would have to be lower than 0.001, 50 times stricter than today’s standards.

On the other hand, if you’re fine with 50% of research being false, then anything goes. Even a field as unlikely to be real and meaningful as priming can go on at current standards for publication, or even weaker standards. This is why accepting that half of research is not true is unreasonable, because almost no avenue of inquiry will ever be abandoned.

Worse, if you keep accepting the same standard of evidence, your ESP papers and your priming papers become surprising and “important” because ESP and priming are unlikely to be real.

The power of post hoc

This analysis depends on assumptions like any other and the poll is small. If we want more precise numbers, we should ask for exact estimates instead of buckets like “tiny.” And it assumes sample sizes stay around what they are in psychology. (If you keep p-value standards the same but increase the sample size standard instead, priming studies would cost about 6 times more than they do now.)

Additionally, there are so many researchers who are on the fence in the poll that interpreting “tiny” as 0.02 yields different results compared to 0.1. This is important because effect sizes of 0.1 are called “indispensable” or “crud” depending on who you ask. Despite anchoring to indisputable numbers, we’re back to arguing what “important” means. If respondents on the fence meant tiny is an effect size of 0.1 and the top group meant 0.2, then anything goes and current standards are fine, even for priming.

The consequences of being fine with 50% false research should seem eerily similar to the world we live in where anything you’d like to be true can be found in the literature somewhere. It’s also interesting to note that the entire range of implied standards, from open science radical to status quo radical, depends on whether you call 0.1 or 0.02 “tiny.”

So we must all live in awe of the immense power of post hoc reasoning that can change the definition of tiny for the purpose of publication. After the fact, one can simply say 0.1 is important. Findings that many would argue are indistinguishable from noise are suddenly essential building blocks that count in priming’s favor. Journals and researchers are equally innocent on this point since each can blame the other. The only anchor in the epistemic stew is that priming is really one of the worst. There was a replication crisis over it.

Rebuttal

A common rebuttal here is to say that researchers weren’t interpreting the 5% p-value standard as a 5% chance the finding is false. p-value standards don’t guarantee anything, actually. In fact, one could say at this late date that they have always assumed that half of published research is false and p-values are just a convention to weed out completely ridiculous claims. It would be very late as p-values just turned 100.

A journal caught up in a scandal covered by The New Yorker made this move a few weeks ago. They declared that it wasn’t that its case report was not true. All of its case reports aren’t true. They just neglected to say so. Again, we stand in awe of the power of post hoc reasoning. Not saying so in advance truly runs the world.

According to the poll, there really isn’t a compromise among the public, journals, and priming research. One has to go. You can’t tell the public that 50% of what they pay for is false and standards don’t need to go up as a field becomes more implausible. You can’t get journals to agree to 10% because it would mean publishing less and admitting 10% is actually quite strict. Even an optimistic estimate put science-wide falseness at 14%. If you could get the public and journals to agree to something like 10% that is a real stretch for both parties, priming would still be wiped out.

The larger point is that we’re not sure the public would approve of p-values at all. Not without an estimate of what they mean about a field. We also don’t know what reform would make this estimating work, and who does it, but the information we need is in the poll.

In another stunning act of normalcy, the project doesn’t seem to want its emails released. The grantees claimed theirs were proprietary. (NIH can’t do this for its own emails.) The shocking part is how you can really dive in anywhere into science and you eventually get to the same place: fudging the numbers. From my long running FOIA request:

February 25, 2026

Valery,

Have you had any luck with your investigation? Normally I would send you a blog post for comment. This time the email is the blog post. Please let me know if there’s anything you don’t want published in your reply.

To recap, I was told my FOIA request for emails about NIH’s Community for Rigor project was returning too many records to be processed. This is because the short name for it, “C4R” was matching what the FOIA office called “junk.” About a month later, you sent the number of records for various searches. The number of records for the term “C4R” in conjunction with other search terms was lower than “C4R” alone. I told you this was impossible by normal search term logic and you said you had noticed this already and were investigating. That was two weeks ago.

The format of the numbers was different for one of the search results as if someone had copied some numbers directly and typed one by hand. I think most people would conclude that the likely explanation is that someone wrote down the wrong information.

To further recap, Community for Rigor is one of the largest metascience projects ever funded at around 16 million dollars. The goal was to produce training material on rigorous scientific practices. It was designed so that NIH would be co-managing the project with the principal investigator at the University of Pennsylvania.

To most people who have had direct contact with science reform, it should have been clear that the project would be a failure. I think that NIH, and the eventual head of the project, Konrad Kording were finding out in real time how big the scandal in science is. Current NIH Director Jay Bhattacharya said recently that researchers might not find doing the rigorous practice of replication glamorous and may avoid this kind of project, even if it’s funded. All of these people were no stranger to problems in science, but maybe they didn’t know how ubiquitous and difficult these problems are.

The funding director Walter Koroshetz once implied some NIH grantees are “crooks.” Kording wrote about how researchers can’t understand the brain because they can’t even understand a microprocessor. One of the grantees under this project is famous for writing a book called “Calling Bullshit.” And Bhattacharya has had almost the entire scientific community denounce him. Maybe these people were aware there are problems, but they’ve never had to get researchers to actually follow rigorous practices that might be harmful to the researcher’s career. It’s easy to talk about abstractly. Getting a particular person to follow through and undermine their own reputation, or their employer’s, is very difficult.

The purpose is to say, as I’ve said before, you are not the story. I don’t really care if even the worst interpretation of this email chain is true, that the FOIA office is falsifying records. As you probably know, it would cost me thousands of dollars just to threaten to sue. And on the scale of NIH FOIA scandals, this one is tiny. Two presidents and two congresses have tried to get NIH to release COVID origin records and often failed. (I can only speculate on what’s in them, but for some reason, NIH has not been cooperative.) I spoke with the person who would know best on this issue, and he told me I’m getting “the runaround” from NIH.

What I do care about is what NIH has said it cares about, that the average scientist doesn’t know how to apply the scientific method. This wording, “the average scientist,” is from the C4R funding proposal. What NIH, Konrad Kording, Walter Koroshetz, maybe even Jay Bhattacharya found out is that the best scientists handpicked by NIH don’t know how to apply the scientific method, or don’t want to.

Case in point, the slide from the Community for Rigor conference that started this is categorically wrong:

The image doesn’t depict rigorous practices and project leaders at NIH know this. When the NIH project leaders give talks, they present this version of the scientific lifecycle:

What the image above depicts is the head of a lab doing what has always worked: have a stream of grad students and postdocs attempt to farm data sets for publishable results. First one wins. Don’t worry what stage of the process you’re in. They’re all “on ramps” and you can always “revise your hypothesis.” The slide is what has, and perhaps should, end your career. It ended the career of USDA leader Brian Wansink who was just doing what everyone else was doing. It shouldn’t be part of the handpicked project to educate researchers on rigorous practices.

After I emailed Community for Rigor about this and similar issues throughout the conference videos, the videos were removed from youtube without explanation.

We’ve known about how common poor practices are for a while. The reason why the scandal didn’t resolve itself is because everyone is doing it. There weren’t bad apples to weed out because the bad apple view of science is wrong.

I hope you and my readers can tolerate a little grandstanding here. My point is that I don’t want to get involved in every nook and cranny of this story, and that is where this conversation is headed. I can’t figure out if the discrepancy is due to you, or Lauren, or the central search department. Common sense says that you’re just the latest people who don’t want damage to your reputation or to science’s reputation and you’re not going to return the records for some reason. All I can do is keep asking and underscoring how important it is that you follow through.

However, the longer this goes on, the clearer it will be that the problem is the average scientist just as NIH believes by funding this project. This could go on for another year, and really it’s money well spent because NIH did an experiment by funding this project and these are the results. Hidden videos, hidden materials, and of course emails.

Please let me know about the results of your investigation and whether or not you have an estimate for completing the request. In case it wasn’t clear, I am not blaming you or Lauren for the delays. I have great respect for all of the people I listed above and I wouldn’t blame them either. You’re welcome to ask to be anonymous, and if you want to hand me off to someone else, I’m happy to continue with them.

Best,

Alex

The numbers

The NIH FOIA office usually changes the subject without commenting on previous implausibilities and they did so in this case. They are most certainly writing down the wrong information, though. A few days later, I got a reply that included statistics for the same searches again:

The numbers had been changed and the formatting was consistent this time. There was no mention of the changes in the email. The impossibility of adding more “OR” terms and getting fewer results had been fixed, so to speak, due to the deletion of 517 emails from program officer Devon Crawford’s total.

Did someone delete 517 emails from Dr. Crawford’s mailbox? It seems unlikely since the other search should have been reduced by 517 too. After a year of this FOIA request and eight months of writing about it, those emails disappeared just as I complained that the numbers don’t make logical sense. (She and the FOIA office didn’t respond to a request for comment.)

There are a few other inconsistencies in the results. Since the numbers seem to conform to whatever the FOIA office wants to argue at that moment, I figure it’s best not to tell them. I’m starting to understand the typical response to the crisis in science. Maybe most people figure it’s best to leave science alone and wait for it to dig itself deeper.

These physics defenders have lined up against critics who say certain non-empirical theories are nonsense. The defenders claim 1) that these theories are not all of physics, 2) not a lot of physics, and 3) even if they were, it’s fine because the theories might be true.

All of these responses focus on the theories, not the glaring issue of empiricism physicists have been elaborately underscoring for decades.

In 2015, string theorists turned explicitly against evidence. They brought out an intellectual superweapon, something that would not only save their theory but all theories. By the time the philosopher behind this admitted physicists may have misunderstood, the genie was out of the bottle.

Despite this obvious danger to the scientific bedrock, empiricism gets lost in squabbles over rhetoric: should Physics World have said “cult” or “cult-like?” Should they have put “obedient idiots” in quotes?1 On YouTube, Professor Dave, a science communicator with 4 million followers, escalated the fight well beyond his “poisoning society” line to the idea that we need “warriors for science” in his critique of Sabine Hossenfelder’s frequent use of “science is broken” and “I don’t trust scientists.”

Hossenfelder’s rhetoric is no doubt blunt, but she’s made clear that the issue is a lack of attention to empiricism. Before she started, as she admits, “trying to annoy physics into doing something,” she wrote the article, the paper, the book, spoke at the conference, and did boring videos that few watched. The fact is that showing that something is non-empirical in a paper doesn’t enlist every physicist on Earth to your cause, even though it should. It gets a few dozen citations. What she gets now are longer and longer responses that avoid the issue.

Here is the simplest form of the argument that can’t be misunderstood: non-empiricism is an intellectual superweapon. It gives the possessor extraordinary and unearned powers of false discovery. Defending science from it is exhausting because many would like to possess it, and it quickly gets turned on you.

The debate has become so detached it seems to be about denying the superweapon exists, let alone who’s using it. Professor Dave isn’t telling the troops what they’re fighting for. What he says over and over, in a style that can only be called “relentless,” is that defending academia is defending empiricism.

The Misunderstanding

In 2015, physicist and philosopher of science Richard Dawid published what would become his most cited paper. It rests on the definition of the word “confirmation,” which he has rewritten so it looks like a finding:

“If one measures confirmation in terms of increase in degrees of belief, as Bayesians typically do, T is confirmed by E whenever P(T|E) > P(T). Throughout the paper, we use this inequality as a criterion for when a piece of evidence confirms a hypothesis.”

Dawid doesn’t cite anyone here because there’s no one to cite. The Bayesian epistemologists he cites elsewhere use different definitions and 1. make clear that “confirms” means “is evidence for” not the usual definition “verify” and 2. use “iff” instead of “whenever”:

The use of “confirms” is not a finding. It is the definition of a confusing philosophical term. What he’s doing is a sort of scientific arbitrage, importing a trivial term from philosophy into a field where he knows it means something else. The authors he cites, who are also his co-authors, had no problem making this clear for their readers in philosophy.

Dawid recycles this definition in a later paper and again ties it to ordinary Bayesians by rewriting Bayes’ theorem:

Somehow dividing both sides by P(T) means Bayes’ theorem “implies that data E confirms H if P(E|T) > P(E).” This is algebraically equivalent to the definition for the word “confirms” above, except he drops the “if and only if.”

Confirmation, as before, can occur even if P(T|E) is near zero, which means that a theory is not worth the paper it’s printed on.

But this is just evidence that Dawid knows he’s guilty. The evidence against the formulation itself is simple. The fact that P(T|E) > P(T) is not confirmation is self-evident. It means you can confirm the moon is made of cheese by its color. The definition is so extreme it is the basis for the famous Raven Paradox. Dawid doesn’t cite this or any other dissent.

Furthermore, ordinary Bayesians would say that if you put one of the 10⁵⁰⁰ versions of string theory into Bayes’ theorem, you get something like 1/10⁵⁰⁰, or zero, plus or minus some evidence of which there is almost none. At a conference organized to debate the “confirms” paper, Dawid himself said there is none. According to Bayes’ theorem, zero is a very good estimate.

Benefit of the Doubt

Let’s make a strong case for Dawid, though, and say all he is guilty of is endorsing an extreme form of cherry-picking. Any evidence you can find for something does technically increase the odds it’s true. One has to admit that other evidence may decrease it, but so what? It doesn’t matter if he’s used a word confirmation that might look good on a grant application or in a newspaper. He’s guilty of fighting for his side with a little wordplay. Let’s also suppose this society we’re so afraid of poisoning can survive extreme cherry-picking and claiming any evidence as confirmation. (It can’t.)

The really egregious part is that the point of Dawid appealing to the power of evidence is to ditch evidence. This is just one paper in his careerlong argument for non-empirical confirmation: You can accept your theory on the thinnest possible evidence. If you can’t find any evidence at all, that’s fine too.

In a later paper, Dawid cites his own work to endorse “arguments that resemble empirical confirmation.” (For logicians, an argument that resembles confirmation but isn’t confirmation can’t raise the probability that a theory is true according to that if-and-only-if Dawid left out.)

The Warnings

George Ellis, one of the world’s most prominent physicists, warned against Dawid in 2014. His and other non-empirical ideas were called the “most dangerous ideas in science” at the time. Here’s Ellis at that conference doing all that you can do sometimes:

After the conference, Ellis claimed the same victory, saying that string theorists had admitted that the theory wasn’t “’confirmed’ in the sense of being verified.” Two years later, Dawid responded in the literature, stating conclusively that there’s an “unfortunate mismatch” in the way the term is used in philosophy compared to physics, which may lead to some misunderstanding.

The Future

None of this stopped the paper from being cited. The “no alternatives argument” is still being used today, ironically to show that there isn’t a crisis and the critics are just sore losers.

Sour grapes, as Professor Dave would say.

There’s a much larger audience on YouTube than Physics World could dream of, where Professor Dave defends string theory as empirical. He might be benefiting from the superweapon unwittingly, who knows. But he seems to think the issue is as serious as it gets. The question is whether or not he can be convinced that he’s the one with the bomb.

Two of these questions in psychology were probed in 2016 and never repeated in such an explicit way since: “Are we in a replication crisis?” and “If so, why?”1

Spencer Greenberg’s group at Clearer Thinking published results of a survey a few weeks ago that helps to end this drought and may help us know if anything is working. They asked psychologists if psychology is still in crisis and why.

Clearer Thinking has been picking up academia’s slack in several ways, notably rapid replication, and proposing solutions and hypotheses too impolite towards academics to have come out of academia. These are not outlandish or overly insulting ideas. For instance, the proposal that scientific reforms might act as patches to a leaky pipe that causes greater pressure on other cracks, or that replications should be done soon after publication to incentivize researchers to practice greater rigor. These are good ideas but they are too impolite. That is the background on Clearer Thinking, and academia lobbing itself softballs happens to be the whole problem in a nutshell.

The problem with Clearer Thinking is they are still a little bit polite, which I hope to correct here.

Are we in a crisis?

The poll almost perfectly agrees with Monya Baker’s 2016 poll in Nature, except most respondents now say “substantial progress has been made.” Baker’s poll found 90% of researchers believed there was a reproducibility crisis in science, and 52% said it was significant. The psychologists in the poll agreed (n = 44) at a slightly higher rate. In other words, 93% of psychologists would have used the word “crisis” then, and 95% would use it now except that most now believe a lot of progress has been made nonetheless.

What are we doing about the crisis?

The poll asks what improvements participants have made in their own research, and the ones who have made changes generally adopted preregistration (61%). So if we’re looking for progress, the best place to start might be preregistration.

As the survey suggests, preregistration is a clear and obvious winner among scientific reforms. It addresses base rate neglect, p-hacking, HARKing, and publication bias. Preregistration is backed by 2,000 years of thought on post hoc reasoning. It’s backed by statistics and something better than statistics: logic. Preregistration is already part of what protects us from dangerous drugs and medical devices. However, if preregistration adoption is the significant progress these psychologists are citing, it has not shown up in any studies of the literature.

The percentage of papers in psychology that are preregistered was measured in the early 2010s and again in 2022 and it rose from 2% to 7%. This is significant when you consider the intransigence of scientific practice and its size, the billions of dollars wrapped up in publishing and so on. However, a 5% increase doesn’t substantially move the replication rate.2

Much more importantly, the slow adoption of preregistration shows what type of problem scientific reform is. It shows 21st century reform isn’t the kind of thing that just catches on.

Compare preregistration with a practice that slightly increases one’s chances of publication, the Benjamini-Hochberg procedure, which has been cited 120,000 times. There are no non-profits set up to encourage use of the Benjamini-Hochberg over the less forgiving Bonferroni. Adoption happened automatically. 21st century reforms are not like this. They’re spinach, not ice cream.

One of the most important questions in the epistemic life of Earth’s 8 billion people is: Are scientists going to adopt reform or are they going to resist? This is the question we need answered. We need to interpret what a 5% increase in adoption — maybe 6 or 7% by now — means.

Change that reduces publication, even in metascience’s home turf of psychology, is spinach. We know it is. Even the small amount of progress is among self-selected adopters. They could be the ones who can afford to preregister. Their work is more certain to start with. Their subfield is more vague. Their job less in doubt. It moves the needle of discovery much less than its own prevalence might suggest, and it doesn’t predict future adoption.

The status of p-hacking in 2025

The survey asked how serious p-hacking is at the top 5 psychology journals. The results look like this:

To put “top 5” in perspective, the top journals occupy a hierarchy that is shaped like other highly competitive environments:

Only a few are elite and many exponentially less so. Impact factor is not a good measure but it does measure influence. If p-hacking is a problem in the elite, its findings are at least influential there. It also stands to reason that it’s probably a problem everywhere.

Why do researchers bend the rules?

Famously, early reformers in psychology blamed the unconscious and forgetfulness. Feynman blamed “fooling yourself.” These charitable explanations are certainly partially true but they are starting to lose steam as predictors. If being charitable was supposed to be a catalyst, it is failing at that too.

In the free-form responses to the “reforms adopted” question, nobody mentioned combating their unconscious or forgetfulness because nobody believes those hypotheses. Nobody has bothered to find out if they are correct because they’re obviously not. A better predictor consistent with available evidence is Darwinian: Academics are p-hacking for survival in their profession and in their life’s work. They are doing so most likely on purpose or in deep denial. If they didn’t bend the rules, they might not have a job and all their friends, loved ones, and the rest of science would blithely continue on without them, winning Nobel prizes, educating students, and truly discovering some things. Discovering less maybe, but it’s the price of survival.

The replication rate

In another indication that 95% of psychologists are right that there still is a slight crisis, participants guessed the replication rate for the top 5 psychology journals and the average guess was 55%. We’re used to this ≈50% metric now, but in 2005 it was shocking. After all of this progress, the participants’ favorite journals are still only around a coin flip. Hopefully progress is not just inuring us to the numbers.

One optimistic way to interpret the higher adoption of reforms in some top journals (as high as 58% by one count) is to say that true discoveries will come out of better journals. The total replication rate doesn’t matter as long as some good work has been done and it shades out the rest. This is truly hopeful and it is possible. Unfortunately, the data didn’t indicate elite work either. Respondents were asked which findings have held up over the last 15 years. Other than a small cluster who said metascience, and a larger one who said there aren’t any, every answer was different.

Responses:

There’s much more in the data and Greenberg says there will be more analyses in the next few months. Announcements of new posts come through their newsletter. Their post was by Director of Replications Amanda Metskas. I clarified the replication estimates came from participants after a review by Clearer Thinking.

Both Baker and Clearer Thinking shared data. The code for this post is here.

Another would be to doubt the veracity of the news. It makes sense to doubt that a single independent researcher without apparent credentials could overturn 70 years of a scientific discipline by opening an unsealed archive called “box 4.”

Surprisingly, psychologists don’t seem to be doubting the source. The source is The Journal of the History of the Behavioral Sciences and independent researcher Thomas Kelly who claims that the seminal When Prophecy Fails from the 1950s is bunk.

When Prophecy Fails is about cult leaders and cognitive dissonance from which they suffered. Kelly is daring to say a cult wasn’t as wrong about something and researchers who infiltrated it were more so. Surely he could be doubted. The debate is the efforts of a major scientific discipline over 70 years versus a guy with a box. Yet even psychology’s defenders don’t want to bet against the box.

If you’ve ever had vast swathes of your life decided by psychology, or watched some of the great questions of the day weighed by psychologists, or have sympathy for people with skin in the game so to speak, you might feel angry. Your reaction might not be scientific. You might not demand something be measured.

You would be wrong, though. The thing to measure is the thing itself: how much scientists lie or are consciously biased. Kelly takes the rare path and states it: “the researchers’ notes leave no option but to conclude the misrepresentations were intentional.”

As he suggests, the possibility of intention has a mathematical quality to it now. There doesn’t seem to be any option left but to consider it.

Unfortunately we’re very polite about lying in science. So much so that it took an anonymous researcher reading 2,500 papers to finally say, “Stop assuming good faith” in 2020.

Metaresearchers — that is, people who study other researchers — leave out intentionality except in extreme cases. This year, a paper considered only two possibilities on the topic: innocent bias, and deliberate misconduct like making up data. Deliberate use of the grey area is, and has always been, unmentioned and unmeasured. P-values move on their own. If the agent is involved at all, bias is due to forgetfulness or the unconscious.

None of the psychologists who said unconscious bias is a major factor in the replication crisis have updated their beliefs over 15 years. Yet p-hacking is obviously more conscious after one knows what p-hacking is, otherwise there’d be no point to discussing it. But psychologists still hang on to the unconscious explanation.

At least we can state the very obvious fact that we don’t know if questionable research practices are mostly intentional or not. We at least don’t know because we can’t diagnose at a distance. This goes unstated and it’s trivial to anyone outside.

The easy answer as to why reformers don’t want to make this rude suggestion about their colleagues is that they are biased too and want to preserve their place in academia like anyone else. A harder answer to face is that the measurement of lies is volatile. Once it’s out there, “researchers are lying” can be used to support any conclusion you like. “Researchers are lying” could be the mantra from the lowly extremist to the highest reaches of our politically powerful, including the most moderate meta-researcher in between. We might not want to know what’s in that box.

The more modern take, though, is consistent with the available evidence that nobody is surprised anymore. Maybe we already know what’s in the box.

Responses:

Both Devezer and Kelly responded with brief notes, which I appreciate very much. I changed two misleading words and linked to the full thread from Devezer for context.

In 2016, a prominent editor and social psychologist Susan Fiske called metascientists “methodological terrorists.” Statistician Andrew Gelman wrote a response on his blog that became a classic. To onlookers, the importance of Dr. Fiske’s post wasn’t just that she was being hostile and ad hominem. It was an admission that psychology didn’t have a better defense against the replication crisis that was having its first empirical confirmations.

It is therefore interesting to try to understand the defense that physics has put up against a major threat to its credibility, the “String Theory Wars.” Works in this genre of critique include Farewell to Reality, The Trouble with Physics, Lost in Math, and recently, The Ant Mill. These are all base-rate books. Their authors doubt certain theories in part because there are trillions of variations on them and, without empirical evidence to promote one variation over another, the probability that a particular variation is true is near zero.

The threat to physics’ credibility is, in other words, Bayesianism, a threat that all of academia has been fighting off since the 1800s. It is unlikely that physics has a good argument against Bayesians because no science has had a good argument. It’s unlikely that physics has been dealing with it internally because no science has.

Bayesianism means being able to publish less. It admits the degree to which results are subjective. It admits the tenuousness of expertise. These are all things that academics don’t like.

Physics is not string theory

There’s a long tradition of debate between physicists who believe string theory and other non-empirical and implausible theories should taint physics as a whole, and physicists who think string theory is alone and on the fringe.

What’s dismaying to onlookers is that a lot of mainstream physics’ defense against critique has been ad hominem. It has not stuck to the truth of string theory, or whether or not more of physics is also untrue. This is the history of the debate, to paraphrase physicist Angela Collier:

• Some string theorists wrote popular science books for 30 years.

• The great success of string theory these books portrayed was a “lie.”

• In the early 2000s through to the present day, other physicists wrote books to counter this lie but they also tainted physics as a whole.

• The public stopped wanting to support physics because of this and now “we have to earn their trust back and that sucks.”

After Dr. Collier summarized the debate in this way, her YouTube channel was flooded with commenters she called “physics haters.” She took great umbrage at this, and in September, made an argument that physics should not be put in the same bucket with the lie.

Dr. Collier’s argument was an extreme form of the bad apples defense. When someone loses credibility, science is free to disavow them and claim they were a bad apple, and it often does. The public is left to sort through representatives of science rather than scientific facts.

Dr. Collier makes matters worse by wading in, particularly on the “melon heads” who visited her channel. She calls these alleged physics haters “stupid,” “pathetic,” and listening to opposing physicists like Sabine Hossenfelder “embarrassing” over a dozen times.

The irony is that these commenters don’t represent all of the public either, and just because some used absolutist language like “physics is broken” doesn’t mean there isn’t a more nuanced point to be made. It has been made by physicists for decades. Also, the public generally can’t distinguish her good topics like antimatter from string theory or anything else. For all the vast majority of the public knows, they could be accused of being stupid and pathetic for watching Angela Collier in a few years.

The only solution is to try to debate the probability of whether or not string theory and the rest of physics is true. One way to do that, as the authors above have, is to count the number of possible hypotheses in string theory, a number first published in 2005, and argue that this means that string theory is not true. This is the same argument that was successful in Genome-Wide Association Studies and in Why most published research findings are false. It is an argument the public can understand and the equation is easy to prove.

The question is, if this argument is credible enough for physicists to tell us, why isn’t it credible enough for them to tell each other so we don’t need to get involved?

The answer is simple. Bayesianism is how physicists think. Frequentism is how they publish. Frequentism keeps the lights on.

The insults

Philosopher Robert Crease wrote a review of The Ant Mill last month in Physics World that is quintessentially insulting in a way that would make a youtuber blush. He compares the author, Jesper Grimstrup, to a jilted ex, one you barely remember but has written a whole book about you.

The “ex” part comes from Grimstrup’s departure from academia, what Crease calls a “bad breakup.”

Academia shouldn’t listen to its ex Grimstrup, the review says. Listen to current academic Richard Dawid, a string theory supporter. What would the ex analogy be without a new beau?

String theory opponents have criticized its lack of falsifiability, the domain of perhaps the 20th century’s greatest philosopher of science, Karl Popper. Academia’s new beau doesn’t make elaborate demands like Popper and Grimstrup. He accepts the way things are and simply writes them down:

“Dawid, you see, is making the formalism follow the practice rather than the other way around.”

Dawid himself isn’t as quotable as string theorists may like. He admits string theorists have to “make the best of it” in a “drought” of empirical evidence. Where paradigm shifts were once due to great empirical confirmations, they are now brought on by lack of evidence.

Critiquing Popper is not new. String theorist Leonard Susskind called critics “Popperazzi.”

Going after falsification is also a supremely poor accommodation. If you can overturn the greatest philosopher of science of the 20th century, you should publish it, then go back to physics, not wait until the cards are on the table.

Along with the insults, it’s telling how much these academics describe their own bodies in space, how it felt to read the physics haters. Collier describes how it “feels bad” continually. Crease ends up deciding that The Ant Mill is so “over-the-top” that he finds reading it a “hoot.”

How they felt is not even good evidence for them, let alone for us.

Why use the word “crisis?”

String theory is not all of physics. It is not even a lot percentage-wise. However, the argument for crisis doesn’t solely rest on how much of physics is nonsense because string theory is mixed in. It rests on the fact that all fields have largely rejected Bayesianism.

Bayesian physicist Giulio D’Agostini published a paper in 1998, well before Bayesianism became news in 2005. He describes a recent discovery claim in physics. Although the discovery had a tiny p-value, he couldn’t get any of his colleagues to stake even odds that it was true.

This means that, despite the finding being publishable, experts didn’t think there was even a 50% chance it was true. It was clear to D’Agostini that physicists believed in Bayes’ theorem but didn’t use it when publishing.

Predictions are good evidence. Unfortunately, we can’t go back and make D’Agostini’s paper into a famous prediction. It was certainly predictive, though. “Less than 50%” became the predominant finding for 20 years. In physics, experts being secret Bayesians predicts that the base rate for a prominent theory like string theory might be calculated, which would further confirm to physicists that it’s not true. It predicts that other physicists wouldn’t be able to make a Bayesian argument against the theory because they regularly doubt findings behind the public’s back. Making a Bayesian argument would sink all kinds of ordinary physics papers too.

With no good alternatives, some physicists would spend 20 years writing books for the public instead.

The other reason to use the word “crisis” is self-explanatory. Getting rid of empiricism lets in a lot of things we’d rather keep out of science like religion, mysticism, and quackery. Given physics’ place in science and its obvious ability to inspire the foundations of scientific philosophy, continued empiricism during the drought seems like the least we could ask.

What had been theory in the 2000s and confirmed empirically in the 2010s was now accepted by most researchers. It turned out they knew partially from their own experience that it was true and worrying. The loop had been closed, it seemed.

The “hard” fields of physics, chemistry, and engineering fared best in the poll. Now, thanks to physicists Sabine Hossenfelder and Jesper Møller Grimstrup, we may need to start rethinking what became a popular idea, that the crisis is due to the “softness” of social psychology (and, for some reason, several other fields).

Reading Hossenfelder and Grimstrup’s books Lost in Math and The Ant Mill is a delight and there’s real metascience in them. The crisis in physics is a natural experiment in which many of the other potential causes of the crisis from other fields have been removed, leaving only the pure passion for publication and funding.

In physics, you take away the questions of validity and flexible interpretation from psychology. You take away the closed data and incomplete methods from cancer biology and others. (The existential threat of not getting along with your mentors and co-authors and the other familiar pathologies of science are still there.)

This experiment suggests that there’s one common factor in science, the fact that the people who support you have no idea what you’re doing and you have little incentive to tell them.

Both authors have made an argument the public may have a chance of understanding, that hypotheses chosen from a near infinite set with nothing to recommend them have no chance of being true. Grimstrup estimates there are 10⁵⁰⁰ possible hypotheses in string theory. Each can be a paper.

It turns out, according to these very detailed accounts, that physicists will happily milk string theory just as psychologists invented new primes.

Familiar patterns

Much of the observations in these books are familiar. They have been observed in other fields and by other reformers. Goodhart’s Law, the Matthew Effect, a disturbing element of evolution in science towards worse practices.

Some observations are refreshing that you don’t often see, certainly not in psychology. The idea that researchers should be judged not only by the compromises they make for the sake of their careers, but for their silence in the face of such an obvious elephant in the room. Grimstrup even calls for — nondestructive — reform from outside. Hossenfelder has done more than her share on that front.

These are, in other words, ordinary observations from an extraordinary place few understand and few would suspect.

Physicist Alan Sokal once derided the encroachment on academia by those who deny the existence of objective reality. Years later, physicists started to deny reality too. Hossenfelder’s conversation with the eminent cosmologist George Ellis in which he laments this development, that some physicists are claiming theories don’t need to be supported by experiment, and feels powerless to do anything about it, is particularly affecting.

The most worrying part of these books is there doesn’t seem to be any natural limit before we’ve written about all 10⁵⁰⁰ hypotheses. Grimstrup paints a picture of physics most certainly continuing with groupthink fueled by government grants and the metric boosting multi-authored paper.

The power of the crowd

Hossenfelder has 1.7 million followers on YouTube and is an influential physicist in her own right. Her discussion with George Ellis was on something they’ve been saying since 2014. Hossenfelder and Ellis represent the widest influence imaginable both popular and scientific. We should be worried that the most powerful people who have worked in science, people like Walter Koroshetz, Jay Bhattacharya, Jacob Cohen, Daniel Kahneman, Csaba Szabo, Simine Vazire among many others, are often powerless to direct the crowd.

John Ioannidis is the world number 8 in citations. It doesn’t mean he gets what he wants, and published research is still theoretically, and many times now empirically, half wrong.1 It is certainly not yet Bayesian.

The books underscore that going against the grain in science is lonely and one’s certainty starts anecdotally and personally. Grimstrup’s copies of emails from professors, editors, and peer reviewers are not as powerful as a conversation he recounts with his friend in which his friend admits he can’t take the chance on any new theories as much as he may like them. He has to play the game and look for easy publication and surer funding.

It is almost literary, to know your tribe’s ideas are wrong but you and your loved ones can’t stop pretending to believe them. In these moments, Grimstrup excels.

I spent seven years at the Niels Bohr Institute where I have been at countless seminars on string theory and I have never, not even once, heard anyone in the string theory community mention the very obvious problem that this approach has: its inability to produce falsifiable prediction and the fact that it has failed in those few occasions where it has made contact with data.

The books are a delight, and horrifying. In a way they are reassuring. The crisis has reached the basement as Hossenfelder puts it, the bedrock of science. Although it doesn’t look like there will be any end forthcoming, it is clear now that there is no field where the crisis could have stopped. The bedrock might be our only hope.

Psychologists around the world were activated by Bem’s paper and criticized it thoroughly. What arose was a realization, or a confession we’ll never know which, that normal papers can come to any conclusion they like.

Researchers then spent 15 years trying to prevent papers like Bem’s and while they’ve come up with some good methodology, these methods are rarely practiced and rarely taught. In the most scrutinized field, psychology, post-publication critique was recently measured at 0%. In a newly-scrutinized field, physical therapy, the rate of replication attempts was found to be 0%. The other metrics aren’t encouraging.

Now another happenstance has occurred not unlike that very embarrassing paper. The president and HHS secretary needed to show progress on autism. They took a recent paper and propagated it as strong evidence that Tylenol causes autism, boosting it into the 99th percentile of discussed papers, and triggering an adversarialist’s dream.

The paper, Prada et al., has now been pored over by right and left-wing commentators, and the results are devastating. I disagree with some critiques that Prada represents “no evidence.” “No evidence” is a lazy and totalizing comment that is harder to refute than it deserves to be. Choosing papers to call “evidence” or “not evidence” is another way to prove anything you like.

Nonetheless, this paper has gotten the scrutiny it deserves and many of the criticisms land. The first one is simple. It is a literature review, a design that is inexpensive, plentiful, and doesn’t make strong statistical claims. Published literature reviews, almost by definition, find what the author intended since they can be discarded at no great cost. One of the common criticisms of the paper is on conflict of interest. This is great, although with literature reviews, the first author to summarize the literature may simply be the most motivated one.

The critiques

American College of Obstetricians and Gynecologists

Association of Women’s Health, Obstetric and Neonatal Nurses

The Daily podcast (New York Times)

Cremieux Recueil

Jesse Santiano MD

Sensible Medicine

Secrets of Grimoire Manor

Updating list from PubPeer

Updating list from Altmetric

The last few months in science reform has seen two prominent claims that research is like an evolutionary ecosystem by Richard McElreath, and by Thomas Morgan at the University of Arizona. Evolution is an excellent metaphor, made famous by McElreath’s paper The Natural Selection of Bad Science (2016) and even more so by the phrase “publish or perish.”

The problem with invoking evolution is that it’s hard to claim an evolved system where the strong cheat can be fixed with more rewards, ones the strong aren’t going to get ahold of.

The current plan for replacing “publish or perish” is “don’t publish as much but we’ll make sure you live, we promise!” It isn’t going to work. We need “publish poorly and perish” like every ecosystem and in fact, profession.

Richard McElreath, who did not respond to a request for comment, is a legendary Bayesian author. The rewards in his piece are many, such as the 18th century reference to careerist “bread scientists,” and McElreath’s general disagreement with open science tisk-tisking its way to a better system. Dr. McElreath is an exceptional educator and “The Natural Selection of Bad Science” is one of the best papers in metascience.

In the 2010s, as psychologists talked about subconscious, or forgetful p-hacking, McElreath and his co-author Paul Smaldino got at the unfortunate truth that science is an ecosystem like any other where the strong survive, sometimes for the wrong reasons.

Ten years later, McElreath accepts the consequences of his own paper by again comparing working scientists to an evolutionary system. So he has no excuse for making the same mistake others have made by advocating for new tweaks and tunes that will fix it using academia’s own meager appetite for reform.

Academia’s appetite for reform is truly minuscule. Science is a system that is indeed “a bit shit,” as he says. McElreath is a Bayesian, so its major reform, yet to be realized, should have swept through around 1763.

Foxes and bunnies

In short, McElreath thinks many of the standard reforms won’t work, and proposes some of his own, which are, for instance, training in data management and statistical modeling. He correctly points out that other high-level officials in science like himself have enjoyed evolutionary success in a bad system and so may be the worst instead of the best. We’ve seen many instances of this. Gino at Harvard, Tessier-Lavigne at Stanford, Wansink at USDA, Masliah at NIH.

McElreath’s error is ubiquitous. Despite having simulated the effects of labs propagating bad methods and “dying” if they don’t, the real horror is the honest researchers not knowing data management while dying. In other words, instead of the more obvious and tested system of checks and balances, of “foxes and rabbits” in evolutionary terms, we can breed the bunnies to jump off a high cliff once they’re not serving the ecosystem anymore. Evolution doesn’t work that way.

It certainly hasn’t worked that way in science. As food (discovery) becomes scarce, scientists feast on p-hacking and base rate neglect. This is obvious. As McElreath says, there are things everyone already knows. I suspect this is one of them.

The fact is that bunnies don’t stop eating even when faced with ecological collapse. Scientists don’t stop publishing and methods don’t stop propagating. Survival is how they got here.

We love these particular bunnies. They’re brilliant and kind. However, when it’s time for their careers to suffer, they do untold violence against the scientific record and thereby society to save their own skin. As almost anyone would. Their skills in data management and statistical modeling don’t matter.

No excuse is necessary to say that researchers who use poor methods shouldn’t have careers, but in case this seems cruel, consider that publish or perish is a silly metaphor. Science — as scientists have endlessly pointed out — is actual life-or-death for the rest of us.

The foxes this site has advocated for are independent post-publication reviewers separate from academia. They may not be as educated as the bunnies but they know when a paper should perish. It is often obvious, held back only by the researcher’s indomitable will to survive.

I believe, as McElreath does, that this system will arise with or without our help and it already has. In the United States, the foxes are now, by law, political appointees and the ecosystem is going through collapse.

The Natural Selection of Bad Science must be read, and we have to accept what it means. The fact that McElreath himself does not is the perfect illustration of academic reform.

Getting informal advice from a lawyer, and telling NIH I had, helped immensely.

The subject of the FOIA request, covered previously, is called Community for Rigor or C4R. It is an NIH project that supports the production of training materials on research rigor for schools and labs to use for free.

NIH anticipated the project would be difficult. Rigor costs researchers money, in future funding and in reputation. So NIH stipulated that they the funder would have an ongoing leadership role on the project. This is unusual.

The hole in NIH’s logic was that the professors might not follow through and produce material on these costly rigorous practices, or NIH might not have the fortitude to accuse their own grantees of not following through. Perhaps the average grantee at Community for Rigor would be like the “average scientist” referred to in the C4R proposal who “fails to consistently follow the relevant principles.”

It was readily apparent that this nightmare scenario was coming true at C4R’s first annual conference, and in videos of the conference that C4R removed — apparently in response to feedback — and I’ve restored here.

How cynical does a person have to be to anticipate that a project that prides itself, and in fact was named after, grassroots governance would take down videos after criticism? Not cynical enough it turns out. I didn’t expect NIH would use the grant number to search for C4R emails. As if the director of Research Quality discusses the culmination of his twenty years of effort on research quality as UC2NS12836.

Of course, the NIH FOIA office knew this because the matching emails referred to the project as “Grassroots Rigor” and “Community for Rigor (C4R)” and because I specified all of those names.

The other ironies are too numerous to list here.

The point is that NIH doesn’t seem to care about FOIA. Six months of correspondence had made that clear already.

I emailed eight officials connected to the project and asked them if they thought the FOIA response reflects the request and whether they’d discussed Community for Rigor in emails that weren’t included. None responded.

The Watchmen Problem

The other point is that metascience keeps getting more meta. If researchers at one level of “watchmen” aren’t rigorous, it’s up to watchers of the watchmen and on and on. As you go up the chain, watchmen become fewer and fewer, and unless actual legal repercussions are involved, less powerful.

Science does self-correct to some degree. It would be obtuse to ignore this. However, the degree to which it patently does not correct itself can be extreme. Some “meta” failures from the last few years:

Fabricated data in research about honesty. You can’t make this stuff up. Or, can you? (July, 2023)

The Distortions of Joan Donovan Is a world-famous misinformation expert spreading misinformation (June, 2024)

This Study Was Hailed as a Win for Science Reform. Now It’s Being Retracted (September, 2024)

The Business-School Scandal That Just Keeps Getting Bigger The rot runs deeper than almost anyone has guessed. (November, 2024)

NIH is set to spend a lot of money on metascience. I wholeheartedly support spending money on metascience. In fact, if all the funding results in a bunch of Community for Rigor projects, I wouldn’t be opposed. It’s better than nothing and some of C4R is great. However, we have to know the effect of paying the same people who caused the replication crisis, the average researcher, more money. If it’s all for nothing because the remaining incentives defeat the purpose, we should know that and be cynical enough to anticipate it.

A thank you note

With overwhelming gratitude, I have to thank the Public Citizen Litigation Group and Adina Rosenbaum for helping me solve this by telling me the right questions to ask the FOIA office. I told her I hoped this reminds her this work is important and I meant it. Contacting Public Citizen was incredibly helpful and probably decisive.

Requesters who ask for grant applications/materials usually want to receive only material that will help in understanding the process that led to the awards, or to improve their own methods of drafting grant applications. Requesters usually do not want material that applicants believe would harm them if released. We have found that the spirit of the FOIA can be enhanced through a spirit of cooperation1 among requesters of materials and those who submitted the materials.

[Emphasis added.]

I’ve written about the subject of this FOIA request extensively. It’s called Community for Rigor, a “grassroots” project to produce educational materials for researchers on scientific rigor. I decided to FOIA Community for Rigor after a particularly terse response from them in February that went like this: “All the information you’re requesting is part of of our internal planning process. If you would like to stay up to date on our progress, please make sure to sign up to our mailing list.”

As I stated to NIH in the request, the FOIA had a scientific purpose. This purpose is the same as the purpose of my emails to Community for Rigor: a small number of professors funded clearly don’t know much about rigor. They, I suspect, are like the schools they work for and resist learning about or teaching rigor for financial reasons. The request:

According to NIH research (https://doi.org/10.7554/eLife.55915), higher education institutions have resisted teaching rigorous research practices that may prevent future funding opportunities. “Grassroots Rigor” [Community for Rigor, or C4R] was intended to produce training materials on rigorous research practices. The scientific question this request is intended to answer is whether or not the higher education institutions funded under this grant continued to resist teaching these practices. (Date Range for Record Search: From 08/08/2023 To 02/09/2025)

At minimum, the search needed to include the email accounts of the three main NIH officials in charge of the project: Walter Koroshetz, Shai Silberberg, and Devon Crawford.

The admonishment about “harm” appears in the FOIA office’s final response letter along with a small cache of emails that is obviously incomplete. Its incompleteness will hopefully become clear shortly. To summarize, the emails cover a single topic, a request to carry over an unobligated balance due to slowness in initial hiring. Nonetheless, work was underway and the grantees were able to complete 4 out of 5 milestones. (All of the documents are linked below.)

The reason it’s notable that two of the top NIH officials don’t appear in the documents is that NIH promised “substantial programmatic involvement that is above and beyond the normal stewardship in awards” on Community for Rigor. They were to have an “ongoing role” including at least “monthly meetings” with grantees, an annual conference with Shai Silberberg as keynote speaker, “regular communication,” establishing a steering committee and “providing advice and guidance” throughout. According to the FOIA response, none of this “substantial programmatic involvement” generated a single email in a year and a half.

The documents show that communication was conducted by email, and specific messages were referenced but don’t appear.

According to Konrad Kording, the central grantee on the project, Shai Silberberg lobbied the institute for 20 years to get Community for Rigor funded. Silberberg has been working on its design for the last five. Yet the documents don’t include him other than to say he is a program director. This means that, according to the FOIA response, Silberberg never sent or received an email on the project he had spent two decades building, and on which he had a self-imposed “substantial” role. This is despite common sense that suggests it’s simply not so, and despite the fact that I sent him four emails to make sure my feedback on the project was FOIA-able.

My correspondence with NINDS director Walter Koroshetz and program officer Devon Crawford, published in June, also doesn’t appear.

In a separate, but related matter, there are no emails to or from any individual professors (grantees) even though NIH spent months requesting objections from them. The large number of grantees was cited as the reason my request was designated “complex” and would take extra time to complete.

NIH’s justification for omitting emails involving these professors in charge of producing educational materials was as follows:

“...we asked the grantee for advice concerning patent rights and other confidential commercial or financial information and the material that we are furnishing reflects that advice.”

This single grantee seems to have served as both subject of the request and subject matter expert, and asking their advice took over two months. This person also exempted the many others who presumably weren’t planning to patent their materials, as well as discussions that weren’t about future patents. This advice meant that emails were left out entirely, rather than partially redacted as NIH requires. NIH discourages applicants from including any proprietary information in their records, particularly without disclosure. If these submitters don’t heed this warning, “a significant substantive justification will be required to withhold the information if requested under FOIA.” So we don’t know what the professors discussed with NIH but we can speculate that it was always 1. substantial, and 2. financially volatile.

This is all to say that the FOIA response is very likely incomplete. Either NIH did not participate in the project they said they were participating in, or emails were withheld without justification. The professors’ justification sounds coincidental since financial gain (Exemption 4) was the excuse available.

The only other justification was the one above about “harm” and the “spirit of the FOIA.”

I don’t have to explain why harm is neither the spirit nor the reality of the Freedom of Information Act, nor why personal judgments and personal interest in harm are not good justification for ignoring it, or indeed laws in general. The law applies, according to NIH, “regardless of the intended use of the information.”

On improving the “drafting of grant applications,” NIH rightfully tries to prevent grantees from FOIA-ing their way to more grants.

More importantly, the wider concept of “harm” includes the more narrow “reputational harm,” which is the most plausible reason for NIH and grantees’ evasiveness. This vague language that the FOIA office used throughout always favored NIH and non-transparency. “Cooperation” in the “spirit” of the law with those who may deserve, potentially egregious, reputational harm is at least noxious if not wholly corrupt.

FOIA request

Foia Request

36.9KB ∙ PDF file

Download

Download

Correspondence with NIH FOIA office

Nih Ninds Foia Request 63454

97.1KB ∙ PDF file

Download

Download

Final Response To Ninds Foia #63454

45.6KB ∙ PDF file

Download

Download

Response document (email cache) and official letter

#63454 Final Response

20.5KB ∙ PDF file

Download

Download

Uc2ns128361 Correspondence Redacted

4.02MB ∙ PDF file

Download

Download

Addendum

As in the past, the subjects of this post are not “bad apples.” They’re responding to threats to their role in science that are often existential (e.g. publish-or-perish). This site is about the impossibility of self-governance in science. By that I mean that researchers share the same interests and will seek those interests over self-governance and self-correction. Community for Rigor is only the apotheosis of this idea of self-governance and, predictably, when they need to self-govern they simply don’t. Independent checks are necessary. All of this is no more complicated than that.

Responses

I requested and did not receive comments from any of the subjects of this post. I also emailed the FOIA office two weeks prior, notifying them that I would be writing on this subject in a few weeks and asking if there was a mistake in the FOIA response. They did not reply. This space will be updated with future comments and corrections.

• SARS emerged in 2002 from a wet market. This is a common argument for a zoonotic origin in 2019.

• There are over 30,000 wet markets in China. Individual markets are very safe.

• The market hypothesis rests on the safety of an individual market.

• It is wrong to confuse the safety of all 30,000 markets together with the safety of an individual market.

• The proximity of the lab and the market means we need to compare individual market safety to individual lab safety.

• We don’t know how safe individual labs are but we only need a rough guess for the comparison. One way to do this is to consider 30,000 coronavirus labs in China.

Keeping with simplicity, all of these points will be translated into decks of cards and other things we understand well.

A brief philosophical justification

It is a well-accepted truth in science that method is what matters. We cannot evaluate research by its result. Otherwise, we may come to any conclusion we like. Suppose we want to conclude that the Earth is flat. All we need to do is accept the results of those who claim it is flat and reject others. This principle doesn’t simply work because we strongly believe the world is round. It works because evidence is like reality. Ignoring it doesn’t make it go away.

So ignoring evidence is a sort of epistemic sin. You shouldn’t do it.

Of course, there is such a thing as bad methods and we are allowed to doubt findings on the basis of methods. The problem with doubting methods on and on, however, is we are faced with an infinite regress. The methods are background assumptions that rest on their own background assumptions going back as far as you like (Bayes, Duhem-Quine, Hume). The infinite regress of doubt is real, but we can’t retreat to it in order to win. It’s always there, and strategic retreat is like any rhetorical device that always works. That which proves anything proves nothing.

This retreat-or-ignore phenomenon is a good description of the COVID origin debate. The simple argument is ignored and the experts have “kicked the debate into the long grass” of complexity and methodological doubt. Experts claim it is complex and so we should trust the 20% chance estimate from experts instead of the roughly 64% estimate from non-experts. This is the assumption we’re going to question here. What if the answer is simpler than experts say?

The Lab Leak Hypothesis

An estimate accepted on both sides of the debate is that experts put the chance of a lab leak at 20%, on average. 20% is a large chance and it’s an epistemic sin to ignore it. It’s hard to imagine some conventional wisdom pre-pandemic that we should disregard future conspiracy theories that have a 20% chance of being true.

COVID origin may be simple. If it is, the common sense argument about proximity to a coronavirus lab may be sufficient to direct the debate. This argument can be translated into math and the math is simple and one-sided. The math is one-sided because there are many wet markets in China.

Each wet market is safe

We find this unintuitive in the West. Each wet market is very safe. Just like airplanes that do crash in aggregate, individual airplanes are safe. It takes the force of millions of flights to make a single crash. Using the force of all those flights together to convince you to be scared is something your brain does in flight but nonetheless you are safe. You’re only on one plane and there are many others.

It took the force of 30,000 wet markets in China to create one pandemic in 2002 (SARS).1 Even if we grant the 2019 pandemic to zoonosis, that’s one pandemic every 17 years for 30,000 markets. If there were only two markets in China and they led to a pandemic in 2002, and another pandemic emerged next to one of them 17 years later, it would be justifiable to say it came from the market.

Confusing individual and aggregate probabilities is obviously wrong. Roll 100 dice. One of them is almost always a 4. This is a rate of “almost every time.”

“Almost every time” is not the rate of 4s on a single die. Rolling a single die will only give a 4 every few throws.

Even if the statistics are not clear, you can’t get a 4 almost every time and a 5 almost every time, and a 6 and so on using one die. You can get all those numbers easily with 100 dice. Similarly, you can’t get mostly heads and mostly tails with one coin.

Some other obviously false statements that rely on the same confusion:

• Someone in the casino wins $100,000 every day. If I try this slot machine for a full day, I will likely win $100,000.

• One plane crashes every few years. Planes are unsafe.

• 1,000 people are throwing a dart at a giant dart board. Closest to the center wins. Player 5 has a good chance of winning because everyone gets one dart: Player 5 has the same chance as the winner!

All of these statements can be corrected by dividing by the number of things working together:

• If I try this slot machine for a full day, I will have about a 1/(the number of slot machines) chance of winning $100,000.

• One plane crashes every few years. This plane has a 1/(number of flights per year × number of years) chance of crashing.

• Player 5 has a 1 in 1,000 chance of winning.

The logic of proximity using decks of cards

The logic of proximity is intuitive and we use it all the time. If you turn your back to an elevator, then hear a ding, you don't assume it was an elevator five miles away.

Shuffling two decks together

Take two decks of cards with different colored backs. Put 3 jokers in the red deck, and 1 joker in the green deck. Shuffle them together. Then draw cards without looking at the back until you draw a joker. Which deck is more likely the source? Obviously, red.

Now suppose that there were a bunch of decks on the table that you didn’t choose from. Does it matter that there were other decks? No, because you didn’t choose from those decks.

This sounds too simple for something unintuitive but it bears repeating. You didn’t choose from those decks. You can’t hear an elevator five miles away.

Let’s put an even finer point on this. Use the cards to represent a map of China. Take 25 green decks, each with one joker, and arrange them around a table. Draw one card from each. You’ll draw a joker about every other round of draws from all decks. That’s the aggregate rate of pandemics from markets. It takes the force of all the decks together to make a rate that high.

Proximal decks

Now start with 25 green decks. Take a green deck (1 joker) and a red deck (3 jokers). Instead of shuffling them together, put them close to each other. Draw from them with your eyes closed such that you can’t tell which one you’re drawing from each time. Suppose you draw a joker. Which deck did it more likely come from? The red deck, same as before. Knowing the number of jokers in the two decks is sufficient for answering this question.

Adding an adversary

Studies have shown that adding an adversary makes unintuitive math problems easier. Adding an adversary frames the problem as “cheating detection.”

Suppose you’re playing a game in which your opponent gets a point when a joker is drawn from a green deck and you get a point when it comes from a red deck. Take the same scenario with the green decks representing markets, and one red deck close to a green deck.

Suppose you draw from the two decks with your eyes closed and get a joker. Your opponent snatches it and tosses it into the discard pile before you can look at the back. He says the joker probably came from a green deck and so he gets a point. When you draw from all the green decks at once, you get a joker every other time; if you draw from the one red deck, you hardly ever get a joker. So the joker probably came from green.

You say that you weren’t drawing from all the decks.

Nonetheless, he goes on. The rate is much higher using all his green decks together than it is from your one red deck. He has more jokers in total. You reply that it wouldn’t matter if he had a million decks and a million jokers, if in fact, all the physical universe were packed with green decks end to end. You didn’t draw from those decks.

Estimating individual lab safety

We don’t know how fast labs produce pandemic-level viruses. A pandemic-level virus has never come from a lab. This could mean that pandemic-level viruses never come from labs, or the rate could be 1 pandemic every 100 years. As with other background assumptions, we can’t choose the estimate based on personal stakes.

We also can’t say it’s likely zero because it’s never happened. Experts on both sides of the debate agree that it is technically possible that COVID came from the lab.

We can guess, though, because we don’t need to know exactly how safe individual labs are. We only need to know how they compare to individual markets. One simple way to do this is to imagine 30,000 labs in China.

Does 30,000 labs start to approach the danger of 30,000 markets? 30,000 is a lot of labs. But even then, equal comfort with 30,000 labs only means you think the lab and the market are equally likely to be the source. In order to ignore the lab, you would need to feel safe despite the existence of many more than 30,000 labs. And you would need many many more than that to get to dismissable conspiracy theory territory.

So the belief that lab leak is a dismissable conspiracy theory requires the belief that many tens of thousands of coronavirus labs the size of the Wuhan Institute of Virology in China are equally dangerous to the current status quo. To put this in perspective, building 3,000 WIVs would only make us about 10% more worried about pandemics coming out of China.

One easy rebuttal to the many markets argument is that some human activities that seem unsafe are safe like flying in a plane. Having many labs in China could be like having many airports. This is true. Labs could be something we’re able to do safely. But consider how insensitive we have to be to not figure out if it is like airplanes. It means we’re okay with one lab and about equally okay with a lab every few miles.

Conclusion

Aggregate probabilities can be unintuitive. We’ve probably all heard casino schemes like the one above, and everyone has had doubts flying.

The safety of wet markets can also be unintuitive.

Combined — and with the emphatic assurances of many experts — COVID origin is more fraught than debates about drawing cards or getting on an airplane, but the naive probabilities are the same. Even if you don’t agree that they point towards a lab origin, they at least demand a better estimate of the missing quantity: individual lab safety. Even very low (safe) probability points at lab origin.

The assumption the many markets argument rests on is that COVID origin is simple. It may not be. It is worth considering that COVID origin is simple, though, because the stakes are high and because simple arguments are easy to understand. It at least won’t take long.

This post is now watchable, although my public speaking skills may not do the topic justice. There’s a friendly debate with president of SIPS, Priya Silverstein afterwards that may be interesting to those deep enough in open science lore to appreciate the undercurrents. She says she’s more optimistic about adversarial collaboration as opposed to adversaries. I said I hope she’s right.

SIPS was more than charitable giving me a spot and Dr. Silverstein was constructively critical. Obviously, I am not optimistic and that’s the central disagreement highlighted in the post and on this blog.

The undercurrents, and the SIPS pre-conference discussion with early replication crisis reformers may be worth a follow up at some point. For now, I can recommend the pre-conference discussion about what worked ten years in, and the metascience reading list as great bookends on the replication crisis.

COPE is back in the news. They are promising an update this month that would recommend giving credit to people who point out retractable errors in papers. Not giving credit to people reporting errors is one of the most indefensible policies in publishing.

After I posted this response, I emailed the authors again because they had sent me an email asking what remaining questions I had. The matter seemed settled to them: subtracting effects is standard practice (it’s not) and the numbers sometimes don’t look anything like subtraction because they’re meta-analyzed.

A great rule of thumb for debate in general is “that which proves anything proves nothing.” Saying “just run the code,” or that something technically could be true because anything is possible in statistics are not good arguments. In this case, I publicly stated that I’m wrong if they can provide a handful of numbers that produce two rows of the table and they have declined.

(Regardless, I did run the code and could not find any numbers that result in more than 1-2% difference from subtraction. This is mostly apparent just looking at the code, or reading the paper where they include an equation.)

The Lancet subtraction question is also notable because it is the most obvious error with the lowest reputational stakes and highest public health stakes (by which I mean vaccine effectiveness is important). The first author went on to another career outside science. The apparent errors are in the supplementary material and wouldn’t require retraction on their own.

Nonetheless, I did not get a response. I suspect the email asking for further questions was one author trying to distance themselves without explicitly disagreeing with the co-authors. I’m not going to indulge this possibility because I’d been asking for over a year for any statements by the journal and the authors to be made publicly, not by email, and I still am.

COPE recommends using the same platform where the error was reported (PubPeer).

One of the major topics in these posts is that Community for Rigor claims to take community feedback publicly, and then really resists it behind the scenes. They said this was because “we cannot guarantee that all users’ ideas will be reflected in our materials. That would unfortunately be impossible to implement” and that hundreds of people had signed the wavier to give feedback, contrary to my objections to the wavier.

The update is that I wanted to know if they have now taken any feedback from these hundreds of people and they did. The slides for the confirmation bias lecture have been updated to include some of my feedback:

• Correct attribution of “researcher degrees of freedom” to Simmons et al. (2011)

• Addition of pre-analysis plans with preregistration

• Addition of more post hoc choices

• “Employing peer review,” which I said usually means journal peer review instead of adversarial collaboration or red teaming, in other words things researchers have a choice to adopt, has been removed.

• An activity now doesn’t have implausible Cohen’s d values, and are labeled in standard deviations.

• The possibility of unmasking even if treatment has no beneficial effect is now included.

Additionally, the materials mention the term p-hacking now. I didn’t recommend this, but I should have.

My expectation was that no changes would be made, certainly not ones I’d suggested, so that’s great. My thesis about Community for Rigor is that they shy away from confrontation with their colleagues. I think that’s been tried in metascience and it didn’t really work. (Of course, C4R deserves credit for doing anything at all, which is great too.)

Update 2: FOIA

My long-awaited and long-haggled-over FOIA request for Community for Rigor emails has been promised early this week. The same thesis about conflict applies: I suspect NIH/NINDS and the administrators of the project at University of Pennsylvania knew that some grantees were avoiding conflict with their colleagues by saying the practice of science is great already. No reform necessary. In general, I think all of the participants will tend to avoid conflict and say the practice of science is better than it is.

Concrete examples were given in the original post: Mary Harrington depicting the research life-cycle as a highway one can get on and off at any time, and where you can collect and analyze data and then change your hypothesis. Benedict Kolber’s group saying “most published research is not truly confirmatory (and that’s ok)” (which was amended in the materials at Penn by adding “However, exploratory work is sometimes reported as confirmatory. (and that’s NOT okay!!)"). The guest speaker saying preregistration is not a good idea and everyone nodding along.

My hypothesis about the FOIA documents is that the limiting factor in metascience is not cognitive (See, for example, Wiradhany et al., 2025). It is conflict of interest. Scientists and metascientists don’t want to get on the wrong side of mainstream science, which outnumbers metascience by many multiples. In other words, scientists working in academia and even NIH officials are not appropriate adversaries.

I will be shocked if this doesn’t show up in the documents and I’ll admit I was wrong about C4R if it doesn’t. Obviously, if I see a lot of conflict — particularly productive conflict — that counts against my hypothesis.

As mentioned in a previous post, I had not realized when I posted about the fragile p-value paper that the author had said essentially the same thing on Bluesky, that the effect could be more p-hacking.

There was an otherwise great post in Sensible Medicine that used Holden Thorp’s op-ed to support the idea that “comparatively small numbers of scientists... can drive negative public narratives about science.” I would recommend the companion paper too. My criticism of the authors is they tiptoe around the behaviors that plausibly caused the replication crisis, and that many many researchers engage in. That rebuttal will be published soon.

I emailed Andrew Gelman redteamofscience.com and he provided a counterpoint on Jay Bhattacharya’s dedication to debate that I had published in this post. Andrew says that Jay misrepresented his views (see Gelman, and Bhattacharya). The exchange is accessible and interesting, and I agree with Andrew. “[Gelman] incorrectly thought we had not accounted for the possibility of false positives” is not a good way to characterize what happened. Andrew’s criticisms of the Santa Clara paper has everything you’d want in a post-publication review: expertise, humility, attention to detail, and a few barbs. The CI and specificity arguments are sound, and it’s notable that paper authors go to great lengths to avoid crossing zero.

That said, I still think the political platform about restoring debate in science is valuable enough to support Jay’s whole shtick. I’d support a liberal NIH director with the same principles. That’s not to say Andrew’s objection doesn’t “move the needle” on this support at all. It does. It’s a ding on Jay's reputation, no more or less than what I think was intended.

In an unsurprisingly unpopular post about how the place you all work is bad, I also decided to discuss COVID origin. Andrew Gelman happened to post about it too, which is extremely helpful, and the reason I emailed him above.

We can’t take things like the lab leak with a reasonable but small chance of being true and say, “Well, then they’re not true.” One anonymous commenter on Gelman’s blog put it this way, using the same statistic I used:

Some scientists (20% of experts) lean towards a lab origin. It’s a number higher than the percentage of experts who lean towards creationism or climate denial, so therefore we can certainly agree the truth is less certain than on those two issues. However, after reviewing the evidence I lean towards >99.9% zoonosis.

I think we need to stop this, and carving out extreme positions is very clearly just caused by the internet.

I’m going to dedicate a post to a simple statistical argument on COVID origin in the coming weeks. (This is a good time to say that subscribing anonymously is easy with an RSS reader and there are lots of good metascience blogs available this way. The feed is: redteamofscience.com/feed)

The thesis of this series is that science and the culture war using it as a political football are trapped in the politics of embarrassment. This is anti-Lakatosian because we stake out more and more extreme positions instead of risking embarrassment by admitting the future event that would prove us wrong. Predictive ability is a demarcation between science and pseudoscience that will expose the extremes instead of feeding them. Lakatos, as a victim of Nazis and a Marxist expelled by Marxism, knew these consequences better than we do.

This essay is full of arguments a fortiori. For instance, if the pinnacle of rigorous science, metascience, can’t get its shit together, then neither can normal science. If the pinnacles of anti-culture war rhetoric and universal values like Rauch and Harris defend universities without end, then who can get them to reform?

As before with Brian Nosek, I don’t want to single out people I otherwise respect for not deferring to a philosopher few have heard of. It’s one of several faults this reasoning has. However, American life is full of accepting the leadership of people who’ve achieved these pinnacles like Rauch and Harris have. Throwing them some hard questions can’t hurt. They are both symbolic, and actual representatives.

I sent a version of this post to Quillette as a rebuttal, and emailed Sam my objection at the time. On Wednesday, I sent a copy of this updated post to both for comment and received no reply. As always, the offer stands after it is published.

The essay “Universities are Worth Saving” by Jonathan Rauch is brilliant and nonetheless wrong in its conclusion. Universities are certainly worth having, but their current incarnation as irreproducible science factories is not worth saving. We have had institutions too big to fail in the past, namely banks. We require both to follow rules that tether them to reality, and we should agree that universities are not an institution we should protect by definition. There’s a certain point where they stop being the thing they claim to be.

In Rauch’s view, universities are a core part of the reality-based community because they follow rules. But they don’t. They hacked those rules long ago.

For twenty years, academics have been supporting poor methodology with full knowledge of the replication crisis they caused, and have done very little about it. The depths of this crisis have still only been plumbed in a few fields, and it is not taught in schools. If rules are how universities stay in the reality-based community, they’re out. It’s as simple as that.

Rauch explores replication only to underscore the kind of reality we’re talking about: “The rules are impersonal: Persons are interchangeable, so anyone can vote and gets the same vote. Applying this idea to science: anyone can replicate an experiment, and it better come out the same way. Anything that anyone can do, anyone else can do. No one in particular is in charge.”

This is true, but denying the implementation of these rules is insulting to reality and to ordinary people. In fact, the true reality-based community may be better described as the large swath of humanity that depends on reality and lacks the power to change it at will. It would be more charitable to reserve “reality-based community” for the people who will cease to be if they don’t stay tethered to reality instead of the community that gets to choose when to follow the rules based on — let’s face it — government grants.

The replication crisis

In 2005, John Ioannidis famously conjectured that half of published research is false. This is a complex story, but he was repeatedly found to be prescient. In 2011, a paper showed that “the rules” can easily be hacked. These are only two of the most famous papers exposing more or less the same flaw – which can be summed up as “flexibility.” There are hundreds.

Those who have read the metascience literature that followed the replication crisis will be hard-pressed to say that universities acted in good faith or acted very much at all except to accelerate publication. Publication wins grants and rungs on the university rankings. Never mind sick people or society, they said. We have U.S. News and World Report to worry about.

Each year that goes by makes it more convincing that universities are willfully maintaining their place in the reality-based community at the expense of reality. As results came out in the early 2010s suggesting science is not self-correcting and it can be hacked, academia plugged its ears. Worse, the early papers were, through no fault of their own, also a how-to guide to academic success through p-hacking. Furthermore, academics now know how many others are hacking too, and how little we can do about it.

In other words, academia’s Natural Selection of Bad Science was once selecting for the trait “p-hacker,” and now it’s selecting for the trait “remorseless p-hacker.”

Friends of p-hackers

P-hacking, or running analyses and checking the results to see which one you like, requires computers and those computers are getting faster, cheaper, and better at p-hacking. This bending of the rules wasn’t discovered in 2011 and slowly fixed over the next decade as one would expect. It has arguably accelerated. Two recent incidents in p-hacking give us a peek at the character of academia, and maybe a little on the moral hazard of defending an institution without end.

1. We now entertain the ludicrous conjecture that researchers can force their p-values below 0.05 to get published, but other round numbers like 0.01 are safe from meddling. This (slight) p-value migration was taken as a good sign in both ordinary science circles, and in science reform. The reasoning went that more p-values in these carefully-chosen buckets was a sign that academia is good, actually, and reform worked. Instead, it's a phenomenon even the author admits could be just more p-hacking:

   

2. At a conference on scientific reform called META-REP, which was otherwise excellent, a workshop organizer slipped up and suggested that we could use a software library to select which analyses to run (p-hacking) as opposed to showing all analyses, something done for the sake of transparency. When I asked what the difference is, then, between this workshop and extreme p-hacking, the organizers’ response was, “Well, we hope people wouldn’t do that.” They would and they will.

So it’s important to know which rules can, and are, being broken. A little bit of flexibility is expected, but that flexibility can’t simply be used thusly: “take any number that represents academia as a whole and cross it out. Write another number that’s better.”

In banking terms, the rule-breaking that caused the financial crisis was obscure like the mass measurement of p-values above. The more obvious rules (don’t take money from the vault) were followed and the more obscure ones weren’t.

There’s no need to trust anecdotal evidence. There are heaps of studies on this. A 2025 paper found that knowing about poor methods is only weakly correlated with not participating in them, and, to the surprise of the authors, 30% of respondents “endorse p-hacking.”

One of the early reformers, Leif Nelson who, along with his co-authors, coined the term p-hacking back in 2014 has gone further, saying, “Everyone p-hacks.” His fellow reformer, Simine Vazire, argued a few months ago that it isn't good for your career to hold reform views by which she meant, in part, not to p-hack. “We’re bleeding early career researchers” she said, and we need to protect them.

The “everyone p-hacks and it’s dangerous to tell them not to” environment is one that people working on this issue know well, and the literature backs them up time and time again. And, despite everything we thought about the pinnacle of the reality-based community and their commitment to peer-reviewed literature, the literature has been ignored.

Right vs. Left

The fight for and against universities is often couched as political, right versus left. Consider a radical hypothesis to the contrary. What if both political parties are willing to control a sort of “knowledge machine” like the universities? What if the fight for universities isn’t inherently left versus right, it only appears that way because the left has them now and the right wants them? Churches were once influential knowledge machines and they were politically potent too. The right predominantly occupied them and protected their territory.

Right vs. left, and reality-based community vs. non-reality-based community are not fitting dichotomies in the modern university crisis because they don’t encapsulate the fact that academia doesn’t always follow its own rules. A better one might be common sense versus academic sense. Common sense is much maligned but it is at least held by people who can’t ignore reality to the same degree academics can. Academic sense is held by people who have dedicated their lives to knowledge production but are also in a position to break a few rules.

Sam Harris defended intellectual authority around the same time Rauch did. This was also the period that science.org was using phrases like “god help us.” He criticized the intellectual chaos on the right and appealed to the “vast reservoirs of integrity in academia.” I agree with this as a broad-brush statement. However, it underestimates the power of academia being able to break its own rules. This is not simply the “smart” versus “freaks.” Like Rauch, he seems to deny that anyone is in charge in this area of the world.

On whether or not ordinary people are “smart,” let me reiterate that ordinary people generally don’t have a choice of being tethered to reality. As Sam says, some aren’t, and some academics aren’t either. But there are “vast reservoirs of integrity” — and what some might call smartness — among laypeople too.

As an extraordinary example of common versus academic sense, in the United States, most non-academics believe COVID leaked from a lab, and this is a surprisingly bipartisan view. Virologists and epidemiologists themselves put it at about a 20% chance on average. It seems there are issues that the uneducated simply can’t accept no matter how vehemently the educated tell them they’re wrong.

However likely you think the lab leak is, even if it’s less than polling virologists implies, it’s too close for comfort. Suppose there’s a lesser, 15% chance that COVID came from a lab. If COVID did come from a lab, it means things with the academy are much worse than we thought: Normal people were more correct than scientists about a substantive matter. A new contender might be emerging to contain the academy that conservatives couldn’t: normies.

COVID fits neatly into the recent Stand Up for Science petition covered in part II, which uses millions of lives lost at the extreme ends of the political spectrum from Lysenkoism to Nazism as a demarcation of where things have gone certainly wrong. True-to-form, however, the academic petitioners get to ignore the fact that we may have just experienced another great loss of life, and the villain may have been science in both its production and cover-up. What irony then that even this incredibly rough demarcation — which is less a demarcation and more a way of figuring things out after millions of people have died — is still wrong.

In another telling poll, Rauch points out that the number of people who think universities are negative for society has jumped 19 points. 45% of people think universities are net negative. (A more recent poll from New America asked a similar “positive or negative effect” question and got an equivalent 44%.) This jump happened in the same period as the unmentioned replication crisis.

Rauch then concludes that these normies are wrong. They could be right. Even if we assign a small chance to them being right, the implications are profound. It means the height of the reality-based community that is in control of humans’ greatest invention, the scientific method, might be on balance bad for society. This could be worth looking into before another 20 years goes by.

The power of selective reporting

Rauch and Harris didn’t mention the replication crisis, the “moon landing” of discontent with intellectual authority. This is called selective reporting. It is the supreme breaking of the rules.

As Rauch puts it, universities have rules. Banks have rules too. Rules matter because, if you cross out “vault” and write “piggy bank,” you can take all the money you want. There’s no bigger intellectual piggy bank than selective reporting. It is a sin to do it, and yet it encompasses p-hacking, publication bias, and ubiquitously ignoring the metascience literature.

To be fair, these debates are full of cherry picking. Articles about the attack on science regularly, relentlessly cite vaccine science, climate change, and flat earth instead of social psychology, cancer biology, economics, Alzheimer’s, neuroimaging and the many hundreds of scandals in knowledge production today. Journalists should subscribe to Retraction Watch and browse PubPeer.

Harris and Rauch broke the rules. Academics break the rules. So when someone without all the privileges a good education confers says that academics are all a bunch of self-publishing p-hackers, I say let them say it and don’t interrupt. If academia wanted to take care of their reputation, they should have sorted this out many years ago. Particularly, people left out of the reality-based community won’t be charitable.

Conclusion

Reform of intellectual authority is a hard problem. Science is broken at exactly the places it can stay broken. P-hacking is done in secret. The “base rates” pointed out in 2005 even doctors don’t understand. And publication bias is a domain largely controlled by private publishing companies. But it must be faced, and it is one of the reasons the right believes academics are very frequently wrong. They are frequently wrong.

So the question for Rauch and Harris is, “At what point would you agree to ‘burn the universities down’ metaphorically speaking?” Suppose the replication rate of academic science goes to 25%, or 10%. Suppose the universities thwart movement after movement, on the left and right? Suppose the ability to generate knowledge with a hacked machine is too powerful. Is there a metric, and a point on that metric at which you will say your views have changed?

To avoid hypocrisy, I’ll do this too. Stuart Buck estimated the replication rate should be around 80-90%. The rate is not only difficult to estimate, but in general we don’t measure the thing we are after. We measure p-values, not probability of hypotheses. Despite this, I concur the rate can’t be expected to go to 100%, and 80 is not a bad guess. 80% replication rate is about where I would join team “defend.” I joined team “criticize” at 50%. Recently there were hints of 26%.

In short, section 3 of the executive order represents, in his own words, Brian’s exact principles. The north star of Brian’s Center for Open Science has always been included government intervention with “make it required” at the top of the oft-repeated pyramid of change. Yet something in this relatively milquetoast executive order has led to this petition comparing it to Nazism and Lysenkoism. It’s a pattern reflected in open science’s rejection, or at least mute response to other news in the field: billions of dollars in funding, caps on publication fees and overhead costs, ending the embargo period, and the general call for debate. [Brian correctly points out there’s further response to the order at cos.io.] All of these policies claim to stem from metascience’s core principles and yet metascience rejects them somewhere in the implementation of “make it required.”

I’m trying to stay away from how I would want to “make it required,” and I admit I don’t think the right is objecting to incontrovertible statistics. They just don’t like that science is so liberal, which is another argument about outcomes and not methods. However, this executive order does get into methods and it’s important to know why that too is being rejected.

Like the essay that this series is based on, “Science and Pseudoscience” by Imre Lakatos, the petitioners invoked Lysenkoism on the left. They invoke Nazis on the right. Lakatos was both a victim of the Nazis, and expelled by Moscow in Lysenko’s era. Unlike Lakatos’ call for rationality in the face of these threats to society, the petition calls for ascendance of political truth, and it is a petition. It gets its power from the same place Trump did. A lot of people were willing to sign on.

From the petition:

At face value, this order outlines a supposed commitment to federally funded research that is 'transparent, rigorous, and impactful' and declares that policy is to be informed by 'the most credible, reliable, and impartial scientific evidence available.' However the EO as written will actually undermine scientific rigor and the transparent progress of science. As scientists, we are committed to a discipline that is decentralized and self-scrutinizing.

Anyone who has sat through hundreds of lectures, paper introductions and discussions, policies and other word salad exercises on metascience and open science will recognize these words. The petitioners have simply had their word salad stolen. It wasn't specific or meaningful when they said it either.

When these words were spoken before, truly ad nauseam, did they hide a secret liberal agenda to centralize science behind a sheen of metascientific validity? I’ve never thought so. I’ve considered the liberal bent of metascience to be a byproduct of trying to build a coalition within science that is only 6% conservative. This petition invites opponents of established science, metascience, and open science to deny this word salad — essentially a list of words we associate with truth — has any face value. This might be for the best, but it doesn’t bode well for finding common ground.

“As scientists,” it says, “we are committed to a discipline that is decentralized and self-scrutinizing.” Brian Nosek, his Center for Open Science, and some metascience groups have been planning a centralized metascience organization for years. It was formally announced only one month later.

On self-scrutinizing, Brian deserves some credit. He has invited his critics to speak at events and engaged with them. However, it remains to be seen how responsive he will be to the complaint that metascience and open science is too liberal. I've never seen this objection raised or hypothetically entertained in this community.

There is a non-political way of putting this type of objection without partisanship, namely the framework of progressive and degenerating research programs. However, for the sake of argument let’s put this self-scrutiny to the test. Would Brian and the open science community entertain the scrutiny that it is simply too liberal? I suspect they have not, and will not.

Suppose that, as it probably will, the Trump administration uses these universal principles in its declarations, but then won’t entertain one and only one objection, that it's too conservative.

How absolutely laughable would this be? How silly would the Trump administration look showing its cards like this? That is precisely how, to the degree and for the same reasons, the Center for Open Science, FORRT, and ReproducibiliTea look to conservatives.

Never forget what they've done

Conservative media and social media is filled with lists of links as the petitioners have listed, “look what they did here, here, and here” or “never forget what they did.” The fact that the petitioners put together such a list is not surprising. I even agree with much of it. Canceling grants because they’re too liberal is not Lakatosian.

All I will say in the conservative defense is imagine that science was dominated by religious people. I've taken the conservative complaint about science and replaced “liberal” with “religious”:

Imagine a world in which, for fifty years, non-religious people had been complaining that behind all the supposedly rigorous, math-based results of science was a hidden, religious agenda. Non-religious results were filtered out, either by the researchers themselves or journals, or non-religious results were protested so vigorously by religious people and so many editors thrown out that in the end the scientific literature is, as a whole, simply too religious. It’s hard to say exactly how much, and particularly because religious people get to point to all of the religious results that have already been found as scientific support. Many people excuse the whole thing by saying, “well, reality has a religious bias.”

It may come to pass, then, that once the non-religious manage to vote in a non-religious president willing to take on this issue, this president may have difficulty sorting through the millions of scientific articles and the billions in scientific grants to find exactly where religiousness has had a major effect.

Religious people may then complain that grants are being canceled just because they have the word “faith” in them. They may point to all the good work done for AIDS, homelessness, and orphans by religious organizations. Would you want all this to stop?

Find-and-replace arguments have never been effective on partisans. They were not effective for the (literally) religious people in the thousands of years of religious history when one group with devoutly religious people would encounter another group from another part of the world. Why does it matter how strongly you believe these things? As Lakatos put it, “the history of thought shows us that many people were totally committed to absurd beliefs. If the strengths of beliefs were a hallmark of knowledge, we should have to rank some tales about demons, angels, devils, and of heaven and hell as knowledge.”

Nonetheless, this is an argument that the list of complaints about canceled grants in the petition has some merit. However, the list is mostly just maximally offensive to liberals and that is why it was chosen. Liberals may not realize how truly and deeply offended conservatives are by other lists.

These lists of grievances are faulty demarcations. Some of metascience has decided to draw the line of demarcation at political appointees. Some like to list climate and vaccine science. Very few science authors will choose equivalents on the right, and very few will admit that we've found all kinds of evidence that science is — broadly — half wrong.

Some authors go straight for the jugular and invoke childhood leukemia. We don’t need philosophy of science to suggest these topics. A PR firm could have come up with them.

They are not demarcations. The demarcation between science and pseudoscience is not drawn around RFK’s latest quotes, or even around topics like vaccine skepticism. It is not drawn at political appointees. These are demarcations of political embarrassment. They are chosen so no one will disagree.

The sex binary

This is a claim made strongly enough in the petition to refute on universal grounds: “the order comes from an administration that... incorrectly defined sex determination as binary, when biology proves it is not.”

Biology doesn’t prove anything.

Scientists in normal practice claiming to have proven something are laughed out of rooms, and denied publication. There are reams of philosophy of science and normal science to support this: Hume, Duhem-Quine, Bayes. Lakatos put the history of science thusly, that to counter the absolute facts of the Catholic Church, science was originally absolute in its statements. This has not only gone away, but the most famous early statements of proof were overturned by Einstein.

There may be more entertaining ways of demonstrating that scientists don't generally talk about proving anything other than to point at an infinite regress. Do these 10,000 petitioners and their many thousands of papers ever say they “proved” anything? There could be an odd case. The authors of the petition argued against their own words, announcing it in The Guardian, saying “Science does not proceed by sequentially establishing unassailable conclusions.” Even climate change and vaccine safety they describe as “robust, valid conclusions,” not proven.

The particular claim of the sex binary was recently taken up by the Freedom From Religion Foundation (FFRF). Having published a blog post about how sex is a spectrum, several distinguished board members resigned and posted a scathing critique, which not only disputes the sex spectrum, its complaints about FFRF mirror this complaint about metascience institutions.

The scientific consensus on the sex binary was hinted at when 58% of UK researchers said they believe in the sex binary in 2024. This is not to say that scientific consensus is right, only that this proof doesn't seem to have convinced many people. When, how, and by whom the proof occurred is not answered in the petition.

A synopsis of the FFRF incident was written up by the “Friendly Atheist” in which he calls the resignation of Pinker, Coyne, and Dawkins “the trash taking itself out” and suggests the atheist community cut ties with FFRF. If you are part of the 58% and you can read the Friendly Atheist’s post without fearing that you and your organization would be subject to global online humiliation as a result of your views, you may have transcended the politics of embarrassment.

The organizer of the petition, Colette Delawalla, regularly uses the sex binary and published a preprint saying, “The sample represented all 50 states, approximately half were female (50.8%), and 58% were white, 21% Black, 15% Hispanic, and 5% Asian” a week before the petition.1 The fact that the coauthors, a Nobel laureate, the head of the metascience alliance, and a man who wrote “Calling Bullshit” didn't object to the “proof” wording illustrates science in 2025. Politics is in charge. (The coauthors also use the sex binary, in some cases on LGBTQ issues and on gender disparities.)

This is all indecisive rhetoric, though. Liberal and conservative science can go around in circles making claims like this without anything being overturned. Showing something is “too liberal” just as it is to show something is “too conservative” is not enough for partisans. Something needs to be shown to not be true, at least to the best of our abilities in a universe without proof.

The Lakatosian argument against the sex spectrum is first to ask adherents, “at what point would you give up on the idea of a sex spectrum?” If there is no evidence that would convince you that sex is not a spectrum, then there's no evidence that has convinced you that it is a spectrum. This is true in a Popperian sense that the demarcation between science and pseudoscience is this exact point, that your beliefs can be falsified. And it is true in a trivial way that all the evidence used to support the spectrum may have been due to some kind of error, of which there are many in science.

Lakatos pointed out that saying “never” is primarily done for ethical reasons. A “good” and moral sex spectrum adherent is not going to go on the record saying that they could start to believe in the sex binary.

Another response would be to ask what the sex spectrum predicts. Is there some state of nature that could be discovered that would contradict it? Almost by definition, no. No species could be discovered, no result found that the sex spectrum could have made a novel prediction about. Any prediction from the large and small ova camp can be accommodated automatically by simply pointing out some remaining uncertainty. The bar is quite low for a spectrum argument. Spectrum-ites don't even need to place a phenomenon on the spectrum, which could be falsified by results that show the thing is actually at a different spot on the spectrum.

The large and small ova camp can predict that a new species will produce large nourishing ova, and small ova meant only to temporarily carry genetic material. This could be falsified by a species in which the nourishment function of the ova is shared to such a degree that the dichotomy is unclear, or a species whose ova nourishment comes mutually from the environment. I don't know if such a species exists, and with apologies to Dawkins and Coyne if I am fumbling the argument, I'm willing to lose a few points for the team. I can't write a series about embarrassment without risking any of my own.

The fallaciousness of the spectrum argument can be demonstrated by omission. If it had been proven, the petitioners could have named even one other sex that has been “proven” to exist. The rhetorical answer to this is that there isn't another nameable sex. Different sex determinations all go into an indeterminate goo about which we cannot make further claims.

This is the argument that the sex spectrum research program does not progress. It doesn't make novel predictions. It has a protective belt that protects by definition. It's not falsifiable. And its adherents won't admit what would work against it.

Even if this is not convincing, and it probably won't be, (Dawkins and Coyne still disagree about exactly where intersexuality fits into the debate) I have a lower bar in mind. The far left is not a scientific antidote to the far right. The deference to nature is gone from this petition because the far left has no intention of sharing power with her, now or ever. The answer is incontrovertible principles of thought. General enough to agree on and certain enough to steer us away from both Lysenko and Nazis.

Scientific misconduct

The petition says, “direct presidential appointees are given broad latitude to designate many common and important scientific activities as scientific misconduct.”

The executive order defines scientific misconduct:

“Scientific misconduct” means fabrication, falsification, or plagiarism in proposing, performing, reviewing, or reporting the results of scientific research, but does not include honest error or differences of opinion.

For the purposes of this definition;

(i) “fabrication” is making up data or results and recording or reporting them;

(ii) “falsification” is manipulating research materials, equipment, or processes, or changing or omitting data or results such that the research is not accurately represented in the research record; and

(iii) “plagiarism” is the appropriation of another person’s ideas, processes, results, or words without giving appropriate credit.

In a word, this is correct. These are the correct definitions of these words in terms of their usage, and correct in a moral sense that these are good lines to draw between types of behavior. (FORRT defines these terms similarly in its code of conduct, which has laudable goals but is much more a liberal document ripe for abuse than the EO is a conservative one.)

Researchers should not be engaging in these behaviors as “common and important scientific activities,” not only should they be penalized as recipients of federal grants, they are already penalized for these behaviors. So the order's treatment of scientific misconduct is a particularly poor way to criticize it.

Executive orders

Most of this post is concerned with how poorly the metascience and open science movement's approach to politics is supported by any philosophical underpinning other than a partisan one. I suspect that, to compensate for researchers’ general loss of scientific status that needing to be more rigorous imposes, the metascience movement is offering researchers political power. That, or metascience is taking a trojan horse approach that mirrors its fawning assumptions of good faith.

For that purpose, the reaction to this order has been illustrative. Executive orders are, however, a weaker execution of democracy than laws passed through congress. Although I think the executive order is much more banal than its critics claim, the criticism that this is an executive order at all lands. Both the arguments that something like this should go through congress, and the argument that government shouldn’t be so specific even with replication, which is difficult to measure and never gives a binary answer, have merit.

If “make it required” was known to be something that could be hijacked to cost humanity millions of lives or a new holocaust, the Center for Open Science could have said, at least, that this doesn’t mean state dictated scientific truths in the way the petition uses this term.

“State dictated scientific truths” is, of course, too broad to encompass the fact that the state dictates that court cases should be decided on evidence, and randomized controlled trials should be based on sound scientific principles like preregistration, and the extremes of Lysenkoism and Nazism.

Some specifics are necessary. That the executive order was written by a party you don't agree with or even a person you don't agree with, or are suspicious of, doesn't count. Keep in mind suspicion goes both ways. We need to bridge suspicion on principles.

Conclusion

I’ve framed this debate around Lakatosian philosophy of science, and his essay “Science and Pseudoscience,” and the anti-Lakatosian phenomenon risk of global online infamy. There are many framings that would work. Furthermore, evaluating political movements and scientific programs once you know the progressing/degenerating framework is not particularly hard. Evaluating whether or not something is affected by the omnipresent internet embarrassment is not hard either. Many so-called discoveries and advancements like this blog post in this era of self-censorship may not be true leaps forward because everyone already knows them. They just can’t say anything.

Nonetheless, these things must be said. It’s better to do them with as much epistemic support as possible. Maybe this is not useful for convincing the individual. It is for giving the individual cover against the online mob of either political tribe.

One limitation to this series is that I may have the arrow of causation in the wrong direction. It could be that both the executive order, and the whole of metascience were already Lakatosian because he helped inspire them. So what we’re seeing is not failure to listen to his principles but failure of human beings to follow through with the personal sacrifices they entail. It suggests that maybe as individuals we need to start bearing those sacrifices without knowing that an end point is in sight.

Brian was able to respond briefly given the time frame. I am used to giving long response periods with no response, so I appreciate this very much. Future amendments will go here, and I will link to statements from any of the above organizations on the topic of bridging the political divide.

Response from Brian Nosek

• “The north star of Brian’s Center for Open Science has always been government intervention with “make it required” at the top of the oft-repeated pyramid of change.” This is false. The pyramid is a behavior change model. That is, it is a general model for how to change any behavior. It suggests that gaining 100% adoption of a behavior ultimately necessitates the behavior being required, ideally with the other elements in place so that people can do the behavior effectively. The pyramid does not itself indicate: which behavior(s), whether any given behavior *should* be ultimately be required, and *who* should do the requiring. We use the behavior change model to guide how we organize our behavior change efforts. Importantly, there are many reasons to stop and even reverse course in the behavior change model depending on the evidence of effectiveness for the behavior. Some behaviors might be tried and shown to be ineffective; others might work in some contexts and not others; some might be too costly to implement even if they would be useful. COS uses an iterative approach of testing and engaging communities on behavior change to determine whether and how the behaviors can be effective in their context. As evidence of effectiveness accumulates, so do efforts to move along the pyramid. That process is dynamic and iterative. You can see an example of how we specify that process in this grant proposal for testing the value of preregistration in the biological sciences, https://osf.io/r3txm (see Figure 1 for an overview).

• You ask whether science or metascience is too liberal. My personal view is that I have benefited greatly in my research from collaboration and exchange with people that have different points of view. For example, some of my most exciting and productive collaborations have been with a close colleague, Jon Haidt (e.g., https://www.projectimplicit.net/nosek/papers/GHN2009.pdf). He and I differ on assumptions and perspective on many topics, but I loved collaborating with him because we were constantly testing each other's assumptions. So, I support a scholarly environment that is rich with competing perspectives that engage with each other in a shared commitment to knowledge production. That perspective, however, is not the same as the current political environment. The current US administration — which I perceive as neither conservative nor Republican based on how I understand what that party used to be — is not productively engaging or supporting free and open scholarly inquiry. And, on the Executive Order for gold-standard science in particular, it is co-opting terms while embedding implementation directives that are counterproductive to those aims. COS's comment about it is here: https://www.cos.io/about/news/cos-statement-on-restoring-gold-standard-science-executive-order. I will continue to engage with the administration as federal support for science is critically important. I hope that the orientation shifts toward being a productive partner to ensuring that the US retains its leadership role in science and innovation.

Footnotes

His essay, “Science and Pseudoscience,” was not only scientific. It was about the connection between science and politics. By encompassing both, and emphasizing how important it was to demand “progression” and reject “degeneration,” in public life, Lakatos anticipated culture war battles today.

It’s safe to say we didn’t listen. In American politics, the tiniest whiffs of Lakatosianism, like reluctantly saying the other side has a good point, are scarce. Let alone “I was wrong.” Instead of daring our scientific and political enemies to test their theories, we attempt to do so much reputational harm that they never get the chance.

Science, politicians, and many institutions, notably journalism and higher education continue to erode public trust for this exact reason. They have rigged the game so they can never be wrong. Once a scientist or politician makes it into the public eye, they’re not going to let reality get in the way. Normal people know this because it’s simple, and because they’ve observed it countless times in other normal people and in public life.

Lakatos was a great philosopher. His ideas are relevant to pure science and particularly to the replication crisis that defines the modern era of science and has now slipped into politics.

But Lakatos’ life also underscores how great a grace he was offering his foes. His family was killed by the Nazis. Having become an ardent, even extreme Marxist, he was eventually tortured and expelled by Marxists for “revisionism.”

To be fair, Lakatos didn’t think Marxists (or Nazis) would pass his test. (He didn’t think Freudians would either, a prediction that has also held.) His offer was a challenge. It was a way of excluding movements from public life that were gaming the system, but he didn't shun his opponents. He gave them terms for peace.

Today, at least in the United States, such suffering, let alone a peace offer to people who have caused you personal and bodily harm, is rare. Polarization persists. We’re busy burning everything so the other side can't have it, including principles of thought and thereby anything that could bring us back together. The political and scientific theory of the day is that soon everyone on the other side will be expunged, or forced to recant under threat of global online humiliation.

20th century philosophy of science

Like the early 21st century, the early 20th century was a time of upheaval, both scientifically and politically. From the 1910s to the 1970s when Lakatos was writing, philosophers of science were struggling to generalize about science in a way that accounted for Einstein having so suddenly turned it upside down. Lakatos landed in a position between the more absolute Popper, and the more relativistic Kuhn.

What connected all three philosophers was a narrative since the Enlightenment that science had steadily advanced, but it was difficult to say how. Leaps forward were impossible to predict (Kuhn) and impossible to say are “forward” (Popper). Einstein showed that Newton was right about methodically asking questions of nature, but that Newton’s physics was, shockingly, wrong. It was so shocking that science and philosophy took a half-century to recover.

Unscientific research programs don’t make predictions that can be proven false — and, Lakatos thought, they don’t make predictions that come true. They particularly don’t say things that others can use to make predictions on their behalf as Einstein did. Lakatos believed that political movements and research programs had to progress by making predictions.

This is not to say that political movements need to become soothsayers or stock market analysts. Predictions are a way of keeping each other honest, as the arrow of time is agreed to go in one direction.

With this, Lakatos gave the world a mental model of successful scientific research programs, and belief systems in general. There’s a “hard core” to a belief system that makes, or implies, predictions about the future, and there’s a “protective belt” that gives the system a little wiggle room that accounts for all kinds of incidental error and false accusation.

It is, of course, cheating to design a belief system to be irrefutable, either with an unfalsifiable core, or a protective belt that makes the core impervious by definition. Whether the core falls to criticism is mediated by nature, not semantics or weaseling out of predictions post hoc.

The research programs Lakatos focused on in his essay were never settled in the 50 years since, which may indicate human beings’ inability to follow through and admit fault. However, the truth of what he said is more apparent than ever. It is of vital importance to specify in advance what the terms of peace are with your intellectual and political rivals. If you know your rivals will be proven wrong, all the better. You will have done the right thing and won the argument too.

Part II: Examples

There are two culture war battles that exemplify degenerating schools of thought: the far left contingent of metascience, and the fight for metascience’s traditional rival, the universities. I will argue that the left wing of metascience is far left compared to the general public, obviously political rather than scientific, and has drawn the movement into certain contradiction and no possibility of peace with either non-political metascientists or administrations to their political right. This will only prove successful if reality has a really strong leftward bias, which even they admit isn’t likely given their invocation of the millions killed by Lysenkoism.

On universities, two authors who have been critical of the far left, Sam Harris and Jonathan Rauch have recently come to higher education’s defense without mentioning the replication crisis, which is a convenient rhetorical opportunity to declare the fight to save universities is degenerating. The reason can be stated in one sentence: no one, even their political critics, will name a point at which the universities have slipped too far into irreproducibility.

Both the existing metascience movement and universities are threatened by a very specific person, Donald Trump, who does not represent any kind of scientific or Lakatosian ideal and doesn’t strive in that direction. However, focusing on him is shortsighted. The problem is bigger than he is. The problem these institutions face is that normal people can tell they are wrong.

Part II continues next week.

A working paper called “The effect of deactivating Facebook and Instagram on users’ emotional state” (Allcott et al. (2025)) recently went viral with over 5 million views:

Not only is the “screen time” literature rife with practically-null findings and evidence of p-hacking even at the meta-analytic level, Allcott et al. has clear red flags that should show up in any post-publication or peer review.

Despite the small effects, screen time is famously so fractious that when researchers did a Specification Curve Analysis — all defensible analyses to address a large range of statistical opinions — researchers on the other side of the debate redid the analysis on a subgroup and vehemently disagreed. The “potatoes vs. hard drugs” debate continues to this day with each group comparing social media to potatoes, and hard drugs respectively.

In the case of Allcott et al., the debate may be lopsided. While 5 million people saw the tweet, only about 15 downloaded the preregistration.

The following prompts are aimed at classroom discussion with unanswered take-home questions at the end. A number of issues in the paper are not covered that students can discover themselves, some of which I may have missed myself. Also, the paper will likely be critiqued again in peer review and in other posts. Students might enjoy trying to predict where the debate goes.

Introduction for students

How to do critical appraisal of research

When considering these problems, try to put yourself in the mind of a researcher. Have you ever been late trying to finish an essay for class and you can't think of anything interesting or original to write about? Have you felt pressure to find something, anything to say, and tinier and tinier observations about the material suddenly seem relevant?

Researchers face the same pressures except their work may take years to plan and carry out. There's tremendous pressure, both internally and from the journals that publish papers, to analyze and then write so it sounds like an interesting discovery has been made.

The paper we'll discuss was preregistered, which is a way of helping to prevent authors from claiming a discovery too easily.

Consider the difference between looking at some clouds and saying, “That looks like a rabbit!” and saying, “I think there's going to be a cloud that looks like a rabbit tomorrow.” A preregistration is saying what you're looking for in advance. Timing matters and it matters how specific your prediction is.

The authors didn't follow their preregistration exactly, which is considered legitimate if disclosed. The question is “did they disclose everything they changed?” and “do their changes threaten the validity of the findings?”

As you go through the prompts, think about the idea that researchers feel pressure to publish, even if their results are null, or “not a discovery.” (Null results should be published too, but they are unfortunately harder to publish and get attention for.) Do you find evidence of this? Are there outright errors or miscalculations? Are you being too skeptical and seeing errors that aren’t there?

These are things to think about when reviewing evidence in a scientific paper. You're not going to know if you’re detecting the authors’ motivations correctly unless the authors come out and say, “We didn't find anything interesting but we wanted to get into this journal anyway!” Thinking about the publication process is a way to help find the most common red flags. Always check yourself too and think about ways you could be wrong. If you are wrong, backtrack and try to make your review stronger. See if you can hold up to scrutiny too!

Prompt 1: The Abstract

This is where the authors summarize their paper and state their main findings.

We estimate the effect of social media deactivation on users’ emotional state in two large randomized experiments before the 2020 U.S. election. People who deactivated Facebook for the six weeks before the election reported a 0.060 standard deviation improvement in an index of happiness, depression, and anxiety, relative to controls who deactivated for just the first of those six weeks. People who deactivated Instagram for those six weeks reported a 0.041 standard deviation improvement relative to controls. Exploratory analysis suggests the Facebook effect is driven by people over 35, while the Instagram effect is driven by women under 25.

What are the main findings and their effect sizes (magnitude of the discovery)?

Suggested answer:

The main findings are that Facebook and Instagram deactivation was associated with an effect size of 0.060 and 0.041 standard deviations respectively on an index measuring happiness. Social media users who were randomly selected to deactivate their accounts reported being happier compared to the control group.

The secondary findings were that subgroups of users had different experiences. Facebook users over 35, and Instagram users under 25 had the greatest effects.

Prompt 2: Study design

In this paper, we report the results of the largest-ever experimental study on the effect of social media deactivation on users’ emotional state, which we carried out as part of a broader study of political outcomes before the 2020 U.S. presidential election. We recruited 19,857 Facebook users and 15,585 Instagram users who spent at least 15 minutes per day on the respective platform. We randomly assigned 27 percent of participants to a treatment group that was offered payment for deactivating their accounts for the six weeks before the election. The remaining participants formed a control group that was paid to deactivate for just the first of those six weeks. Our baseline and endline surveys elicited three measures of self-reported emotional state—how much of the time during the past four weeks that people felt happy, depressed, or anxious...

What are some rigorous aspects of the study design? Are there any confounds or other issues with the metrics used? (A confound is a factor that may work against the evidence of a discovery. It may not be declared openly.)

Suggested answer:

Rigorous aspects

The study is randomized, which means the researchers controlled who got the “treatment,” in this case deactivation of social media accounts, randomly so that any effect the treatment has can be attributed to the treatment and not other differences between the two groups. In an observational study, where the researchers have no control over who gets the treatment, the effect may be due to what caused the subjects to get the treatment instead of the treatment itself.

For instance, observational studies of patients at a hospital can, ironically, have more healthy people than the general population because healthy people tend to be more concerned with health, and go to the hospital more often for some treatments. (In other areas, this is reversed and only the most severe cases of a mild disease, for instance, end up in the hospital.)

The study also has a very high sample size. In human subject research, this is the number of participants. Higher sample sizes are generally considered more rigorous, although as sample size increases, smaller effects can be detected. Very large studies are more likely to find something, particularly if they've collected many variables.

These discoveries could lack clinical or practical significance, meaning that they're likely to be real, but humans have to decide how important they are.

Non-rigorous aspects:

The most obvious confound is that participants are paid to participate. If the payments are significant enough, the payments could be what's making users happier.

Self-reported measurements are generally considered less reliable than measurements that can be taken in a lab, or measurements that are less subjective. For instance, participants may be more enthusiastic about social media deactivation than the general population, which may bias their self-assessments.

Note: another source of measurement error is not apparent yet and will be covered in Prompt 4.

Importantly, the study was not blinded. The participants were told the lengths of time their accounts would be deactivated so they knew which arm of the trial they were in. In blinded studies, participants and even researchers are prevented from biasing the results with this knowledge.

Prompt 3: Study design continued

Participants who completed the baseline survey were randomized into two groups: Deactivation (27 percent) and Control (73) percent. The Deactivation group was told that they would receive $150 if they did not log into the focal platform for the next six weeks, while the Control group was told that they would receive $25 if they did not log in for the next week.

Are these payments significant enough to confound the study? Why or why not?

Can you think of a way of measuring the effect of the payments? Was there a way to design the study so participants could be paid but the payment wouldn't affect the results?

Suggested answer:

The effect of the payments could be measured by household income. If participants with lower household income were much more likely to say they felt happier with the higher payment, it could indicate that the payment was a factor.

The payments could be compared to an hour of work. If the effect of an hour of work a week is similar to the effect the study found, the confound could be of the same magnitude as the treatment.

Participants could have been paid the same amount whether they were in the deactivation group or not.

Prompt 4: The survey questions

This paper focuses on the three emotional state questions that were also included:

'Please tell us how much of the time during the past four weeks you felt [happy / depressed / anxious].' The response options were 'All of the time,' 'Often,' 'Sometimes,' 'Rarely,' and 'Never.'...

These three questions were drawn from the European Social Survey Well-being Module (Huppert et al. 2009) and are similar to other established emotional state measures.

From Huppert et al., (2009), the questions should be phrased:

I will now read out a list of the ways you might have felt or behaved in the past week.

Please tell me how much of the time during the past week:

(a) … you felt depressed

...

(d) … you were happy

...

(j) … you felt anxious

...

[Response code: 1 – None or almost none of the time, to 4 – All or almost all of the time]

Are the questions the same as the European Social Survey standard? If not, how are they different and could this affect the findings?

Suggested answer:

The questions do differ from the standard. The wording is slightly different, and the paper used 4 instead of 1 week, which the authors disclosed.

The paper also uses 5 possible responses and the standard has 4. The most significant way they differ, however, is the standard does not label the responses “rarely,” “sometimes,” and “often.” The standard question is written to be exhaustive of the time period: “How much of the time during the past week...,” [emphasis added] whereas the paper's responses can easily be confused to mean “individual instances” of being happy, depressed, or anxious.

This is important because a weaker treatment might affect individual instances of happiness that wouldn't affect whole portions of a week. Saying, “I was happy frequently this week” is different from, “I was happy most of the week.”

Short instances can also both increase at the same time. You could choose that you frequently have happy moments and frequently have sad moments in a single week. However, you couldn't have been happy and sad for 75% of a week. 75% + 75% > 100%.

Prompt 5: Baseline differences

We submitted an initial pre-analysis plan (PAP) on September 22 and a slightly updated final PAP on November 3rd, the day before the endline survey began...

The PAP originally implied that results for all outcomes would be presented in a single paper. However, as we drafted the paper, it became clear that it was not possible to fully present the motivation, related literature, robustness, and interpretation for both the political and emotional state outcomes in a single paper, so we split the results into two.

There was a significant difference between the emotional state of the treatment and control groups before the experiment took place. This may happen by chance, particularly when many variables are being measured. The authors also chose this variable to publish on separately. Does this matter in interpreting the results?

Suggested answer:

A preregistration and pre-analysis plan (PAP) is intended to restrict the authors' ability to make statistical choices that take advantage of random noise inherent in any dataset. If the analysis is set before the authors see the data, they won't be able to choose variables that happen to have noise that looks like a discovery. (There are many complexities to this. Exploratory data analysis may be very valuable and it is considered rigorous if disclosed and corrected for multiple comparison, multiple ways of analyzing and subgrouping the data.)

In this case, the authors chose to highlight a variable that randomly had a large imbalance before the experiment began without much justification. We know this is due to chance because the authors randomized participants into the two groups.

Variables with large baseline differences, or imbalance can be corrected or go uncorrected depending on how much the variable is known to be correlated with the outcome of interest. Imbalanced variables are subject to extra scrutiny because they may regress to the mean (become more moderate naturally) or, in this case, unhappy people may simply behave differently and perfectly correcting for this may not be possible.

Participants who score lower on the Emotional State Index may be more affected by the financial incentives, or by the election. The choice of this particular variable to highlight in another paper, which the authors may have reasoned has a better chance of significance and therefore publication, is a potential “red flag.” The fact that the effect size is around the same as the baseline difference also makes it a convincing confound.

Prompt 6: Effect size

Facebook deactivation improved happiness, depression, and anxiety by 0.053, 0.045, and 0.031, respectively. Similarly, Instagram deactivation improved happiness, depression, and anxiety in those original units by 0.037, 0.031, and 0.027, respectively. The average of these six effects is 0.038. This is equivalent to 3.8 percent of people saying they feel happy 'often' instead of 'sometimes.'

From the conclusion:

...the estimated effect sizes are smaller than benchmarks such as the effects of psychological interventions, nationwide mental health trends, and previous experimental estimates in smaller samples.

The authors admit the effect size is small. The benchmarks they used are plotted below:

How does this affect your interpretation? Is any effect small enough to ignore on important issues? What about the role of random noise in experiments?

Suggested answer:

A common critique of “screen time” literature is that small effects are not put in context (Allcott et al. may be less guilty of this than others). There's no rule about how small something has to be before it can be ignored, particularly in effect sizes, which are unitless. The clinical or practical importance of the variable matters a lot here too.

The other factor to consider is publication bias, which tends to skew the results we're able to see (generally) upwards.

Researchers suggest that the wide range of experiences of screen time and social media may be a factor too. Does a small effect for the whole population hide a major effect for a small subset of the population, or for particular experiences online? Some of these factors are addressed in the paper (and they're subject to their own statistical caveats). Others may be harder to measure. How do you measure bullying? Or comparing one's self to others online? The true, large effects may simply be in small corners of screen time research.

Searching for large effects comes with its own caveats just like searching through a particular dataset does. However, there's an argument to be made that “screen time” or even social media is too varied to generalize on all at once.

Take-home

1. The authors say the preregistration said they would use a p-value threshold of 0.05. Did they break this promise? Copy any statements from the paper.

2. How do the other changes in the preregistration and the timing of the changes affect your interpretation of the paper? Are all preregistration changes generally equal or are some more worthy of critique? Are there any “red lines” for you?

3. How would you compare and contrast the responsibilities of the authors with others involved in scientific communication like journalists and social media influencers?

4. How did your prior beliefs about social media affect your interpretation of the paper? Should these beliefs be taken into account? How much do you weigh your own experiences on social media in your beliefs?

5. Do you sympathize with authors of research papers, or these authors in particular who may feel pressure to publish more interesting papers? Do you think pressure affected the paper at all? Do you think some pressure to find statistically-positive results is beneficial?

Bonus question: A PubPeer comment on the companion paper to this one points out undisclosed connections to Facebook's parent company. Do you think this is a bigger factor than pressure to publish? Why or why not?

Ethics of post-publication review

Post-publication review, peer review, and critical assessment are not intended to completely invalidate research outside of some cases of flaws impossible to justify. These critiques act as a “check” on the authors' interpretation and research practices. It's generally good review practice to invite author commentary and comments from others with a different view, and never claim your review is “the final word.” This lesson is no different and responses are welcome.

The post, by image integrity sleuth Michael Rossner, now has a response from Csaba Szabo whose book I am forced to thoroughly recommend because he's right about a lot of things, notably the endless promotion of replication crisis interventions that are “palatable to official stakeholders but have neither worked nor will ever really work.”

Dr. Szabo has also been willing to reach across the proverbial aisle. I don't know what his political leanings are and don't need to, but the aisle is generally to the right of most researchers and his suspicion that “pushback may be less about the substance of the proposals and more about political dislike of those proposing them” is unavoidable. (Dr. Rossner denies having political motivation and that could be true.) Dr. Szabo has supported NIH Director Jay Bhattacharya, someone who has a “restore debate,” and forgive-your-political-opponents platform.

I also responded to Dr. Rossner, and wrote a similar response to Holden Thorp’s op-ed this week with the restrained title of “No YOU’RE a slug” on the same topic. Thorp’s op-ed is also a response to the Republican adoption of the replication crisis, and has more endless promotion of interventions that don't work.

My response to the Retraction Watch post is below. The usual promise to link to or publish rebuttal applies.

The head of the Center for Open Science, Brian Nosek, recently commented on Trump's “Gold standard of science” executive order. “If I was just reading section 3,... I’d be delighted - oh, fantastic! These are the things I talk about all the time - that it is my mission to advance, that the open science movement has been advocating for.” This was despite Brian's vociferous opposition to other parts of the order and to Trump in general.

In other words, “even a broken clock is right twice a day.” I agree. I am forced to agree because broken clocks are indeed right twice a day, and administrations I disagree with can say things I'm forced to agree with because that’s how true things work. They're independent of what I necessarily want.

The essay by Dr. Rossner is slightly different. It is a claim that broken clocks are not right twice a day and there is nothing to agree with in Trump's policies even the one generically about giving billions of dollars to replication. The author carefully evades being associated with Trump even though it's hard to tell what the difference is between science being “an iterative process” and funding replication. The broken clock is wrong at 2 PM because it’s actually 2 in the afternoon?

The essay is not only political, it is inaccurate. “Failure to replicate once does not mean that you won't replicate on the second, third, or fourth attempt.” It does mean you are less likely to replicate, otherwise we’re in real trouble. And scientists “replicating their own work enough times before deciding it is ready for publication” without publishing the failures is malpractice.

I say this not because I want to debate whether or not Trump is a bad man, or explore how thoroughly screwed I am having said the same thing as he did. I say this because the incredibly important topic of reproducibility crisis interventions has been plagued for ten to twenty years with one important flaw. We need adversaries in science but we don't want, you know, adversaries. Not people who are against us so much that we have to change. People who are against us just enough so we don't have to change.

One tactic I would suggest is to have ideas that even people who disagree with you on a fundamental level are forced to agree with. An idea is no good if only people with the same interests agree with it, and it's doubly no good if it's a perfect litmus test. Reject any idea that perfectly divides interest groups. It’s probably more interest than idea.

If you might have a liberal bias, ask an average conservative to critique it. If they’re able to poke holes in it based on common sense, it's probably not a good idea. As a non-academic, and non-PhD-holder I experience this all the time. I am in literal and virtual rooms full of PhDs frequently. The most popular ideas for reproducibility crisis reforms could be debunked by literate high schoolers or ChatGPT. Usually, refuting the latest reproducibility crisis reform is as simple as asking, “what if researchers don't want to, and don’t?” Researchers are regularly switching outcomes on their RCTs, and rigor is not often taught in schools. I don't think less tightly-regulated interventions, or less incontrovertible ones will work given that track record.

On the problem of being associated with the Trump administration, my advice to the open science movement is to say what's true, even if it hurts. The more it hurts, the more trust you will gain. So there’s no time like the present.

All of these documents depend on the ebbing reputation of science. They will seem more or less legitimate depending on whether you think there is a replication crisis, a credibility revolution, or not much of a crisis at all because science is about “bumbling around.” Boosting any of them is unfortunately political because, as we all know, American life is a zero sum game from birth to death where your every breath must be spent extinguishing those who would cost you political power. Never cease this, even when plumbing reality itself. All must perish in these flames.

But enough about that. Holden Thorp’s editorial was, as usual, dependent on the ebbing reputation of science. What you think of the practice of science will help determine what you think of a Holden Thorp essay. If you think science is mostly okay, you might agree with him that the process of writing papers and getting peer reviewed results into journals produces knowledge so certain that making counterfactual claims about it is “not covered by free speech.” Maybe you think science is really okay if you agree with that one.

The problem is that Holden’s boy Science who has knocked over your planter is really a good boy, he swears, and yet the boy is peeking between Holden’s knees with that sneer on his face. This is just boy stuff! What he needs is discipline, and he’s going to get it when he gets home. Don’t worry.

The essay is full of replication crisis intervention language, typically “We need to encourage schools to tie promotion to good practices. We should empower early-career researchers.” Journals, like the one Dr. Thorp heads, “need to be held to higher standards.” And so on. All of these things are true and yet they don’t seem to work. I agree with COPE guidelines and teaching the replication crisis in school and my opinion that their implementation in reality is complete and utter horseshit should reinforce that. Don’t pretend to care about the words in a declaration if you don’t care about the reality they entail.

Thorp takes this wordiness to an extreme. Why, it wasn’t my boy. It’s this sluggishness and defensiveness that has taken over the town. It’s those bad apples creating a “steady stream of problematic images.” That is what has led to this “so-called ‘replication crisis.’” (For anyone unfamiliar, saying it’s “so-called” is a cheeky way to dispute twenty years of science in a few keystrokes.)

It began, he says, “mostly in psychology,” in underpowered studies, and it “created the impression that unreliable research is widespread and not reproducible.” He goes on to say that there are many errors in Alzheimer’s research as well.

How the crisis began in psychology, was not widespread and yet leapt to Alzheimer's research is not explained. Maybe it was carried on this “steady stream” of problematic images connecting the two events in this verbal playground.

Dr. Thorp is skipping over cancer biology, lots of psychology, economics, cancer biology again, ecology, nutrition science, and hundreds of papers covering every aspect of the scientific process and its ebbing reputation. In 2016, most researchers agreed there’s a replication crisis off the top of their heads.

“The scientific community needs to face up to the extent to which its own actions have fed this perspective.” Perception. Perspective. These are the worst parents. The “don't get caught” parents.

“Although science is driven by data, politics and public opinion are shaped by anecdotes and storytelling. Thus, pointing out that many of these incidents are the result of a small number of actors does nothing to change the political narrative.”

[Emphasis added.]

The replication rate and the many practices we know with mathematical certainty are driving those numbers lower are not practiced by a small number of actors. One of the reasons we know the “incidents” are not isolated to psychology and then cancer biology and then coincidentally economics and recently Brazil is that they all have the same practices that are, very simply, caused by hypothesis testing for the purpose of publication. These practices are p-hacking, base-rate neglect, and the proverbial granddaddy, the one we should never, ever go to a publisher for an opinion on, publication bias.

The number of actors involved in publication bias is not small. It’s very large and Holden Thorp doesn't get to say what we do about it.

The essay goes on. Dr. Thorp deserves some credit for saying that the issue lies in the general area of his boy. If you work in science and you’re “sluggish” or “defensive,” your ears are really burning.

He also deserves some credit for saying, “the scientific community should be engaging in a conversation about problems and potential solutions.” Although “conversation” is another favorite of essay writers and bad parents, I think this is good. Let’s converse, and let’s include everyone. Let’s do this quickly for god’s sake. The boy’s in college.

The open science movement has been making steady progress upward in adoption of rigorous practices, however compared to the size, and increasing size, of science, it has a long way to go. Those of us watching and occasionally sticking our necks out to have our heads chopped off have had increasing angst as to when humans would start to more-often-than-not care.

The message of metascience and the open science movement has been that, even if it hurts, tell the truth. I disagree with this message in that telling the truth may end your career and so we should make sure that truth tellers are protected too. However, the basic principle is correct. For instance, let's say you have a hypothesis that wasn't supported by your own study. The open science movement says that it should be published anyway, even if it hurts.

Another way to put this is that individuals' incentives are not as important as science. Your personal desires have to serve truth, not commercial interests, fame and so on.

The angst of the open science movement increases when open science and metascience itself is in trouble. For instance, when metascientists seem to put personal desires above science. Not only does this potentially halt the movement, the purview of the movement is all of science. So there's something inherently worse about bad metascience. One teacher who teaches retrying an analysis can't do as much damage as one metascientist who teaches that p-hacking is okay.

Open science may therefore be discredited, and its discredit will spread around the world faster and more permanently than its slow progress towards credibility. There will be more discussion of errors in metascience than the brokenness of scientific practice in the first place because most of science still has the old incentives.

Unreality enters stage left

One of the worrying trends that may discredit open science is a small but apparently influential contingent of critics of positivism in open science. Positivism is the belief that there is such a thing as true and false. It is, as one non-positivist commentator recently said, “synonymous with science.” I agree.

The hosts of the ReproducibiliTea podcast — William Ngiam and Sarah Sauvé — went on to say that ideas of “complete reproducibility are out the window” in their recent, “radicalized” view. Confusingly, from their radicalized view, ordinary science is the the radical one. According to them, ordinary science and therefore positivism lies on a spectrum at “the most extreme end.” Apparently, if you don't want to be in the extreme, you have to accept a least some of what they have to say.

ReproducibiliTea, which gets its name from reproducibility, is a network of journal clubs that discuss metascience papers and is a rare open science success story and one of its most important institutions. Over 100 ReproducibiliTea journal clubs around the world meet periodically to host speakers and discuss metascience research. They are not openly political. Nor do they generally question or even characterize most people's epistemology. As reproducibilitea.org says, it can be isolating to be interested in improving research practices, and ReproducibiliTea meetings give people a place to go to discuss metascience and to feel less alone.

This is not the first time open science has met questions about reality. At one of its other success stories and most important institutions, the Center for Open Science, Training and Education Manager Crystal Steltenpohl has sometimes lectured about beliefs that seem incongruous with the major tentpoles of the movement.

These ideas are collected in a 2023 paper “Rethinking transparency and rigor from a qualitative open science perspective” that pokes holes in all the standard open science reforms, reproducibility and replicability, preregistration, and data sharing. This is ostensibly to give qualitative researchers a seat at the “table,” although it slips into a more broad agenda: “Open science guidelines fail to account for research based on epistemologies that are not strictly positivist.”

One of these fields, it claims, is epidemiology, or the study of diseases in large populations. This is because the journal Epidemiology once wrote that its policy is to not consider papers to be strictly “right” or “wrong.”

Both Steltenpohl and the journal are using a familiar objection to open science principles, that they cast previous work as “right” or “wrong.” Not only does this have nothing to do with positivism or whether there's any such thing as “right” or "“wrong,” metascientists go to elaborate lengths to underscore that they don't think replication success is the same as “right” or “wrong” and attempt to measure it in as many ways as possible. Arguably the most famous paper in the field, led by the Center for Open Science makes this clear. “Even research of exemplary quality may have irreproducible empirical findings because of random or systematic error.”

Once qualitative researchers get a seat at the table, Steltenpohl says, they have a lot to teach the others at the table. This “mutual understanding” and “respect” are obviously not the standard definitions. Usually, mutual understanding doesn't mean one party involving the other in its decisions, or adopting the others' teaching. Non-positivists have stepped to the debate stage, quoted from the rulebook that says we should have mutual respect, and declared themselves partial winners.

A Warning

I'll argue for positivism. However, the argument is very short and won't defeat the non-positivist movement, which is immune to rhetoric. So please indulge this warning. Imagine how choosing this moment to question reality looks to an ordinary person. For instance, imagine talking to a second grader who has gotten a very positivist quiz back with right answers and wrong ones. Let's say it has two answers marked wrong out of ten. This system is necessary, in part, to choose who gets to be college professors.

Imagine telling the second grader that if they stick with it for another 14 years and get their grades up, they can apply to spend another 8 years getting a PhD, finally getting their first grant when they're middle aged, rise to the apotheosis of rigorous science, metascience, rise a little further to be influential in that field just in time to burn it all down. They might find this defeating.

The reason influential people in metascience got to be that way was they worked harder, got their grades up to nine or ten out of ten right — while the kid next to them got 5 — is the belief in true things and false things. Society — that subsidized and encouraged their rise, watched their own kids be beaten out without claiming that 50% is just as good as 80% because there's no such thing as truth — is now being rewarded with a lit match.

Imagine further that we live in a world where the accusation of being anti-science has political connotations. Both sides of the political divide insult each other this way. Metascience and open science have been careful to stress that they are pro-science, that science is being practiced the wrong way. Imagine how it sounds, then, to say at this late date that the concept of science itself is wrong.

Academics have a hard time imagining what ordinary people think and how often other academics look like emperors with no clothes. If an academic can imagine this well enough, they might pause, consider whether or not the people at the bottom of the intellectual status ladder are always wrong, or maybe that things have gone so far that it doesn't take any sophistication to see the emperor has no clothes. Imagine the kid who got 50% on his test is still smart enough to understand that what an academic says could be baloney.

This warning is not proper debate. I think it is necessary, though, because the success of non-positivism, which is very clearly lapsing into anti-positivism, was not built on good arguments. It was built on fear. The people who nominally support anti-positivism, or say nothing when it's lectured, are afraid. They're afraid of being embarrassed and denounced by anti-positivists.

The Debate

The argument for positivism writes itself. There are true things and false things. We usually don't know perfectly what's true and what's false, and it's difficult for semantic and epistemic reasons to perfectly categorize even our most certain beliefs as true or false. This doesn't mean true and false things don't exist and we can't know them at all. It is the reason we need science in the first place and expend ungodly amounts of time and energy on it.

Why spend so much on science if the most certain findings can be erased with the snap of an anti-positivist's fingers? In their telling, none of it is true by definition. The early finding that started metascience that half of scientific papers are false is meaningless. The many findings that scientific papers are often “false positives” are also meaningless to the non-positivist.

When statistics refers to “true” and “false” positives, it is not just semantic. The field doesn't work without some concept of measurements that correspond to reality. Replication, the basis for what was called the replication crisis is also meaningless without the concept of true because there's no statistical measurement without the belief that asking nature is fruitful.

Positivism is, in other words, fundamental to the science we’re trying to protect. As a direction for science and for the open science movement, it’s trivially correct. On a philosophical level, it’s less trivial, but I won’t be more convincing than Descartes so I won’t bother.

So much rhetorical chaff has been thrown at this debate, which has the dual purpose of confusion and aiding a general sense that there's something else out there other than true and false. There's a grey zone of perception and unknown unknowns. Certainly there's something there unclassifiable. I can't knock all of this down, other than to plead with readers to consider the size and multitude of those arguments as a stroke against them.

It won't last

As stated, I don't think the argument against there being true things and false things that we can perceive, has succeeded on merit. I think it has succeeded because people are frightened of arguments that seem overtly political, and of people willing to torch the concept of truth. So, some more warnings and predictions.

The non-positivist movement won't last. Things that aren't true are hard to agree on. So there are infinite versions of anti-positivism and it will be difficult for anti-positivists to agree on things without being able to ask nature to settle their disagreements.

Being overtly political doesn't help non-positivists because being political comes with political commitments. Once someone senses which political commitments the non-positivist have (it won't be hard), they can simply ask if the non-positivist's political commitments are true or false. Political polarization is so bad now that both sides have things that, if you say they're not true, or even that they don't have a truth value, you're kicked out immediately.

This works for all perils, not just political ones. Would anti-positivists be willing to go to anti-positivists surgeons, or be judged by a jury that doesn't believe in truth versus fiction? No. Neither do the rest of us.

What to do

I think everyone should be vocal. Announce whether you're a positivist or not. It's not fair to rise through the ranks of the fragile open science movement and only announce this after you have the keys to an important institution. Positivists too. Dare to say you believe some things are true.

As well, I wonder if open science and non-positivism are the same movements. As one of the podcasters said, qualitative researchers don't see the point. One seems to disagree a lot with the other. I'm not saying, “leave.” In fact, there are a lot of reasons why I think the open science movement is already doomed and should be rebuilt on adversarial relationships. Maybe instead of protecting the existing movement from discredit, it should be discredited, and having some anti-positivists in key places is productive in that regard.

I would say this is an invitation to non-positivists to debate, which is still welcome. However, the defeat of a movement like non-positivism that is immune to rhetoric depends not on rhetoric, but on a mental model of where the movement comes from.

To illustrate, suppose a hypothetical movement starts winning debates by replying, “no it's not” over and over, and then, when that doesn't work, “no it's not, infinity.” This method defeats all rhetoric easily, but we can reason that it's patently against debate by design and ignore it. Or we may point out that their methods can be used to defeat anyone rhetorically, even themselves. (Although, I have heard that arguing, “no it's not infinity times two" is ineffective.)

Whatever the outcome, there's always the final arbiter of history, which has defeated stronger and more convincing foes.

I sent a copy of this post to Drs. Ngiam, Sauvé, and Steltenpohl a week in advance. I have not received a reply. Any response is welcome and will be linked here.