To truly appreciate the angst of the open science movement, it's necessary to understand that for decades a small group of people have known that the practice of science is broken. Humans' greatest invention is being used wrong. This was surprising, and all these years later it is even more surprising that humans didn't frequently care.
The open science movement has been making steady progress upward in adoption of rigorous practices, however compared to the size, and increasing size, of science, it has a long way to go. Those of us watching and occasionally sticking our necks out to have our heads chopped off have had increasing angst as to when humans would start to more-often-than-not care.
The message of metascience and the open science movement has been that, even if it hurts, tell the truth. I disagree with this message in that telling the truth may end your career and so we should make sure that truth tellers are protected too. However, the basic principle is correct. For instance, let's say you have a hypothesis that wasn't supported by your own study. The open science movement says that it should be published anyway, even if it hurts.
Another way to put this is that individuals' incentives are not as important as science. Your personal desires have to serve truth, not commercial interests, fame and so on.
The angst of the open science movement increases when open science and metascience itself is in trouble. For instance, when metascientists seem to put personal desires above science. Not only does this potentially halt the movement, the purview of the movement is all of science. So there's something inherently worse about bad metascience. One teacher who teaches retrying an analysis can't do as much damage as one metascientist who teaches that p-hacking is okay.
Open science may therefore be discredited, and its discredit will spread around the world faster and more permanently than its slow progress towards credibility. There will be more discussion of errors in metascience than the brokenness of scientific practice in the first place because most of science still has the old incentives.
Unreality enters stage left
One of the worrying trends that may discredit open science is a small but apparently influential contingent of critics of positivism in open science. Positivism is the belief that there is such a thing as true and false. It is, as one non-positivist commentator recently said, “synonymous with science.” I agree.
The hosts of the ReproducibiliTea podcast — William Ngiam and Sarah Sauvé — went on to say that ideas of “complete reproducibility are out the window” in their recent, “radicalized” view. Confusingly, from their radicalized view, ordinary science is the the radical one. According to them, ordinary science and therefore positivism lies on a spectrum at “the most extreme end.” Apparently, if you don't want to be in the extreme, you have to accept a least some of what they have to say.
ReproducibiliTea, which gets its name from reproducibility, is a network of journal clubs that discuss metascience papers and is a rare open science success story and one of its most important institutions. Over 100 ReproducibiliTea journal clubs around the world meet periodically to host speakers and discuss metascience research. They are not openly political. Nor do they generally question or even characterize most people's epistemology. As reproducibilitea.org says, it can be isolating to be interested in improving research practices, and ReproducibiliTea meetings give people a place to go to discuss metascience and to feel less alone.
This is not the first time open science has met questions about reality. At one of its other success stories and most important institutions, the Center for Open Science, Training and Education Manager Crystal Steltenpohl has sometimes lectured about beliefs that seem incongruous with the major tentpoles of the movement.
These ideas are collected in a 2023 paper “Rethinking transparency and rigor from a qualitative open science perspective” that pokes holes in all the standard open science reforms, reproducibility and replicability, preregistration, and data sharing. This is ostensibly to give qualitative researchers a seat at the “table,” although it slips into a more broad agenda: “Open science guidelines fail to account for research based on epistemologies that are not strictly positivist.”
One of these fields, it claims, is epidemiology, or the study of diseases in large populations. This is because the journal Epidemiology once wrote that its policy is to not consider papers to be strictly “right” or “wrong.”
Both Steltenpohl and the journal are using a familiar objection to open science principles, that they cast previous work as “right” or “wrong.” Not only does this have nothing to do with positivism or whether there's any such thing as “right” or "“wrong,” metascientists go to elaborate lengths to underscore that they don't think replication success is the same as “right” or “wrong” and attempt to measure it in as many ways as possible. Arguably the most famous paper in the field, led by the Center for Open Science makes this clear. “Even research of exemplary quality may have irreproducible empirical findings because of random or systematic error.”
Once qualitative researchers get a seat at the table, Steltenpohl says, they have a lot to teach the others at the table. This “mutual understanding” and “respect” are obviously not the standard definitions. Usually, mutual understanding doesn't mean one party involving the other in its decisions, or adopting the others' teaching. Non-positivists have stepped to the debate stage, quoted from the rulebook that says we should have mutual respect, and declared themselves partial winners.
A Warning
I'll argue for positivism. However, the argument is very short and won't defeat the non-positivist movement, which is immune to rhetoric. So please indulge this warning. Imagine how choosing this moment to question reality looks to an ordinary person. For instance, imagine talking to a second grader who has gotten a very positivist quiz back with right answers and wrong ones. Let's say it has two answers marked wrong out of ten. This system is necessary, in part, to choose who gets to be college professors.
Imagine telling the second grader that if they stick with it for another 14 years and get their grades up, they can apply to spend another 8 years getting a PhD, finally getting their first grant when they're middle aged, rise to the apotheosis of rigorous science, metascience, rise a little further to be influential in that field just in time to burn it all down. They might find this defeating.
The reason influential people in metascience got to be that way was they worked harder, got their grades up to nine or ten out of ten right — while the kid next to them got 5 — is the belief in true things and false things. Society — that subsidized and encouraged their rise, watched their own kids be beaten out without claiming that 50% is just as good as 80% because there's no such thing as truth — is now being rewarded with a lit match.
Imagine further that we live in a world where the accusation of being anti-science has political connotations. Both sides of the political divide insult each other this way. Metascience and open science have been careful to stress that they are pro-science, that science is being practiced the wrong way. Imagine how it sounds, then, to say at this late date that the concept of science itself is wrong.
Academics have a hard time imagining what ordinary people think and how often other academics look like emperors with no clothes. If an academic can imagine this well enough, they might pause, consider whether or not the people at the bottom of the intellectual status ladder are always wrong, or maybe that things have gone so far that it doesn't take any sophistication to see the emperor has no clothes. Imagine the kid who got 50% on his test is still smart enough to understand that what an academic says could be baloney.
This warning is not proper debate. I think it is necessary, though, because the success of non-positivism, which is very clearly lapsing into anti-positivism, was not built on good arguments. It was built on fear. The people who nominally support anti-positivism, or say nothing when it's lectured, are afraid. They're afraid of being embarrassed and denounced by anti-positivists.
The Debate
The argument for positivism writes itself. There are true things and false things. We usually don't know perfectly what's true and what's false, and it's difficult for semantic and epistemic reasons to perfectly categorize even our most certain beliefs as true or false. This doesn't mean true and false things don't exist and we can't know them at all. It is the reason we need science in the first place and expend ungodly amounts of time and energy on it.
Why spend so much on science if the most certain findings can be erased with the snap of an anti-positivist's fingers? In their telling, none of it is true by definition. The early finding that started metascience that half of scientific papers are false is meaningless. The many findings that scientific papers are often “false positives” are also meaningless to the non-positivist.
When statistics refers to “true” and “false” positives, it is not just semantic. The field doesn't work without some concept of measurements that correspond to reality. Replication, the basis for what was called the replication crisis is also meaningless without the concept of true because there's no statistical measurement without the belief that asking nature is fruitful.
Positivism is, in other words, fundamental to the science we’re trying to protect. As a direction for science and for the open science movement, it’s trivially correct. On a philosophical level, it’s less trivial, but I won’t be more convincing than Descartes so I won’t bother.
So much rhetorical chaff has been thrown at this debate, which has the dual purpose of confusion and aiding a general sense that there's something else out there other than true and false. There's a grey zone of perception and unknown unknowns. Certainly there's something there unclassifiable. I can't knock all of this down, other than to plead with readers to consider the size and multitude of those arguments as a stroke against them.
It won't last
As stated, I don't think the argument against there being true things and false things that we can perceive, has succeeded on merit. I think it has succeeded because people are frightened of arguments that seem overtly political, and of people willing to torch the concept of truth. So, some more warnings and predictions.
The non-positivist movement won't last. Things that aren't true are hard to agree on. So there are infinite versions of anti-positivism and it will be difficult for anti-positivists to agree on things without being able to ask nature to settle their disagreements.
Being overtly political doesn't help non-positivists because being political comes with political commitments. Once someone senses which political commitments the non-positivist have (it won't be hard), they can simply ask if the non-positivist's political commitments are true or false. Political polarization is so bad now that both sides have things that, if you say they're not true, or even that they don't have a truth value, you're kicked out immediately.
This works for all perils, not just political ones. Would anti-positivists be willing to go to anti-positivists surgeons, or be judged by a jury that doesn't believe in truth versus fiction? No. Neither do the rest of us.
What to do
I think everyone should be vocal. Announce whether you're a positivist or not. It's not fair to rise through the ranks of the fragile open science movement and only announce this after you have the keys to an important institution. Positivists too. Dare to say you believe some things are true.
As well, I wonder if open science and non-positivism are the same movements. As one of the podcasters said, qualitative researchers don't see the point. One seems to disagree a lot with the other. I'm not saying, “leave.” In fact, there are a lot of reasons why I think the open science movement is already doomed and should be rebuilt on adversarial relationships. Maybe instead of protecting the existing movement from discredit, it should be discredited, and having some anti-positivists in key places is productive in that regard.
I would say this is an invitation to non-positivists to debate, which is still welcome. However, the defeat of a movement like non-positivism that is immune to rhetoric depends not on rhetoric, but on a mental model of where the movement comes from.
To illustrate, suppose a hypothetical movement starts winning debates by replying, “no it's not” over and over, and then, when that doesn't work, “no it's not, infinity.” This method defeats all rhetoric easily, but we can reason that it's patently against debate by design and ignore it. Or we may point out that their methods can be used to defeat anyone rhetorically, even themselves. (Although, I have heard that arguing, “no it's not infinity times two" is ineffective.)
Whatever the outcome, there's always the final arbiter of history, which has defeated stronger and more convincing foes.
I sent a copy of this post to Drs. Ngiam, Sauvé, and Steltenpohl a week in advance. I have not received a reply. Any response is welcome and will be linked here.