The politics of embarrassment: Part III
Universities are worth having
Part III in a series of essays centered around Imre Lakatos’ “Science and Pseudoscience” takes issue with “Universities are Worth Saving” by Jonathan Rauch and Sam Harris’ podcast episode defending expertise (“Intellectual Authority and its Discontents”). Rauch and Harris are targets of convenience because they are both not part of the far left that has taken over metascience (part II) and seem to want more universal principles in political life. Yet they defended universities without mentioning their primary fault of the last 20 years, the replication crisis.
The thesis of this series is that science and the culture war using it as a political football are trapped in the politics of embarrassment. This is anti-Lakatosian because we stake out more and more extreme positions instead of risking embarrassment by admitting the future event that would prove us wrong. Predictive ability is a demarcation between science and pseudoscience that will expose the extremes instead of feeding them. Lakatos, as a victim of Nazis and a Marxist expelled by Marxism, knew these consequences better than we do.
This essay is full of arguments a fortiori. For instance, if the pinnacle of rigorous science, metascience, can’t get its shit together, then neither can normal science. If the pinnacles of anti-culture war rhetoric and universal values like Rauch and Harris defend universities without end, then who can get them to reform?
As before with Brian Nosek, I don’t want to single out people I otherwise respect for not deferring to a philosopher few have heard of. It’s one of several faults this reasoning has. However, American life is full of accepting the leadership of people who’ve achieved these pinnacles like Rauch and Harris have. Throwing them some hard questions can’t hurt. They are both symbolic, and actual representatives.
I sent a version of this post to Quillette as a rebuttal, and emailed Sam my objection at the time. On Wednesday, I sent a copy of this updated post to both for comment and received no reply. As always, the offer stands after it is published.
The essay “Universities are Worth Saving” by Jonathan Rauch is brilliant and nonetheless wrong in its conclusion. Universities are certainly worth having, but their current incarnation as irreproducible science factories is not worth saving. We have had institutions too big to fail in the past, namely banks. We require both to follow rules that tether them to reality, and we should agree that universities are not an institution we should protect by definition. There’s a certain point where they stop being the thing they claim to be.
In Rauch’s view, universities are a core part of the reality-based community because they follow rules. But they don’t. They hacked those rules long ago.
For twenty years, academics have been supporting poor methodology with full knowledge of the replication crisis they caused, and have done very little about it. The depths of this crisis have still only been plumbed in a few fields, and it is not taught in schools. If rules are how universities stay in the reality-based community, they’re out. It’s as simple as that.
Rauch explores replication only to underscore the kind of reality we’re talking about: “The rules are impersonal: Persons are interchangeable, so anyone can vote and gets the same vote. Applying this idea to science: anyone can replicate an experiment, and it better come out the same way. Anything that anyone can do, anyone else can do. No one in particular is in charge.”
This is true, but denying the implementation of these rules is insulting to reality and to ordinary people. In fact, the true reality-based community may be better described as the large swath of humanity that depends on reality and lacks the power to change it at will. It would be more charitable to reserve “reality-based community” for the people who will cease to be if they don’t stay tethered to reality instead of the community that gets to choose when to follow the rules based on — let’s face it — government grants.
The replication crisis
In 2005, John Ioannidis famously conjectured that half of published research is false. This is a complex story, but he was repeatedly found to be prescient. In 2011, a paper showed that “the rules” can easily be hacked. These are only two of the most famous papers exposing more or less the same flaw – which can be summed up as “flexibility.” There are hundreds.
Those who have read the metascience literature that followed the replication crisis will be hard-pressed to say that universities acted in good faith or acted very much at all except to accelerate publication. Publication wins grants and rungs on the university rankings. Never mind sick people or society, they said. We have U.S. News and World Report to worry about.
Each year that goes by makes it more convincing that universities are willfully maintaining their place in the reality-based community at the expense of reality. As results came out in the early 2010s suggesting science is not self-correcting and it can be hacked, academia plugged its ears. Worse, the early papers were, through no fault of their own, also a how-to guide to academic success through p-hacking. Furthermore, academics now know how many others are hacking too, and how little we can do about it.
In other words, academia’s Natural Selection of Bad Science was once selecting for the trait “p-hacker,” and now it’s selecting for the trait “remorseless p-hacker.”
Friends of p-hackers
P-hacking, or running analyses and checking the results to see which one you like, requires computers and those computers are getting faster, cheaper, and better at p-hacking. This bending of the rules wasn’t discovered in 2011 and slowly fixed over the next decade as one would expect. It has arguably accelerated. Two recent incidents in p-hacking give us a peek at the character of academia, and maybe a little on the moral hazard of defending an institution without end.
We now entertain the ludicrous conjecture that researchers can force their p-values below 0.05 to get published, but other round numbers like 0.01 are safe from meddling. This (slight) p-value migration was taken as a good sign in both ordinary science circles, and in science reform. The reasoning went that more p-values in these carefully-chosen buckets was a sign that academia is good, actually, and reform worked. Instead, it's a phenomenon even the author admits could be just more p-hacking:
At a conference on scientific reform called META-REP, which was otherwise excellent, a workshop organizer slipped up and suggested that we could use a software library to select which analyses to run (p-hacking) as opposed to showing all analyses, something done for the sake of transparency. When I asked what the difference is, then, between this workshop and extreme p-hacking, the organizers’ response was, “Well, we hope people wouldn’t do that.” They would and they will.
So it’s important to know which rules can, and are, being broken. A little bit of flexibility is expected, but that flexibility can’t simply be used thusly: “take any number that represents academia as a whole and cross it out. Write another number that’s better.”
In banking terms, the rule-breaking that caused the financial crisis was obscure like the mass measurement of p-values above. The more obvious rules (don’t take money from the vault) were followed and the more obscure ones weren’t.
There’s no need to trust anecdotal evidence. There are heaps of studies on this. A 2025 paper found that knowing about poor methods is only weakly correlated with not participating in them, and, to the surprise of the authors, 30% of respondents “endorse p-hacking.”
One of the early reformers, Leif Nelson who, along with his co-authors, coined the term p-hacking back in 2014 has gone further, saying, “Everyone p-hacks.” His fellow reformer, Simine Vazire, argued a few months ago that it isn't good for your career to hold reform views by which she meant, in part, not to p-hack. “We’re bleeding early career researchers” she said, and we need to protect them.
The “everyone p-hacks and it’s dangerous to tell them not to” environment is one that people working on this issue know well, and the literature backs them up time and time again. And, despite everything we thought about the pinnacle of the reality-based community and their commitment to peer-reviewed literature, the literature has been ignored.
Right vs. Left
The fight for and against universities is often couched as political, right versus left. Consider a radical hypothesis to the contrary. What if both political parties are willing to control a sort of “knowledge machine” like the universities? What if the fight for universities isn’t inherently left versus right, it only appears that way because the left has them now and the right wants them? Churches were once influential knowledge machines and they were politically potent too. The right predominantly occupied them and protected their territory.
Right vs. left, and reality-based community vs. non-reality-based community are not fitting dichotomies in the modern university crisis because they don’t encapsulate the fact that academia doesn’t always follow its own rules. A better one might be common sense versus academic sense. Common sense is much maligned but it is at least held by people who can’t ignore reality to the same degree academics can. Academic sense is held by people who have dedicated their lives to knowledge production but are also in a position to break a few rules.
Sam Harris defended intellectual authority around the same time Rauch did. This was also the period that science.org was using phrases like “god help us.” He criticized the intellectual chaos on the right and appealed to the “vast reservoirs of integrity in academia.” I agree with this as a broad-brush statement. However, it underestimates the power of academia being able to break its own rules. This is not simply the “smart” versus “freaks.” Like Rauch, he seems to deny that anyone is in charge in this area of the world.
On whether or not ordinary people are “smart,” let me reiterate that ordinary people generally don’t have a choice of being tethered to reality. As Sam says, some aren’t, and some academics aren’t either. But there are “vast reservoirs of integrity” — and what some might call smartness — among laypeople too.
As an extraordinary example of common versus academic sense, in the United States, most non-academics believe COVID leaked from a lab, and this is a surprisingly bipartisan view. Virologists and epidemiologists themselves put it at about a 20% chance on average. It seems there are issues that the uneducated simply can’t accept no matter how vehemently the educated tell them they’re wrong.
However likely you think the lab leak is, even if it’s less than polling virologists implies, it’s too close for comfort. Suppose there’s a lesser, 15% chance that COVID came from a lab. If COVID did come from a lab, it means things with the academy are much worse than we thought: Normal people were more correct than scientists about a substantive matter. A new contender might be emerging to contain the academy that conservatives couldn’t: normies.
COVID fits neatly into the recent Stand Up for Science petition covered in part II, which uses millions of lives lost at the extreme ends of the political spectrum from Lysenkoism to Nazism as a demarcation of where things have gone certainly wrong. True-to-form, however, the academic petitioners get to ignore the fact that we may have just experienced another great loss of life, and the villain may have been science in both its production and cover-up. What irony then that even this incredibly rough demarcation — which is less a demarcation and more a way of figuring things out after millions of people have died — is still wrong.
In another telling poll, Rauch points out that the number of people who think universities are negative for society has jumped 19 points. 45% of people think universities are net negative. (A more recent poll from New America asked a similar “positive or negative effect” question and got an equivalent 44%.) This jump happened in the same period as the unmentioned replication crisis.
Rauch then concludes that these normies are wrong. They could be right. Even if we assign a small chance to them being right, the implications are profound. It means the height of the reality-based community that is in control of humans’ greatest invention, the scientific method, might be on balance bad for society. This could be worth looking into before another 20 years goes by.
The power of selective reporting
Rauch and Harris didn’t mention the replication crisis, the “moon landing” of discontent with intellectual authority. This is called selective reporting. It is the supreme breaking of the rules.
As Rauch puts it, universities have rules. Banks have rules too. Rules matter because, if you cross out “vault” and write “piggy bank,” you can take all the money you want. There’s no bigger intellectual piggy bank than selective reporting. It is a sin to do it, and yet it encompasses p-hacking, publication bias, and ubiquitously ignoring the metascience literature.
To be fair, these debates are full of cherry picking. Articles about the attack on science regularly, relentlessly cite vaccine science, climate change, and flat earth instead of social psychology, cancer biology, economics, Alzheimer’s, neuroimaging and the many hundreds of scandals in knowledge production today. Journalists should subscribe to Retraction Watch and browse PubPeer.
Harris and Rauch broke the rules. Academics break the rules. So when someone without all the privileges a good education confers says that academics are all a bunch of self-publishing p-hackers, I say let them say it and don’t interrupt. If academia wanted to take care of their reputation, they should have sorted this out many years ago. Particularly, people left out of the reality-based community won’t be charitable.
Conclusion
Reform of intellectual authority is a hard problem. Science is broken at exactly the places it can stay broken. P-hacking is done in secret. The “base rates” pointed out in 2005 even doctors don’t understand. And publication bias is a domain largely controlled by private publishing companies. But it must be faced, and it is one of the reasons the right believes academics are very frequently wrong. They are frequently wrong.
So the question for Rauch and Harris is, “At what point would you agree to ‘burn the universities down’ metaphorically speaking?” Suppose the replication rate of academic science goes to 25%, or 10%. Suppose the universities thwart movement after movement, on the left and right? Suppose the ability to generate knowledge with a hacked machine is too powerful. Is there a metric, and a point on that metric at which you will say your views have changed?
To avoid hypocrisy, I’ll do this too. Stuart Buck estimated the replication rate should be around 80-90%. The rate is not only difficult to estimate, but in general we don’t measure the thing we are after. We measure p-values, not probability of hypotheses. Despite this, I concur the rate can’t be expected to go to 100%, and 80 is not a bad guess. 80% replication rate is about where I would join team “defend.” I joined team “criticize” at 50%. Recently there were hints of 26%.