Physics has all the ingredients for scientific crisis
It turns out all you need is money and a googol of hypotheses to write up
In 2016, a famous survey was published in Nature called “1,500 scientists lift the lid on reproducibility.” It appeared to confirm that scientists themselves agreed there was a “significant crisis” in science.
What had been theory in the 2000s and confirmed empirically in the 2010s was now accepted by most researchers. It turned out they knew partially from their own experience that it was true and worrying. The loop had been closed, it seemed.
The “hard” fields of physics, chemistry, and engineering fared best in the poll. Now, thanks to physicists Sabine Hossenfelder and Jesper Møller Grimstrup, we may need to start rethinking what became a popular idea, that the crisis is due to the “softness” of social psychology (and, for some reason, several other fields).
Reading Hossenfelder and Grimstrup’s books Lost in Math and The Ant Mill is a delight and there’s real metascience in them. The crisis in physics is a natural experiment in which many of the other potential causes of the crisis from other fields have been removed, leaving only the pure passion for publication and funding.
In physics, you take away the questions of validity and flexible interpretation from psychology. You take away the closed data and incomplete methods from cancer biology and others. (The existential threat of not getting along with your mentors and co-authors and the other familiar pathologies of science are still there.)
This experiment suggests that there’s one common factor in science, the fact that the people who support you have no idea what you’re doing and you have little incentive to tell them.
Both authors have made an argument the public may have a chance of understanding, that hypotheses chosen from a near infinite set with nothing to recommend them have no chance of being true. Grimstrup estimates there are 10500 possible hypotheses in string theory. Each can be a paper.
It turns out, according to these very detailed accounts, that physicists will happily milk string theory just as psychologists invented new primes.
Familiar patterns
Much of the observations in these books are familiar. They have been observed in other fields and by other reformers. Goodhart’s Law, the Matthew Effect, a disturbing element of evolution in science towards worse practices.
Some observations are refreshing that you don’t often see, certainly not in psychology. The idea that researchers should be judged not only by the compromises they make for the sake of their careers, but for their silence in the face of such an obvious elephant in the room. Grimstrup even calls for — nondestructive — reform from outside. Hossenfelder has done more than her share on that front.
These are, in other words, ordinary observations from an extraordinary place few understand and few would suspect.
Physicist Alan Sokal once derided the encroachment on academia by those who deny the existence of objective reality. Years later, physicists started to deny reality too. Hossenfelder’s conversation with the eminent cosmologist George Ellis in which he laments this development, that some physicists are claiming theories don’t need to be supported by experiment, and feels powerless to do anything about it, is particularly affecting.
The most worrying part of these books is there doesn’t seem to be any natural limit before we’ve written about all 10500 hypotheses. Grimstrup paints a picture of physics most certainly continuing with groupthink fueled by government grants and the metric boosting multi-authored paper.
The power of the crowd
Hossenfelder has 1.7 million followers on YouTube and is an influential physicist in her own right. Her discussion with George Ellis was on something they’ve been saying since 2014. Hossenfelder and Ellis represent the widest influence imaginable both popular and scientific. We should be worried that the most powerful people who have worked in science, people like Walter Koroshetz, Jay Bhattacharya, Jacob Cohen, Daniel Kahneman, Csaba Szabo, Simine Vazire among many others, are often powerless to direct the crowd.
John Ioannidis is the world number 8 in citations. It doesn’t mean he gets what he wants, and published research is still theoretically, and many times now empirically, half wrong.1 It is certainly not yet Bayesian.
The books underscore that going against the grain in science is lonely and one’s certainty starts anecdotally and personally. Grimstrup’s copies of emails from professors, editors, and peer reviewers are not as powerful as a conversation he recounts with his friend in which his friend admits he can’t take the chance on any new theories as much as he may like them. He has to play the game and look for easy publication and surer funding.
It is almost literary, to know your tribe’s ideas are wrong but you and your loved ones can’t stop pretending to believe them. In these moments, Grimstrup excels.
I spent seven years at the Niels Bohr Institute where I have been at countless seminars on string theory and I have never, not even once, heard anyone in the string theory community mention the very obvious problem that this approach has: its inability to produce falsifiable prediction and the fact that it has failed in those few occasions where it has made contact with data.
The books are a delight, and horrifying. In a way they are reassuring. The crisis has reached the basement as Hossenfelder puts it, the bedrock of science. Although it doesn’t look like there will be any end forthcoming, it is clear now that there is no field where the crisis could have stopped. The bedrock might be our only hope.
If this binarization of findings into “right” and “wrong” troubles you, perhaps “half of the effect when remeasured” will ease your mind.
One thing that frustrates me about discussions of the crisis in fundamental physics is the lack of (good) suggestion for what researchers ought to be doing instead of putting out more string theory papers etc. (I realize that Hossenfelder includes a list of suggestions at the end of her book, but I certainly do not find them compelling).
The problem is that there is essentially no data pointing to an alternative to the null hypothesis (the standard model + Lambda CDM). No experiment has returned a really surprising result that held up to scrutiny in decades (the Higgs and gravitational waves were each predicted decades ago, and failure to find them would have been more surprising). Perhaps then, we should accept the conclusion that the null hypothesis is simply true, and that it's time to close up shop and do something else. And perhaps the second part of that conclusion is correct. But the problem with the first part is that the standard model + general relativity is not renormalizable, and thus is very likely mathematically inconsistent as a quantum theory (of course the same could be said about general relativity on it's own).
If we are to do something other than close up shop, it seems the best way out of the issues pointed out by Hossenfelder and Grimstrup is to go all in on experiment in a way that is as hypothesis agnostic as possible, in the hopes of getting some clue. However, critics like Hossenfelder consistently oppose such experiments on the grounds that there is no good theoretical reason to expect a non-null result!
I should point out that, at least to my mind, there is a perfectly reasonable framework in which to look for extensions to the standard model in a very broad (hypothesis agnostic) way, given by the standard model effective field theory (and there is likewise an effective field theory approach to characterizing corrections to general relativity).
Love your posts!