Imagine a universe where a random protein out of billions in your gut doesn’t cure depression. Do you find that universe easy to imagine?
If your response was that it’s easy to imagine because let’s face it, we probably live in that universe, congratulations, you’re a Bayesian.
If you’d asked the same question twenty years ago about the depression gene, one of tens of thousands of gene variants, the answer would be the same and you could have predicted the fall of many millions of dollars in research with a simple thought experiment.
The fragility of this kind of research has to do with the fact that most experiments involve assuming a universe where something doesn’t work (a drug, a gut protein etc.) and seeing how surprising the finding would be in that universe. The part that’s neglected is translating from that universe to our own and it involves a number, which we’ll call B.
Bayesian reasoning is powerful. It’s used by major pharmaceutical companies internally, yet it isn’t often used to filter obvious baloney, or fast track promising treatments at the federal level. That might be changing as the FDA has released updated guidance on its use this year.
To get an idea of how the FDA wants to use Bayesian reasoning, suppose a drug works in adults and the FDA is considering approving it in children. Statistically, you can take the previous effect size of the drug in adults, discount it in some way to account for the differences between adults and children and use that number (B) to directly calculate future success.
Why now?
Bayesian methods have never achieved organic growth. In the middle of the last century, the methods that became dominant were ones that experts preferred. We know this because experts borrowed parts from existing methods in a way that made publishing papers easier, and that version of the method proliferated without statisticians explicitly endorsing it. In fact, the method that dominated has been criticized for decades and continues to be used in the vast majority of research.
The method can be understood with a toy example. (This example unfortunately has a villain in it, which is the way humans understand math.)
Suppose an expert wants to report a finding but doesn’t really want to give away how likely it is to be true. If C is how likely it is to be true, an expert could simply report A where A × B = C and you would never know what C is unless you’re also an expert who could provide the missing number B.
It can’t be that simple, though, right? No, of course not. We can’t just regulate the disclosure of B and force researchers to give up a number. There’s a simple proof of this that comes from Wall Street. Investors spend billions of dollars trying to get the number from experts in the field from different drug trials. The pharmaceutical company GlaxoSmithKline elicits B from its experts by getting them to estimate all at once, averaging their estimates together. Even then, it’s not so easy. Facilitators are told to try to help experts to give their real estimate, not the one they would like to be true.
This might be what happens at FDA and it’s a real concern. Bayesians have been fighting with Frequentists for a century and this is one area where the Frequentists have a point. At the edge of human knowledge you can’t verify expert opinion without becoming one yourself, particularly when it comes to divulging a number. Simultaneous confession by experts working at the same pharmaceutical company might be the best we can do.
So, trying to force experts to formalize what they know might not better inform the public. However, in an age where the general public distrusts experts, this methodology is hard to ignore. It's particularly hard when there are cases where B is not revealed by experts but by common sense or counting. (One in a billion proteins in your gut really is a probability of one in a billion.) There’s an obvious information asymmetry between experts and the public that, at an aggregate level, has been maintained instead of broken down. Surely some form of asking for the number couldn’t hurt.



As a doctor I deal with this coming up constantly. Patients want “tests” but don’t understand pre test probability. They can perceive the denial of testing when pre test probability is low as some sort of conspiratorial oppression. I often also have trouble getting trainees comfortable with commenting on different degrees of uncertainty.
Even medical school graduates constantly get biostatistical questions wrong on the boards.
And all doctors are affected by the availability bias and wind up doing testing that’s overwhelmingly negative. Hence why healthcare is expensive.