It always ends with numbers
How NIH’s project to save neuroscience education is switching sides
Science reform is hard. Even the largest funder in the world struggles with it. But one of the most expensive projects on science reform, NIH’s Community for Rigor has been a success in one way. It showed that when you try to take on what it called “the average scientist,” the average scientist fights back. Average scientists work their way into your project. Average scientists don’t like feedback. Average scientists really don’t like you insulting science. What the $16 million project has shown is something we already knew: questionable research is too common. It’s too average. Each time you look at Community for Rigor, the project is showing in a new way that it’s easier to join the mob than to push back.
In another stunning act of normalcy, the project doesn’t seem to want its emails released. The grantees claimed theirs were proprietary. (NIH can’t do this for its own emails.) The shocking part is how you can really dive in anywhere into science and you eventually get to the same place: fudging the numbers. From my long running FOIA request:
February 25, 2026
Valery,
Have you had any luck with your investigation? Normally I would send you a blog post for comment. This time the email is the blog post. Please let me know if there’s anything you don’t want published in your reply.
To recap, I was told my FOIA request for emails about NIH’s Community for Rigor project was returning too many records to be processed. This is because the short name for it, “C4R” was matching what the FOIA office called “junk.” About a month later, you sent the number of records for various searches. The number of records for the term “C4R” in conjunction with other search terms was lower than “C4R” alone. I told you this was impossible by normal search term logic and you said you had noticed this already and were investigating. That was two weeks ago.
The format of the numbers was different for one of the search results as if someone had copied some numbers directly and typed one by hand. I think most people would conclude that the likely explanation is that someone wrote down the wrong information.
To further recap, Community for Rigor is one of the largest metascience projects ever funded at around 16 million dollars. The goal was to produce training material on rigorous scientific practices. It was designed so that NIH would be co-managing the project with the principal investigator at the University of Pennsylvania.
To most people who have had direct contact with science reform, it should have been clear that the project would be a failure. I think that NIH, and the eventual head of the project, Konrad Kording were finding out in real time how big the scandal in science is. Current NIH Director Jay Bhattacharya said recently that researchers might not find doing the rigorous practice of replication glamorous and may avoid this kind of project, even if it’s funded. All of these people were no stranger to problems in science, but maybe they didn’t know how ubiquitous and difficult these problems are.
The funding director Walter Koroshetz once implied some NIH grantees are “crooks.” Kording wrote about how researchers can’t understand the brain because they can’t even understand a microprocessor. One of the grantees under this project is famous for writing a book called “Calling Bullshit.” And Bhattacharya has had almost the entire scientific community denounce him. Maybe these people were aware there are problems, but they’ve never had to get researchers to actually follow rigorous practices that might be harmful to the researcher’s career. It’s easy to talk about abstractly. Getting a particular person to follow through and undermine their own reputation, or their employer’s, is very difficult.
The purpose is to say, as I’ve said before, you are not the story. I don’t really care if even the worst interpretation of this email chain is true, that the FOIA office is falsifying records. As you probably know, it would cost me thousands of dollars just to threaten to sue. And on the scale of NIH FOIA scandals, this one is tiny. Two presidents and two congresses have tried to get NIH to release COVID origin records and often failed. (I can only speculate on what’s in them, but for some reason, NIH has not been cooperative.) I spoke with the person who would know best on this issue, and he told me I’m getting “the runaround” from NIH.
What I do care about is what NIH has said it cares about, that the average scientist doesn’t know how to apply the scientific method. This wording, “the average scientist,” is from the C4R funding proposal. What NIH, Konrad Kording, Walter Koroshetz, maybe even Jay Bhattacharya found out is that the best scientists handpicked by NIH don’t know how to apply the scientific method, or don’t want to.
Case in point, the slide from the Community for Rigor conference that started this is categorically wrong:
The image doesn’t depict rigorous practices and project leaders at NIH know this. When the NIH project leaders give talks, they present this version of the scientific lifecycle:
What the image above depicts is the head of a lab doing what has always worked: have a stream of grad students and postdocs attempt to farm data sets for publishable results. First one wins. Don’t worry what stage of the process you’re in. They’re all “on ramps” and you can always “revise your hypothesis.” The slide is what has, and perhaps should, end your career. It ended the career of USDA leader Brian Wansink who was just doing what everyone else was doing. It shouldn’t be part of the handpicked project to educate researchers on rigorous practices.
After I emailed Community for Rigor about this and similar issues throughout the conference videos, the videos were removed from youtube without explanation.
We’ve known about how common poor practices are for a while. The reason why the scandal didn’t resolve itself is because everyone is doing it. There weren’t bad apples to weed out because the bad apple view of science is wrong.
I hope you and my readers can tolerate a little grandstanding here. My point is that I don’t want to get involved in every nook and cranny of this story, and that is where this conversation is headed. I can’t figure out if the discrepancy is due to you, or Lauren, or the central search department. Common sense says that you’re just the latest people who don’t want damage to your reputation or to science’s reputation and you’re not going to return the records for some reason. All I can do is keep asking and underscoring how important it is that you follow through.
However, the longer this goes on, the clearer it will be that the problem is the average scientist just as NIH believes by funding this project. This could go on for another year, and really it’s money well spent because NIH did an experiment by funding this project and these are the results. Hidden videos, hidden materials, and of course emails.
Please let me know about the results of your investigation and whether or not you have an estimate for completing the request. In case it wasn’t clear, I am not blaming you or Lauren for the delays. I have great respect for all of the people I listed above and I wouldn’t blame them either. You’re welcome to ask to be anonymous, and if you want to hand me off to someone else, I’m happy to continue with them.
Best,
Alex
The numbers
The NIH FOIA office usually changes the subject without commenting on previous implausibilities and they did so in this case. They are most certainly writing down the wrong information, though. A few days later, I got a reply that included statistics for the same searches again:
The numbers had been changed and the formatting was consistent this time. There was no mention of the changes in the email. The impossibility of adding more “OR” terms and getting fewer results had been fixed, so to speak, due to the deletion of 517 emails from program officer Devon Crawford’s total.
Did someone delete 517 emails from Dr. Crawford’s mailbox? It seems unlikely since the other search should have been reduced by 517 too. After a year of this FOIA request and eight months of writing about it, those emails disappeared just as I complained that the numbers don’t make logical sense. (She and the FOIA office didn’t respond to a request for comment.)
There are a few other inconsistencies in the results. Since the numbers seem to conform to whatever the FOIA office wants to argue at that moment, I figure it’s best not to tell them. I’m starting to understand the typical response to the crisis in science. Maybe most people figure it’s best to leave science alone and wait for it to dig itself deeper.