The mystery is solved. NIH responded to my FOIA request by returning emails only if they contain the project’s grant number. This technique effectively defeats all FOIA requests since NIH can simply pretend not to know how humans talk.
Getting informal advice from a lawyer, and telling NIH I had, helped immensely.
The subject of the FOIA request, covered previously, is called Community for Rigor or C4R. It is an NIH project that supports the production of training materials on research rigor for schools and labs to use for free.
NIH anticipated the project would be difficult. Rigor costs researchers money, in future funding and in reputation. So NIH stipulated that they the funder would have an ongoing leadership role on the project. This is unusual.
The hole in NIH’s logic was that the professors might not follow through and produce material on these costly rigorous practices, or NIH might not have the fortitude to accuse their own grantees of not following through. Perhaps the average grantee at Community for Rigor would be like the “average scientist” referred to in the C4R proposal who “fails to consistently follow the relevant principles.”
It was readily apparent that this nightmare scenario was coming true at C4R’s first annual conference, and in videos of the conference that C4R removed — apparently in response to feedback — and I’ve restored here.
How cynical does a person have to be to anticipate that a project that prides itself, and in fact was named after, grassroots governance would take down videos after criticism? Not cynical enough it turns out. I didn’t expect NIH would use the grant number to search for C4R emails. As if the director of Research Quality discusses the culmination of his twenty years of effort on research quality as UC2NS12836.
Of course, the NIH FOIA office knew this because the matching emails referred to the project as “Grassroots Rigor” and “Community for Rigor (C4R)” and because I specified all of those names.
The other ironies are too numerous to list here.
The point is that NIH doesn’t seem to care about FOIA. Six months of correspondence had made that clear already.
I emailed eight officials connected to the project and asked them if they thought the FOIA response reflects the request and whether they’d discussed Community for Rigor in emails that weren’t included. None responded.
The Watchmen Problem
The other point is that metascience keeps getting more meta. If researchers at one level of “watchmen” aren’t rigorous, it’s up to watchers of the watchmen and on and on. As you go up the chain, watchmen become fewer and fewer, and unless actual legal repercussions are involved, less powerful.
Science does self-correct to some degree. It would be obtuse to ignore this. However, the degree to which it patently does not correct itself can be extreme. Some “meta” failures from the last few years:
Fabricated data in research about honesty. You can’t make this stuff up. Or, can you? (July, 2023)
The Distortions of Joan Donovan Is a world-famous misinformation expert spreading misinformation (June, 2024)
This Study Was Hailed as a Win for Science Reform. Now It’s Being Retracted (September, 2024)
The Business-School Scandal That Just Keeps Getting Bigger The rot runs deeper than almost anyone has guessed. (November, 2024)
NIH is set to spend a lot of money on metascience. I wholeheartedly support spending money on metascience. In fact, if all the funding results in a bunch of Community for Rigor projects, I wouldn’t be opposed. It’s better than nothing and some of C4R is great. However, we have to know the effect of paying the same people who caused the replication crisis, the average researcher, more money. If it’s all for nothing because the remaining incentives defeat the purpose, we should know that and be cynical enough to anticipate it.
A thank you note
With overwhelming gratitude, I have to thank the Public Citizen Litigation Group and Adina Rosenbaum for helping me solve this by telling me the right questions to ask the FOIA office. I told her I hoped this reminds her this work is important and I meant it. Contacting Public Citizen was incredibly helpful and probably decisive.