Metascience Reading List

13 great papers and 7 essential ones

Essential papers are in bold.

What is wrong

• Simmons, Joseph P., Leif D. Nelson, and Uri Simonsohn. 2011. “False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant.” Psychological Science 22 (11): 1359–66. https://doi.org/10.1177/0956797611417632.

• Gelman, Andrew, and Eric Loken. "The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time." Department of Statistics, Columbia University 348, no. 1-17 (2013): 3.

• Silberzahn, R., E. L. Uhlmann, D. P. Martin, P. Anselmi, F. Aust, E. Awtrey, Bahník, et al. 2018. “Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results.” 1 (3): 337–56. https://doi.org/10.1177/2515245917747646.

Prevalence

• Ioannidis, John P.A. 2005. “Why Most Published Research Findings Are False.” In Getting to Good: Research Integrity in the Biomedical Sciences, 2–8. Springer International Publishing. https://doi.org/10.1371/journal.pmed.0020124.

• Manrai, Arjun K., Gaurav Bhatia, Judith Strymish, Isaac S. Kohane, and Sachin H. Jain. 2014. “Medicine’s Uncomfortable Relationship With Math: Calculating Positive Predictive Value.” JAMA Internal Medicine 174 (6): 991. https://doi.org/10.1001/JAMAINTERNMED.2014.1059.

• Aarts, Alexander A., Joanna E. Anderson, Christopher J. Anderson, Peter R. Attridge, Angela Attwood, Jordan Axt, Molly Babel, et al. 2015. “Estimating the Reproducibility of Psychological Science.” Science 349 (6251). https://doi.org/10.1126/science.aac4716.

• Baker, Monya. 2016. “1,500 Scientists Lift the Lid on Reproducibility.” Nature 533 (7604): 452–54. https://doi.org/10.1038/533452a.

• Errington, Timothy M., Maya Mathur, Courtney K. Soderberg, Alexandria Denis, Nicole Perfito, Elizabeth Iorns, and Brian A. Nosek. 2021. “Investigating the Replicability of Preclinical Cancer Biology.” ELife 10 (December). https://doi.org/10.7554/ELIFE.71601.

• Carlisle, J. B. 2021. “False Individual Patient Data and Zombie Randomised Controlled Trials Submitted to Anaesthesia.” Anaesthesia 76 (4): 472–79. https://doi.org/10.1111/ANAE.15263.

Solutions

• Simonsohn, Uri, Leif D. Nelson, and Joseph P. Simmons. 2014. “P-Curve: A Key to the File-Drawer.” Journal of Experimental Psychology. General 143 (2): 534–47. https://doi.org/10.1037/A0033242.

• Steegen, Sara, Francis Tuerlinckx, Andrew Gelman, and Wolf Vanpaemel. 2016. “Increasing Transparency Through a Multiverse Analysis.” Perspectives on Psychological Science 11 (5): 702–12. https://psycnet.apa.org/doi/10.1177/1745691616658637.

• Leek, Jeff, Blakeley B. McShane, Andrew Gelman, David Colquhoun, Michèle B. Nuijten, and Steven N. Goodman. 2017. “Five Ways to Fix Statistics.” Nature 2021 551:7682 551 (7682): 557–59. https://doi.org/10.1038/d41586-017-07522-z.

• Chambers, Christopher D., and Loukia Tzavella. 2021. “The Past, Present and Future of Registered Reports.” Nature Human Behaviour 2021 6:1 6 (1): 29–42. https://doi.org/10.1038/s41562-021-01193-7.

• Simonsohn, Uri, Joseph P. Simmons, and Leif D. Nelson. 2020. “Specification Curve Analysis.” Nature Human Behaviour 2020 4:11 4 (11): 1208–14. https://doi.org/10.1038/s41562-020-0912-z.

• Lakens, Daniël. 2020. “Pandemic Researchers - Recruit Your Own Best Critics.” Nature 581 (7807): 121. https://doi.org/10.1038/D41586-020-01392-8.

Pre-2005

• Langmuir, Irving. 1953. “Pathological Science.” 32 (5): 11–17. https://doi.org/10.1080/08956308.1989.11670607.

• Platt, John R. 1964. “Strong Inference.” Science Science 146 (3642): 347–53. https://pages.cs.wisc.edu/~markhill/science64_strong_inference.pdf.

• Cohen, Jacob. 1994. “The Earth Is Round (p < .05).” American Psychologist 49 (12): 997–1003. https://doi.org/10.1037/0003-066X.49.12.997.

History

• Gelman, Andrew. "What has happened down here is the winds have changed." Statistical Modeling, Causal Inference, and Social Science (2016). https://statmodeling.stat.columbia.edu/2016/09/21/what-has-happened-down-here-is-the-winds-have-changed/.

• Ioannidis, John P. A. 2024. “What Meta-Research Has Taught Us about Research and Changes to Research Practices.” Journal of Economic Surveys, October. https://doi.org/10.1111/JOES.12666.