Great article - not only about the dangers of not statistically accounting for multiple comparisons, but also about the dangers of believing news stories derived from such research!
This was a sharp read, and the spaghetti on the floor is an image that will stick. Your point that the threshold is static while the number of tests is flexible is the part that does the most work, because it explains how a paper can clear genome-wide significance and still not earn its headline.
Your coherence check complements that. A correction tells you how many tests were run, but the pattern across related outcomes tells you whether the hits behave like a signal. A real underlying tendency should show up as a gradient across neighbouring variables: suspended and expelled, father imprisoned and self imprisoned, drunk driving and arrest. Hits that land on one item and skip its siblings look more like the floor than the wall.
That makes the incoherence evidence in its own right, even when nobody can reconstruct the true test count across 138 authors. It also fits your closing line: a finding predictable from sample size alone, yet unpredictable across its own related variables, is telling you which of the two is doing the work.
I was in retail just as they were getting real customer data through loyalty programmes, etc. They were always quite convinced there was a golden behavioural nugget somewhere in the database that could be monetized.
The difference is that if retailers consider there's a correlation between owning pedigree dogs and a liking for malt whiskey, they can run the promotion. If it works well, good. If not, stop it. They're not interested in a universal truth.
Great article - not only about the dangers of not statistically accounting for multiple comparisons, but also about the dangers of believing news stories derived from such research!
This research was inspired by the xkcd "paper ' jelly beans cause acne.
This was a sharp read, and the spaghetti on the floor is an image that will stick. Your point that the threshold is static while the number of tests is flexible is the part that does the most work, because it explains how a paper can clear genome-wide significance and still not earn its headline.
Your coherence check complements that. A correction tells you how many tests were run, but the pattern across related outcomes tells you whether the hits behave like a signal. A real underlying tendency should show up as a gradient across neighbouring variables: suspended and expelled, father imprisoned and self imprisoned, drunk driving and arrest. Hits that land on one item and skip its siblings look more like the floor than the wall.
That makes the incoherence evidence in its own right, even when nobody can reconstruct the true test count across 138 authors. It also fits your closing line: a finding predictable from sample size alone, yet unpredictable across its own related variables, is telling you which of the two is doing the work.
I was in retail just as they were getting real customer data through loyalty programmes, etc. They were always quite convinced there was a golden behavioural nugget somewhere in the database that could be monetized.
The difference is that if retailers consider there's a correlation between owning pedigree dogs and a liking for malt whiskey, they can run the promotion. If it works well, good. If not, stop it. They're not interested in a universal truth.