YaleCoursesThe Replication Crisis | Understanding Medical Research
YaleCourses
A psychic trick and four kinds of studies 0:00
The lecture opens with a trick where the presenter pretends to be psychic and guesses the number three, which some people in the audience happened to be thinking of. This shows how a positive result can happen purely by luck rather than by any real power. From here the lecture defines four types of study outcomes: a positive study of a true hypothesis, a negative study of a true hypothesis, a positive study of a false hypothesis, and a negative study of a false hypothesis. A positive study just means the data supports the hypothesis, not that the hypothesis is actually true, since luck can produce a positive result even when the underlying claim is false. Likewise a negative study does not prove a drug fails, since bad luck can hide a real effect.
Why false positives are dangerous 3:30
Positive studies attract headlines and excitement, while negative studies get almost no attention, even though both false negatives and false positives cause real harm. A false negative means a working treatment gets ignored, so patients miss out on something that could have saved lives. A false positive is worse, because it pushes doctors to give patients a drug that does nothing but still carries side effects, and it sends the whole field chasing a false lead. Replication exists to check whether an early result holds up. A 2005 review in the Journal of the American Medical Association looked at 49 highly cited studies and found seven were later contradicted, seven held up but with weaker effects, twenty replicated successfully, and eleven were never checked again. Replication is rare because it takes years, costs money nobody wants to fund, and brings no fame, since funders like the NIH reward innovation, not repeating someone else's work. The FDA is stricter, generally requiring two independent successful trials before approving a drug, precisely because getting lucky once is easy but getting lucky twice is not.
Medical reversal and the lesson to stay skeptical 9:30
Vinay Prasad and Adam Cifu coined the term medical reversal for when a better designed trial overturns accepted practice. Looking at 363 New England Journal of Medicine articles over ten years, they found forty percent reversed the standard of care, thirty eight percent confirmed it, and the rest were inconclusive. One example is intensive glucose lowering in type 2 diabetes, where doctors assumed lower blood sugar was always better until trials showed pushing it too low caused more heart attacks. Another is mite-proof mattress covers for asthma, once believed to help but shown to make no real difference when properly tested. The core takeaway is that a single study is almost never enough, since it can be lucky, unlucky, or badly designed, and replication is what actually moves science forward even though the system rewards new ideas over confirming old ones. The best defense against exciting medical breakthroughs is a healthy dose of skepticism.
AI-generated summary. It can be wrong or incomplete - check anything that matters against the original.
