The Replication Crisis Decoded

Most published findings fail to replicate. Psychology: 40% replicate. Cancer biology: 10-25%. This isn't a scandal of individual misconduct—it's the predictable result of how scientific incentives are structured. Understanding the crisis reveals how science actually works.

In 2015, the Open Science Collaboration attempted to replicate 100 psychology studies. 36% showed clearly significant results. The "replication crisis" entered public consciousness.

But this isn't news to anyone who understood the incentive structure. The crisis was inevitable.

The Mechanism

Publication Bias

Journals publish positive results. "We found an effect" publishes. "We found nothing" doesn't. This creates a filter: only "significant" findings enter the literature.

P-Hacking

Statistical significance (p < 0.05) is a threshold. Researchers have many choices: which variables to include, which outliers to exclude, which analyses to run. With enough flexibility, significance can usually be achieved.

This isn't necessarily fraud. "Researcher degrees of freedom" are often exercised unconsciously, guided by which results look promising.

Underpowered Studies

Many studies have small samples—insufficient statistical power to reliably detect real effects. Small samples produce noisy estimates. Noise + publication bias = inflated effect sizes that don't replicate.

File Drawer Problem

Studies that don't find effects go in the file drawer—unpublished. The published literature is a biased sample: disproportionately positive, disproportionately significant.

Why It Persists

These problems are well-known. Why haven't they been fixed?

Incentives

Researchers are rewarded for publications, not replications. Novel findings advance careers; confirmations don't. The incentive is to produce publishable results, and publishable means significant and novel.

Journals

Journals want readers. Novel, surprising findings attract readers. "We replicated prior work" doesn't. Journal incentives select for novelty over reliability.

Institutions

Hiring and tenure committees count publications. Quality assessment is hard; quantity counting is easy. The system rewards production, not truth-tracking.

No individual is necessarily acting badly. The system produces bad outcomes from individually rational behavior.

What the Crisis Reveals

Science Is Institutional

Science isn't just "the scientific method." It's institutions, incentives, norms, funding structures. The crisis shows what happens when institutional design conflicts with epistemic goals.

Goodhart's Law

Publications were a proxy for scientific contribution. When publications became a target, they decoupled from contribution. Classic Goodhart.

Selection Effects

What publishes isn't random. It's selected for publishability. Understanding this selection changes how you should update on published findings.

What It Means for You

If you consume scientific findings (health recommendations, psychology insights, etc.):

• Single studies are weak evidence. Most don't replicate. Wait for convergent findings.
• Effect sizes shrink over time. Initial exciting findings usually regress toward smaller effects.
• Novel/surprising should trigger skepticism. Novelty is selected for; accuracy isn't.
• Replication > original finding. When available, trust meta-analyses and replications more than original studies.

Reforms

The field is responding:

• Pre-registration: Commit to hypothesis before data collection, preventing p-hacking.
• Registered reports: Journals commit to publish based on methods, not results.
• Open data/code: Allow verification and reanalysis.
• Replication incentives: Some journals and funders now reward replication.

Progress is real but slow. Institutional change lags awareness.

The Deeper Lesson

The replication crisis isn't primarily about bad actors. It's about systems design.

Design incentives for truth-tracking and you get more truth. Design incentives for publication and you get more publications. The system optimizes for what it selects for.

This generalizes beyond science. Any institution claiming to serve one goal while incentivizing another will drift toward what's actually incentivized.

— Decoded by DECODER