BERT

By Satwik · January 14, 2026

BERT (Devlin et al., 2018) is the paper that turned pretraining from a promising direction into the default. Its central move was bidirectional pretraining through a masked language modeling objective: randomly hide tokens and train the model to reconstruct them using context from both sides at once. Earlier language models read strictly left to right; BERT read the whole sentence.

Why it mattered

Alongside masked language modeling, BERT trained on next-sentence prediction and, crucially, fine-tuned the entire pretrained Transformer for each downstream task rather than treating it as a fixed feature extractor. The payoff was immediate and broad: state-of-the-art results across a wide range of benchmarks, often by large margins, from a single recipe. Within months, "just fine-tune BERT" became the sensible baseline for most NLP problems.

BERT also normalized the two-stage mental model that still organizes the field: expensive general pretraining done once, cheap task-specific fine-tuning done many times. That economic structure reshaped who could build capable systems.

Reading it now, BERT is the hinge of the whole era. It made deep bidirectional Transformers ordinary infrastructure.

For security, BERT crystallized the supply-chain question. Millions of downstream systems inherited weights from a handful of pretrained checkpoints, which means any bias, backdoor, or spurious correlation baked into that base propagates widely and silently. Research on weight poisoning of pretrained models followed directly from this reality. When everyone builds on the same foundation, the foundation becomes the target.