DistilBERT and Distillation

By Satwik · January 18, 2026

DistilBERT (Sanh et al., 2019) applied knowledge distillation to BERT, producing a model roughly 40 percent smaller and significantly faster while retaining most of the original's performance. It made the pretraining era practical for deployment where full BERT was too heavy.

How distillation works

The idea, tracing to Hinton et al., is to train a small student model to match a large teacher's output distribution, not just the hard labels. The teacher's soft probabilities carry "dark knowledge" about how classes relate, and the student learns a smoother, richer target than one-hot labels provide. DistilBERT combined this distillation loss with the masked language modeling loss and a cosine embedding alignment, distilling during pretraining rather than only at the end.

The result was a compact model that kept around 97 percent of BERT's language-understanding performance on common benchmarks at a fraction of the inference cost. Compression stopped being a lossy afterthought and became part of the training story.

Reading it now, DistilBERT marks the field maturing from "can we do it" to "can we ship it." Distillation is now everywhere in efficient deployment.

For security, distillation cuts both ways. It is a legitimate compression tool, but the same mechanism underlies model extraction attacks: an adversary with only query access to a deployed model can distill a functional copy, stealing intellectual property and, worse, building a local surrogate to craft transferable adversarial examples. DistilBERT's benign technique and that attack are the same idea pointed in different directions.