Research Notes

ELMo and Contextual Embeddings

By Satwik · January 11, 2026

ELMo (Peters et al., 2018) broke the one-word-one-vector assumption. Instead of a static lookup, it derived each word's representation from a bidirectional LSTM language model, so the same word received different vectors depending on the sentence around it. "Play" in a theater review and "play" in a sports report finally lived at different points.

Why it mattered

The word "Embeddings from Language Models" is the whole thesis: representations should come from a model trained to predict language, and those representations should be contextual. ELMo delivered consistent gains across a broad set of NLP benchmarks simply by feeding its contextual vectors into existing task models. It was the moment the field saw that deep language-model pretraining, not just word vectors, was the real prize.

ELMo still used a feature-based approach, producing vectors that other models consumed, rather than fine-tuning a single network end to end. BERT would soon show that fine-tuning the whole pretrained model worked even better. But ELMo established the direction of travel decisively.

Reading it now, ELMo marks the pivot from static to dynamic meaning, and from shallow transfer to deep transfer. For our purposes it also raises the stakes on provenance: once representations come from a large pretrained language model, everything that model absorbed, including its biases and its blind spots, rides along into every downstream task. The attack surface stopped being a word table and became a whole learned model.