In-Context Learning as a Phenomenon

By Satwik · February 1, 2026

In-context learning (ICL) is the observation that a trained language model can absorb a task from examples in its prompt and apply it to a new query, without any weight updates. GPT-3 made this visible at scale, and through 2020-2021 it moved from a curiosity to a subject of study in its own right. The puzzle is that nothing in the training objective explicitly asks for this. The model is only trained to predict the next token, yet it behaves as if it were performing a lightweight adaptation at inference time.

Several framings emerged. One view treats ICL as implicit Bayesian inference: the prompt selects among latent "tasks" the model learned from its heterogeneous training distribution. Another line, developed more fully in later years, argues that attention layers can implement optimization-like updates internally, so the forward pass approximates learning from the in-context examples.

Why it mattered

ICL is the mechanism behind prompting-based deployment. Understanding it is understanding how these systems actually generalize, which bears on when they will and will not behave.

Reading angle

For security, ICL is double-edged. The same flexibility that lets a helpful instruction steer the model lets an adversarial instruction do so, since both arrive through the same channel. ICL also means behavior depends sharply on context ordering, formatting, and example selection, so evaluation on one prompt template gives limited assurance about another. Treat ICL as a runtime attack surface, not just a convenience feature, and test models under adversarially chosen contexts rather than benign ones.