GPT-3 and In-Context Learning
By Satwik ยท March 13, 2026
GPT-3 demonstrated that a sufficiently large language model can perform new tasks from a few examples placed in its prompt, with no weight updates. This in-context learning ability reorganized how the field builds systems, and opened an attack surface that persists today.
The result that changed the interface
GPT-3, released by OpenAI in 2020, was a 175 billion parameter autoregressive Transformer trained on a large mixture of web text, books, and Wikipedia. Architecturally it was close to GPT-2 scaled up more than a hundredfold. The paper's title, "Language Models are Few-Shot Learners," names its central claim: at this scale the model could be steered to perform tasks simply by being shown a handful of examples in its input, then asked to continue the pattern.
This is in-context learning. You do not fine-tune. You place a description and a few input-output demonstrations in the prompt, append a new input, and the model completes it in the demonstrated style. Translation, arithmetic, unscrambling words, answering trivia, all could be elicited by conditioning alone. The model's weights never change; the "learning" happens transiently within a single forward pass over the context.
Why it works, mechanistically
The honest position is that in-context learning is still not fully explained, but a coherent picture has emerged. During pretraining the model sees enormous numbers of documents that implicitly contain tasks: lists, question-answer pairs, translated sentences, tables. To predict the next token well across this diversity, the model benefits from inferring, on the fly, what kind of document it is currently in and what regularity governs it. In-context learning is the deployment of that inference machinery: the demonstrations tell the model which latent task the current document instantiates, and it continues consistently.
Research since has shown that Transformers can implement, inside their forward pass, procedures that resemble gradient descent or nearest-neighbor lookup over the in-context examples, and that induction-like attention patterns, copying and completing repeated structure, are a substrate for this behavior. The demonstrations act less as training data and more as a specification that locates the right behavior in a space the model already contains.
Crucially, the ability strengthened sharply with scale. Small models barely benefit from examples; large ones benefit a great deal. In-context learning was one of the first capabilities described as emergent, present at large scale and nearly absent at small, though the sharpness of such transitions later became a subject of careful debate about whether it reflects the model or the metric.
Why it mattered
The practical consequence was a new development paradigm. Instead of collecting a labeled dataset and training a model per task, you could prototype a task in minutes by writing a prompt against a single general model exposed through an API. This lowered the barrier to building language applications dramatically and shifted enormous value onto whoever controlled the largest models. Prompt engineering became a discipline. The API-as-product model, and the platform dynamics around it, follow directly from the fact that one model could do many things without retraining.
It also reframed the research agenda. If capability scales predictably with size and unlocks qualitatively new behaviors, then scale itself is a research direction. GPT-3 was the empirical anchor for the scaling bet that defined the next several years.
The security angle
In-context learning created a durable security problem by erasing the boundary between instructions and data. A model that will follow patterns and directives appearing in its context will also follow directives that an attacker manages to place there. This is the root of prompt injection: if untrusted content, a web page the model summarizes, a document it reads, an email it processes, contains text like "ignore previous instructions and do the following," the model may comply, because at the level of tokens it cannot cleanly distinguish the developer's instructions from adversarial data flowing through the same channel.
This is not a bug to be patched so much as a structural property of the mechanism that makes the model useful. As models were connected to tools, browsers, and other systems, prompt injection matured from a curiosity into the central attack pattern for agentic deployments, since a compromised input can now trigger real actions rather than just bad text.
GPT-3 also sharpened the capability-overhang concern. If capabilities appear suddenly at scale rather than arriving gradually, then a deployed model may harbor abilities its builders have not measured, latent until the right prompt or the right amount of context surfaces them. Evaluation becomes a search problem over an enormous prompt space, and absence of evidence for a capability is weak evidence of its absence. That asymmetry, easy to elicit, hard to rule out, remains one of the hardest problems in frontier model assessment, and GPT-3 is where it became concrete.