GPT-3 and Few-Shot Learning

By Satwik · January 29, 2026

GPT-3 (Brown et al., 2020) was the headline result that reframed how the field thought about language models. It was a dense autoregressive transformer scaled to 175 billion parameters, trained on a large web-and-books corpus with a simple next-token objective. The striking claim was not the size itself but the behavior it unlocked: the model could perform many tasks by conditioning on a handful of examples placed in its context window, with no gradient updates. This is "few-shot" or "in-context" learning.

Why it mattered

Before GPT-3, adapting a model to a new task meant fine-tuning weights on labeled data. GPT-3 showed that a single frozen model could translate, answer questions, do arithmetic, and unscramble words purely by prompting. Performance scaled smoothly with model size, which lent weight to the emerging scaling-laws narrative. It shifted practical NLP toward prompting rather than task-specific training pipelines.

Reading angle

For a security-minded reader, GPT-3 is the origin point of several later concerns. Once a model reads task instructions from its input, the boundary between "data" and "instruction" blurs, which is the conceptual seed of prompt injection. The model also memorized fragments of training data, raising extraction and privacy questions that later work made concrete. Capabilities were uneven and confidently wrong outputs were common, so downstream systems that trusted raw completions inherited a reliability problem. GPT-3 is best read not as a finished product but as the demonstration that scale plus a generic objective produces broadly useful, and broadly unpredictable, behavior.