GPT-1

By Satwik · January 19, 2026

GPT-1 (Radford et al., 2018) proposed generative pretraining: train a Transformer decoder as a plain left-to-right language model on a large unlabeled corpus, then fine-tune it on downstream tasks with minimal architectural changes. It landed around the same time as ELMo and just before BERT, and it staked out the decoder-only, autoregressive branch of the family tree.

Why it mattered

The contribution was showing that a single generative objective, next-token prediction, produced representations that transferred well across diverse tasks: entailment, similarity, question answering, classification. Rather than designing a bespoke model per task, GPT-1 reused one pretrained network with task-specific input formatting and a light output head. This "task-agnostic pretraining, task-aware fine-tuning" framing was clean and general.

Where BERT went bidirectional for understanding, GPT stayed strictly autoregressive, which kept it able to generate fluent text. That choice looked like a modest tradeoff in 2018. It was in fact the foundation of everything the GPT line would later become, since scaling the same generative objective is what produced GPT-2, GPT-3, and beyond.

Reading it now, GPT-1 is the humble origin of a lineage that reshaped the field. For our notes, its lasting significance is that it committed to open-ended generation as the core capability. Generation is where the hardest safety problems live: a model that writes fluent text can also write disinformation, malware, or persuasive manipulation. GPT-1 was small enough that this was theoretical; the direction it set made it inevitable.