XLNet

By Satwik · January 23, 2026

XLNet (Yang et al., 2019) tried to get BERT's bidirectional context without BERT's downsides. Its key idea is permutation language modeling: instead of masking tokens, train an autoregressive model over all possible orderings of the sequence, so each position learns to attend to context from both directions across the expectation over permutations.

The motivation

BERT's masked language modeling has two quiet flaws. It introduces artificial [MASK] tokens that never appear at fine-tuning time, creating a pretrain-finetune mismatch, and it assumes the masked tokens are independent given the context, which is not true. XLNet's permutation objective is autoregressive, so it avoids the mask token entirely and models dependencies among predicted tokens properly, while still seeing bidirectional context. It also folded in the segment recurrence and relative positioning of Transformer-XL to handle longer context.

The result was state-of-the-art on many benchmarks at release, edging out BERT and prompting the training-budget scrutiny that RoBERTa would soon sharpen.

Reading it now, XLNet is an elegant answer to a real objection, even if the field largely converged on simpler masked models plus better training. Its value in these notes is conceptual clarity about what pretraining objectives actually assume. The pretrain-finetune mismatch it identified is a small instance of a general safety concern: models behave differently under conditions they did not see in training, and those distribution gaps are exactly where surprising, sometimes exploitable, behavior appears.