GLUE and SuperGLUE Benchmarks

By Satwik · January 25, 2026

GLUE (Wang et al., 2018) and its successor SuperGLUE (Wang et al., 2019) were the scoreboards of the pretraining era. GLUE bundled nine diverse language-understanding tasks, from entailment to sentiment to acceptability, into a single benchmark with a public leaderboard, giving the field a common yardstick just as BERT and its rivals appeared.

Why they mattered

A shared benchmark did real work: it turned scattered results into comparable ones and gave the pretraining wave a clear target to chase. The trouble was how fast the target fell. Within roughly a year, top models had reached and then exceeded human-baseline scores on GLUE, which prompted SuperGLUE, a deliberately harder set with more demanding reasoning and coreference tasks. SuperGLUE too was largely saturated soon after.

That rapid saturation is the real lesson. When a benchmark is beaten this quickly, it usually means the benchmark, not the underlying understanding, was the thing conquered. Models exploited surface statistical cues and annotation artifacts to score well without robust comprehension, a phenomenon documented across these datasets.

Reading them now, GLUE and SuperGLUE are essential history and a standing caution. For our work the caution is central: benchmark performance is a proxy, and proxies get gamed. A high score can reflect shortcut learning rather than the capability we care about, and the same gap lets a model appear safe or competent under evaluation while failing in deployment. Robust evaluation, adversarial and out-of-distribution, is the direct descendant of watching these leaderboards fall.