Switch Transformer and Sparse MoE

By Satwik · February 9, 2026

The Switch Transformer (Fedus, Zoph, Shazeer, 2021) pushed mixture-of-experts (MoE) into the spotlight as a way to grow parameter count without growing per-token compute proportionally. In a dense transformer every token passes through the same feed-forward weights. In an MoE layer there are many expert feed-forward networks and a router that sends each token to a small number of them, in Switch's case exactly one, the "switch." Most parameters sit idle for any given token, so you get a very large model that is cheap to run per token.

Why it mattered

This decoupled model size from inference cost, letting researchers reach trillion-parameter scale while keeping FLOPs per token manageable. The paper contributed practical tricks, simplified routing, selective precision, and capacity factors, that made sparse training stable enough to be useful. MoE went from a promising idea to a serious tool for scaling.

Reading angle

Sparse routing changes the reliability picture in ways worth noting. Because behavior depends on which experts a token activates, and routing is input-dependent, the effective computation varies across inputs in a way dense models do not. Load-balancing losses are needed to stop the router collapsing onto a few experts, and routing decisions can be brittle. For an evaluation reader, this means capability can be unevenly distributed across the input space, and worst-case behavior may live in poorly served regions of the routing map. MoE is best read as a genuine efficiency breakthrough that also adds a new, somewhat opaque, dynamic component, the router, whose behavior is part of the system's overall trustworthiness.