Research Notes

Sequence-to-Sequence Learning

By Satwik · January 9, 2026

Sutskever, Vinyals, and Le's 2014 sequence-to-sequence paper established the encoder-decoder template that framed nearly everything that followed. An LSTM reads the input sequence into a fixed representation, and a second LSTM generates the output sequence one token at a time, conditioned on that representation and its own prior outputs.

The result mattered because it showed a general-purpose way to map variable-length inputs to variable-length outputs with a single trainable model, no task-specific pipeline required. Machine translation was the demonstration, but the framing applied to summarization, parsing, and dialogue.

A useful trick

The authors found that reversing the source sequence improved results, by shortening the dependency between early source tokens and early target tokens. That detail is a small monument to the fragility of long-range learning in pure recurrence, and it helps explain why attention arrived so soon after.

Reading it now, seq2seq is the scaffold on which attention and then the Transformer were built. The autoregressive decoder it popularized is still exactly how modern language models generate text.

For security, the autoregressive framing is quietly important. Generation conditioned on prior tokens is what makes prompt injection and jailbreaks possible at all: the model cannot cleanly separate instructions it should follow from content it should merely process, because both arrive as the same token stream. That architectural choice, made for good reasons in 2014, still shapes our threat model today.