Transformers Architecture

The problem it solved

Before 2017, the standard way to handle sequences, sentences, audio, time series, was the recurrent neural network. An RNN reads one token at a time, updates an internal state, and moves on. This design has two painful properties. It is sequential, so you cannot process token five until you have processed tokens one through four, which wastes the parallel power of modern GPUs. And its memory fades: by the time it reaches the end of a long sentence, the influence of the first words has been diluted through dozens of updates.

The transformer threw out recurrence entirely. Instead of reading left to right, it looks at the whole sequence at once and lets every position gather information directly from every other position. That mechanism is called attention, and it is the entire reason the architecture works.

The 2017 paper

"Attention Is All You Need" was published on arXiv on June 12, 2017, by eight researchers at Google: Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin. It was presented at the NeurIPS conference that December. The paper's target was machine translation, and its results were strong: 28.4 BLEU on English-to-German and 41.8 on English-to-French, both state of the art at the time, trained in a fraction of the time the previous best models needed.

Nobody at the time predicted what would follow. Within a year, researchers had applied the architecture to language modeling in general, and within five years it underpinned nearly every major AI system in language, then vision, then audio. As of 2026 the paper is among the most cited in the history of machine learning.

Encoder and decoder

The original transformer has two halves. The encoder reads the input sequence, a sentence in English, and builds a rich representation of it. The decoder generates the output sequence, the French translation, one token at a time, consulting the encoder's representation as it goes.

Each half is a stack of identical layers, six in the original paper. Every encoder layer has two sub-layers: a multi-head self-attention mechanism that lets each position look at all the others, and a feedforward network applied to each position independently. Every decoder layer adds a third sub-layer that attends to the encoder's output. Around each sub-layer sit two ingredients that make deep stacks trainable: a residual connection that adds the sub-layer's input to its output, and layer normalization that keeps the numbers in a stable range.

Modern language models mostly use only the decoder half. GPT and its descendants generate text one token at a time, so they need the decoder's machinery but not a separate encoder. BERT, used for understanding rather than generation, uses only the encoder half. The full encoder-decoder survives mainly in translation systems.

Self-attention, concretely

Self-attention is easiest to see as a weighted mixing step. Each position in the sequence produces three vectors: a query (what am I looking for?), a key (what do I contain?), and a value (what will I pass along if chosen?). Every query is compared against every key, producing a score for each pair. The scores are normalized into weights that sum to one, and each position's output is the weighted mix of all the values.

The comparison is a scaled dot product:

```

Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) V

```

The division by the square root of the key dimension keeps the scores in a range where the softmax stays sensitive. Without it, large dot products would push the softmax into regions where its gradients nearly vanish and learning would stall.

Multi-head attention runs this process several times in parallel with different learned projections, then concatenates the results. The intuition is that different heads can track different kinds of relationships: one head might follow grammar, linking verbs to their subjects, while another tracks which pronouns refer to which nouns. In practice, heads specialize in ways that are messier than this, but the capacity to track many relationship types at once is real.

Why order still matters

Attention by itself does not know about order. If you shuffled the input tokens, the attention computation would produce the same mixed outputs in shuffled positions. Since word order carries meaning, the transformer adds positional information to each token's embedding before the first layer. The original paper used fixed sinusoidal patterns of different frequencies; later models mostly learn the positions as parameters. Either way, the network gets to see both what each token is and where it sits.

Why it scaled

The transformer's decisive advantage is that it parallelizes. Computing attention for all positions happens in one batch of matrix multiplications, which is exactly what GPUs do fast. Training on a billion words no longer meant a billion sequential steps. It also handles long-range dependencies directly: the first and last words of a paragraph are one attention step apart, not fifty recurrent steps apart, so the signal between them stays strong.

The price is computation that grows with the square of the sequence length, since every position compares against every other. This quadratic cost is why context windows are limited and why long sequences are expensive. It is also the problem that a whole subfield of efficient-attention research tries to solve.

The family it spawned

The transformer turned out to be a general sequence engine, not a translation trick. BERT (2018) showed the encoder half could learn deep language understanding from unlabeled text. GPT-1 (2018) showed the decoder half could generate fluent text the same way. GPT-2 (2019) and GPT-3 (2020) showed that scaling the same architecture produced qualitatively new abilities. Vision transformers applied the same idea to image patches. The architecture has barely changed since 2017. Nearly everything since has been scale, data, and training method.