1) Input token representations X are projected by the head's three learned matrices: Q = XW^Q, K = XW^K, V = XW^V, each into dimension d_k. 2) The head computes raw similarity scores QK^T and scales them by 1/√d_k to stabilize softmax gradients. 3) An optional mask (e.g. causal masking in the decoder) zeroes out disallowed positions. 4) A row-wise softmax produces the attention pattern — a weight distribution summing to 1. 5) The weights multiply the value vectors V, yielding the head output (a weighted sum of values). 6) The outputs of all h heads are concatenated and projected by W^O back to d_model, then added to the residual stream.
A single attention mechanism averages information within one representation subspace, which limits its ability to model many different types of token dependencies at once. Splitting attention into multiple independent heads lets each focus on a distinct relational pattern (e.g. positional adjacency, syntactic dependencies, coreference) without interfering with one another.
Three learned matrices W^Q, W^K, W^V projecting token representations into the head's subspace of dimension d_k.
Official
The head's core: softmax(QK^T/√d_k)V, producing the attention pattern and a weighted sum of values.
Official
The T×T attention weight matrix (post-softmax), interpreted in mechanistic analysis as the QK circuit.
Omitting the scaling produces large dot products that push softmax into vanishing-gradient regions.
A faulty mask lets a head see future tokens, leaking information during autoregressive training.
When d_model is not divisible by the number of heads, the head dimension is non-integer and tensor shapes mismatch.
The attention head is defined as an independent unit with its own Q/K/V projections (h=8, d_k=64).
Michel et al. show that many heads can be removed at inference with little performance loss.
The Transformer Circuits framework describes a head as an independent, additive operation — a foundation of mechanistic interpretability.
Olsson et al. identify specialized heads (prefix matching + copying) tied to the emergence of in-context learning.
Time complexity: O(n² · d_k). Space complexity: O(n² + n · d_k).
Computing and softmax-ing the T×T matrix grows quadratically with sequence length, the main bottleneck for long context.
Dimension of a single head's Q/K/V subspace; typically d_model/h (e.g. 64).
Number of parallel heads in an attention layer; sets how many independent subspaces exist.
Causal (decoder, autoregressive) vs bidirectional (encoder).
A standard head is dense — every pair of positions is scored (before any masking).
Heads are mutually independent and computed in parallel; during training all positions are processed at once. Autoregressive generation is sequential across tokens.
QK^T and (softmax)·V are dense matrix multiplications, ideal for tensor cores.
TPU matrix units handle matmul-based attention well.
