RoPE
How it works
Query and key vectors in the attention mechanism are rotated by an angle proportional to the token position before computing the dot product. This makes attention between tokens depend on their relative distances rather than absolute positions.
Problem solved
Standard positional encoding (additive or sinusoidal) generalizes poorly to sequences longer than seen during training. RoPE encodes positions through matrix rotation, which naturally transfers to longer sequences.
Components
A set of frequencies θ_i = base^(-2i/d) (base default 10000) defining the rotation angle for each dimension pair. Low dimensions rotate quickly, high dimensions slowly, forming a multi-resolution position representation.
Official
A block-diagonal rotation matrix that rotates pairs of adjacent vector dimensions by angle m·θ_i, where m is the token position. Implemented efficiently as element-wise operations on pairs (x, y) → (x·cos − y·sin, x·sin + y·cos).
RoPE is applied only to query and key vectors (not to value V) before computing the attention dot product. This makes the attention score depend on the relative position m−n of two tokens.
Implementation
RoPE trained on sequences up to N tokens degrades for sequences >N without extrapolation techniques (Position Interpolation, NTK-aware scaling, YaRN, LongRoPE). Naive context extension leads to chaotic attention distributions.
At large positions the rotation angles become very small — float16/bfloat16 computation can cause numerical errors and loss of positional information.
RoPE rotates pairs of adjacent dimensions, so the attention head dimension must be even; an odd dimension requires special handling or partial RoPE.
Evolution
Su et al. propose encoding positions by rotating Q/K vectors, unifying absolute and relative position and enabling use with linear attention.
The 'Rotary Embeddings: A Relative Revolution' post and the use of RoPE in GPT-J and GPT-NeoX spread the method across decoder-only LLMs.
Adoption of RoPE in LLaMA, then Mistral, Qwen, Gemma, and DBRX establishes it as the default positional encoding in open LLMs.
Chen et al. show that scaling position indices (interpolation) extends the context window of RoPE-based models with minimal fine-tuning.
YaRN combines NTK-aware scaling with attention temperature correction, extending RoPE model context more efficiently than plain interpolation.
LongRoPE uses evolutionary search over non-uniform RoPE scaling factors, extending the context window to over 2 million tokens.
Hyperparameters (configurable axes)
Frequency base (default 10000). Increasing the base lengthens wavelengths and is the basis of NTK-aware scaling for context extension.
Fraction of the head dimension subjected to rotation. Full RoPE rotates 100% of dimensions; some models (GPT-NeoX) use partial RoPE (e.g. 25%).
The attention head dimension must be even because RoPE rotates dimension pairs.
Computational complexity
Time complexity: O(n · d). Space complexity: O(n · d).
Execution paradigm
RoPE is a deterministic transformation with no routing or conditional activation.
Parallelism
The rotation is applied independently to each token and each dimension pair, so it is fully parallel.
Hardware requirements
RoPE consists of element-wise trigonometric operations applied to each Q and K vector — fully GPU-accelerated, often fused with attention kernels (FlashAttention supports RoPE fusion).
RoPE requires no specialized hardware — it is a cheap transformation that runs on any accelerator and on CPU.