Rotary Positional Embeddings

Because Transformers need to know token positions in sequences. RoPE injects positional information by rotating query and key vectors by angles that depend on their position.

For token at position i and query/key dimension pair 2k-1:2k, rotate by angle:

$${\theta }_{i,k}=\frac{i}{{\Theta }^{(2k-2)/d}}$$

Where $i$ is the token position, $\theta$ is a constant (like 10000), and $d$ is the dimension of query/key vectors.

This is the rotation matrix for each position $i$ and dimension pair $k$:

$$R_k^i=\left[\begin{array}{cc}\cos ({\theta }_{i,k})& -\sin ({\theta }_{i,k})\\ \sin ({\theta }_{i,k})& \cos ({\theta }_{i,k})\end{array}\right]$$

9/15/25