RMS Layer Normalization

Given a vector of activations $a\in {\mathbb{R}}^{d_{\text{model}}}$¹, RMSNorm will rescale each activation $a_i$ as follows:

$$\text{RMSNorm}(a_i)=\frac{a_i}{\text{RMS}(a)}{g}_i$$

Where,

$$\text{RMS}(a)=\sqrt{\frac{1}{d_{\text{model}}}\sum _{i=1}^{d_{\text{model}}}{a}_i^2+\epsilon }$$

Here, $g_i$ is a learnable “gain” parameter.. there are d_model such parameters total). $ϵ$ is a hyperparameter that is often fixed at 1e-5. This is a mean function by the way :P

9/12/25

Footnotes

1. def. That is, a vector $a$ that exists in the dimensional space of $\text{model}$. ↩