Feed-Forward Networks

Position-Wise Feed-Forward Network

The original Transformer paper had the the Transformer feed-forward network consist of two linear transformations with a ReLU¹ activation between them. We’re not gonna do that, instead taking after more modern LLMs (such as LLaMA 3 and Qwen 2.5) by combining the SiLU (often called Swish) activation with a gating mechanism called a Gated Linear Unit (GLU). This gets us SwiGLU. We will also omit the bias terms sometimes used in linear layers, following most modern LLMs since PaLM. The SiLU or Swish activation function is defined as follows:

$$\text{SiLU}(x)=x\cdot \sigma (x)\text{,where }\sigma (x)=\frac{1}{1+e^{-x}}$$

Note that its similar to the ReLU function but is smooth at zero. Now, Gated Linear Units (GLUs) were originally defined by Dauphin et al. (2017) as the element-wise product of a linear transformation passed through a sigmoid function and another linear transformation:

$$\text{GLU}(x,W_1,W_2)=\sigma (W_{1}x)\odot W_{2}x$$

Note that $\odot$ represents element-wise multiplication.

Gated Linear Units are suggested to “reduce the vanishing gradient problem for deep architectures by providing a linear path for the gradients while retaining non-linear capabilities.²” Putting the SiLU/Swish and GLU together, we get the SwiGLU, which we will use for our feed-forward networks:

$$\text{FFN}(x)=\text{SwiGLU}(x,W_1,W_2,W_3)=W_2(\text{SiLU}(W_{1}x)\odot W_{3}x)$$

Where,

$$x\in {\mathbb{R}}^{d_{\text{model}}},W_1,W_3\in {\mathbb{R}}^{d_{\text{ff}}\times d_{\text{model}}},W_2\in {\mathbb{R}}^{d_{\text{model}}\times d_{\text{ff}}},\text{and canonically, }{d}_{\text{ff}}=\frac{8}{3}{d}_{\text{model}}$$

$d_{ff}$, btw, merely means the inner feed-forward dimension, typically larger than $d_{model}$—and by convention defined as above in most cases.

9/12/25

Footnotes

1. def. $ReLU(x)=max(0,x)$ ↩

2. This is because Non-linear functions (like ReLU, sigmoid) can have derivatives that are zero or very small, causing vanishing gradients. But you need non-linearity for the network to learn complex patterns. ↩