Chapter 25 Modern Dense LLM Blocks: RoPE, RMSNorm, and SwiGLU

Move the red-key clue farther from the question. The model now needs to use a different positional relationship, but increasing its allowed input length does not establish that it can do so. To investigate a proposed block change, we must locate the computation it changes and follow the numbers through it. A configuration name alone cannot establish long-range recall or better judgment.

Chapter 17’s block had three distinct needs. Attention needed information about positions. Each sublayer needed a controlled input scale. The feed-forward network needed a nonlinear transformation at each position. This chapter changes those three mechanisms while retaining causal visibility and residual addition. These are local choices within a dense block, not one inevitable upgrade path.

25.1 Put position into the query-key comparison

Learned absolute positions added a stored position row to a token representation. Another possibility is to change how projected queries and keys are compared. Recall Chapter 3’s two-coordinate arrow. Turning an arrow a quarter turn counterclockwise maps [x,y] to [-y,x]; its length stays the same. The turn changes its direction and therefore its dot product with another arrow.

Example 25.1 — Relative position from two rotations. Before rotation, let both projected query and key be [1,0]. Assign a quarter turn per position for this teaching example. Positions start at 0. We ask for the unscaled dot product, before the attention division by the square root of head width and before Softmax.

Table 25.1 — Rotate both sides, then compare. These are exact constructed values, not a production frequency schedule.

Query position	Key position	Rotated query	Rotated key	Dot product
0	0	[1,0]	[1,0]	1
0	1	[1,0]	[0,1]	0
1	2	[0,1]	[-1,0]	0
0	2	[1,0]	[-1,0]	−1

In the second row, $1\times0+0\times1=0$. In the third, $0\times(-1)+1\times0=0$ again. Advancing both positions by one preserves their difference and hence this comparison. These are algebraic comparisons; a causal mask would subsequently exclude a future key, such as key 1 for query 0. Position encoding never grants permission to see a future token.

A general turn by angle $\theta$ is written:

$$R_{\theta}(x,y)=(x\cos\theta-y\sin\theta,\;x\sin\theta+y\cos\theta).$$

Sine and cosine specify where a unit-length arrow points after a turn. At zero angle, cosine is 1 and sine is 0, so nothing changes. At a quarter turn, cosine is 0 and sine is 1, giving exactly [-y,x]. Angles are commonly measured in radians: a full turn is $2\pi$, so a quarter turn is $\pi/2$. You need no trigonometric table beyond these values for the example.

Rotary position embeddings (RoPE) apply position-dependent rotations to coordinate pairs of projected Q and K. Different pairs normally use different angular rates. For one rate $\theta$, positions $m,n$ give:

$$(R_{m\theta}q)\cdot(R_{n\theta}k)=q\cdot R_{(n-m)\theta}k.$$

$q,k$ are one query/key pair of coordinate vectors; the dot multiplies corresponding coordinates and sums them. Turning both arrows back by the query’s angle leaves their dot product unchanged, so only the remaining relative turn appears on the right. Substitute $m=1,n=2$ and the quarter-turn unit vectors: the right side compares [1,0] with [0,1], giving zero, as in Table 25.1.

In the variant here, V is not rotated. Attention still divides scores by the square root of head width, applies the visibility mask, normalizes over source positions and mixes values. RoPE changes the position-sensitive Q/K comparison, not the meaning of the source axis or the residual connection.

This mechanism does not prove useful long-context extrapolation. Our single-rate example even repeats after four positions. Real multi-rate configurations are less trivial, but changing a maximum-context field does not teach the model to use unfamiliar distances. Training lengths, frequency settings and long-distance tests still matter. The red/blue clue should be moved farther away and changed, not merely shown inside a larger buffer.

25.2 Control magnitude without subtracting a mean

Chapter 17’s LayerNorm subtracted a row mean and divided by a scale derived from its deviations. Suppose we instead divide by the root mean square of the original coordinates: square them, average them, then take the square root. This retains the row’s offset while controlling magnitude.

Example 25.2 — Every term in the denominator. For one token row [3,4], the squares are 9 and 16. Their mean is $(9+16)/2=12.5$ because there are two coordinates in this row, not two tokens in a batch. Its square root is approximately 3.535534. Dividing gives [0.848528,1.131371] to six decimals. The normalized coordinates have mean square one: the original total squared magnitude 25 is divided by 12.5, then averaged over two coordinates.

With a small stabilizer and learned per-coordinate gains, RMSNorm is:

$$y_i=g_i\frac{x_i}{\sqrt{\frac{1}{C}\sum_{j=1}^{C}x_j^2+\epsilon}}.$$

$C$ is the number of coordinates in one token row, $i$ selects the output coordinate, and $j$ visits every coordinate used to calculate the shared denominator. The gain $g_i$ is a learned scale for coordinate $i$. Our variant has no learned offset. The positive stabilizer $\epsilon$ prevents division by zero for an all-zero row. Example 25.2 sets gains to one and epsilon to zero only for its nonzero hand calculation; the supplied config uses epsilon 0.000001 inside the square root.

To expose the difference from LayerNorm, add one to each original coordinate. RMSNorm of [4,5] divides by the square root of 20.5, giving approximately [0.883452,1.104315], a different direction. LayerNorm with unit gains, no offset and zero epsilon instead maps both [3,4] and [4,5] to [-1,1]: their centered deviations are the same. Omitting mean subtraction changes an invariance; it is not merely a shorter spelling for the same operation.

Multiplying every original coordinate by a positive constant cancels between numerator and denominator when epsilon is zero. With a fixed positive epsilon, this cancellation is only approximate. A zero row with zero epsilon is undefined, so the practice rejects it explicitly. The paper reports useful results for RMS rescaling in its settings; neither normalization alone guarantees stable training or correct answers.

25.3 Let one FFN branch modulate another

In Mini GPT, a row expanded to a hidden width, passed through ReLU, and projected back. ReLU replaces negative coordinates with zero. A gated alternative first creates two hidden rows from the same input. One branch controls the contribution of the other through elementwise multiplication. These are coordinate-wise operations at a token position; they do not attend to a new position.

Example 25.3 — A complete two-coordinate gated FFN. Use the row $x=[1,-1]$, no biases, and the following matrices. Rows are input coordinates and columns output coordinates, as in Chapter 5.

$$W_g=\begin{bmatrix}1&0\\0&1\end{bmatrix},\qquad W_u=\begin{bmatrix}2&0\\0&-3\end{bmatrix},\qquad W_d=\begin{bmatrix}1&0\\1&1\end{bmatrix}.$$

The gate projection gives $xW_g=[1,-1]$; the other projection gives $xW_u=[2,3]$. On the gate branch use $\operatorname{SiLU}(a)=a/(1+e^{-a})$, also called Swish with its coefficient fixed to one. At 1 it is $e/(1+e)\approx0.731059$; at −1 it is $-1/(1+e)\approx-0.268941$. Unlike ReLU, this gate retains a small negative value. It is also not a probability: SiLU outputs are not confined to zero through one.

Multiply corresponding entries, not every pair. The first becomes $2\times0.731059\approx1.462117$, the second $3\times(-0.268941)\approx-0.806824$. Finally multiply that row by $W_d$. Its first output adds the two entries, giving 0.655293; the second takes the second entry, giving −0.806824. Thus the FFN output is [0.655293,-0.806824]. Calculations use unrounded values; the display rounds at the end.

This is the SwiGLU feed-forward operation:

$$\operatorname{FFN}(x)=\bigl(\operatorname{SiLU}(xW_g)\odot xW_u\bigr)W_d.$$

The symbol $\odot$ means elementwise multiplication. For an input width $C$ and hidden width $F$, both input matrices have shape $[C,F]$; the down matrix has shape $[F,C]$. The hidden branches must align coordinate by coordinate, and the output returns to width $C$ so a residual row can be added. This local example computes only the FFN; it does not pretend its row is the output of an entire trained block.

Table 25.2 — Equal-width and equal-parameter comparisons differ. Ignore biases and normalization parameters in this FFN-only count.

FFN	Input width	Hidden width	Matrix parameters
ReLU, two matrices	12	48	$2\times12\times48=1152$
SwiGLU, three matrices	12	48	$3\times12\times48=1728$
SwiGLU, three matrices	12	32	$3\times12\times32=1152$

A like-for-like hidden width gives the gated version more parameters. Reducing hidden width from 48 to 32 matches this parameter budget. Which version learns better requires controlled training and evaluation; the count itself proves no quality improvement.

25.4 Put the three changes back in one block

Position acts on projected Q and K

1. A list of numbersX [B, T, 12] → RMSNorm over 12
2. An operationProject Q/K/V → 3 heads × 4 coordinates
3. An operationRotate Q/K pairs; keep V unrotated

Causal attention → join → output projection [B, T, 12]; add the original X to obtain R.

Gated FFN returns to residual width

1. An operationR → second RMSNorm
2. Stored model numbersGate [12, 32] and up [12, 32]
3. An operationSiLU(gate) × up, element by element
4. Stored model numbersDown [32, 12] → add R

R bypasses the FFN as well. This config replaces mechanisms; it cannot load MG-2 weights unchanged.

Figure 25.1 — Three changes occupy three different places. Original book diagram · CC BY-SA 4.0.

Figure 25.1 uses the original teaching schema data/part-v/modern-config.json. Its hidden_size=12, query_heads=3 and head_dim=4 agree because $3\times4=12$. The first RMSNorm reads each 12-coordinate row before attention. Q and K are split into three heads and rotated in pairs; V retains its ordinary projected values. After masked attention, head joining and output projection, add the original input to form the residual row R.

The second RMSNorm reads R. The two FFN projections each produce 32 coordinates, SiLU gates one branch, and the product projects back to 12. Add R to obtain the block output. Both bypass paths remain. pre_norm=true specifies where the norms occur; bias=false explains the missing affine offsets; ffn_hidden_size=32 maps to both branches rather than just one. rope_theta=10000 names the base of a multi-rate position schedule, not the quarter-turn schedule used for hand calculation.

These are teaching fields, not a claim that any framework accepts this file. The 2048-slot limit is a declared configuration capacity, not a measured effective context length. MG-2 has different shapes, learned position rows, LayerNorm offsets and a ReLU FFN. Its existing weights cannot simply be loaded into this changed structure and expected to preserve behavior. The scalar practice reproduces the three local examples; it does not claim to have trained a modern replacement model.

25.5 Summary

RoPE changes how positions enter Q/K comparisons. RMSNorm rescales each token row without subtracting its mean. SwiGLU uses two hidden projections and a coordinate-wise gate before returning to residual width. Their locations, shapes and failure cases matter more than their names. They do not change causal visibility, guarantee long-context skill or implement service optimization. The next chapter changes the comparison axes again: stored history, attention work and the portion of parameters used for each token.

25.6 Exercises

1. In the quarter-turn example, put query and key at positions 2 and 3. Calculate their vectors and dot product. Can a causal query at position 2 use key 3?
2. Normalize [0,2] with unit gains and zero epsilon. Why is the denominator the square root of 2 rather than 2? What happens for [0,0]?
3. Add 10 to both coordinates of [3,4]. Does RMSNorm have to preserve the previous result? Compare the reason with LayerNorm’s centering.
4. Change $W_d$ in Example 25.3 to the identity matrix. What is the new FFN output? Explain why no token-to-token interaction was added by this change.
5. With input width 24 and a ReLU hidden width 96, what gated hidden width gives the same bias-free matrix count? Show both counts.
6. A config doubles max_sequence_slots and leaves training and weights unchanged. What can this establish, and what red/blue test is still needed?

25.7 Reference answers

1. The vectors are [-1,0] and [0,-1], whose dot product is zero. The relative turn is still one quarter. The causal mask excludes key 3 from query 2 despite the valid algebraic comparison.
2. Squared coordinates average to $(0+4)/2=2$, so the result is $[0,\sqrt2]$. Dividing by length 2 would make the sum of squares one, whereas RMS normalization makes the mean of squares one. The all-zero row needs positive epsilon; with zero epsilon the denominator is zero and the operation is undefined.
3. No. RMSNorm uses squares of the original values, so a shared additive offset changes its denominator and output. LayerNorm subtracts that shared offset through its mean, leaving centered deviations unchanged under the declared gain/epsilon settings.
4. It returns the pre-down-projection product [1.462117,-0.806824]. The identity matrix preserves the two coordinates. Every calculation is within one row; this does not introduce access to other positions.
5. The ReLU count is $2\times24\times96=4608$. A gated width of 64 gives $3\times24\times64=4608$. Keeping width 96 instead would increase the budget by one half.
6. It may permit a larger input buffer if the implementation supports it. It does not establish correct use of distant information. Place matched red/blue and changed-color clues at longer distances, keep the query and scoring rule fixed, and report the whole distance-conditioned result rather than one successful long prompt.

25.8 References

• Su et al., RoFormer. Sections 3.1–3.2 derive rotations and relative Q/K comparison; the chapter’s quarter turns are original teaching values.
• Zhang and Sennrich, Root Mean Square Layer Normalization. Section 4 and Table 1 distinguish rescaling from recentering; the epsilon/gain conventions here are explicit.
• Shazeer, GLU Variants Improve Transformer. Equations 4–6 specify gated FFNs and the paper explains the reduced hidden width used for its matched-parameter experiments.