Chapter 39 Continued Pretraining, Fine-Tuning, and LoRA/QLoRA

39.1 “Fine-tune it” leaves two questions unanswered

Suppose a model cannot reliably interpret a specialized document format even when the right evidence is supplied. Chapter 38 tells us to establish a baseline and test the diagnosis before changing parameters. If that evidence eventually supports training, “fine-tune it” still leaves two questions: what data and target will it learn from, and which numerical values may change?

These choices are independent. We can train on desired assistant responses while updating every weight, or train on those same responses while updating a small added set of parameters. Conversely, the same small parameter mechanism can be used with different training objectives. This distinction prevents the common mistake “SFT versus LoRA,” as though they answered the same question.

We will first separate the axes, then multiply a complete small example. Chapter 7 supplies parameter updates, Chapter 23 data provenance, and Chapter 34 the instruction loss. The CPU exercise needs only the existing Python primer.

39.2 Choose what the model should learn from

Continued pretraining resumes a language-modeling objective on additional text. That text might be a domain corpus or a carefully chosen mixture. A causal decoder can predict the next token; a masked-language encoder can continue its masked objective. “Continued” describes using an already pretrained starting point. It does not promise that the added text is current, licensed or relevant.

Fine-tuning broadly means further training a pretrained model for an adaptation purpose. Usage varies: authors sometimes use it broadly enough to include continued language-model training. For clarity, record the objective rather than infer it from the label. SFT here means learning desired responses from demonstrations, with the declared target mask.

Table 39.1 — Data and objective are one axis.

Training choice	Supplied data or feedback	What produces the training signal
Continued pretraining	Additional text, with its source and mixture recorded	Language-model targets derived from that text
SFT	Request plus desired response, optionally a checked solution	Likelihood of the selected demonstrated response tokens
Preference optimization	Comparisons and a rubric	Reward-model/RL feedback or a direct pair objective such as DPO
RLVR	Tasks, sampled responses and a specified checker	Rewards from checking the sampled attempts

For domain adaptation, the contrast is concrete. Continued pretraining on policy documents teaches prediction in that document distribution. SFT on “question + relevant document → correct scoped answer” teaches the response behavior. Both may be useful in a sufficiently supported project, but neither establishes live access to tomorrow’s policy.

Gururangan and colleagues investigated domain/task-adaptive pretraining of RoBERTa before supervised classification, including controls with irrelevant domains. This motivates testing corpus relevance and distribution fit. It does not establish that nightly fine-tuning is the right way to update a changing lodging cap. Their experiments concern specific encoder/classification tasks.

39.3 Choose which numbers the optimizer may change

A trainable parameter is a stored number the optimizer may update. A frozen parameter is held fixed during this job. Frozen layers can still perform computations and transmit gradients needed to update other layers; “frozen” does not mean removed from the model.

Table 39.2 — The update mechanism is a separate axis.

Update mechanism	Fixed part	Trainable part	What must be recorded
Full update	No base-weight subset is frozen by this definition	All selected model parameters	Exact scope, including embeddings, heads and biases
Inserted adapter modules	Usually most base weights	Added small modules and any explicitly enabled extras	Placement, bottleneck size, nonlinearities and extra trained layers
LoRA	Selected base matrices remain fixed	Low-rank factors attached to those matrices	Target matrices, rank, scaling, dropout and any other trainable values
QLoRA-style adaptation	Quantized base weights remain fixed	LoRA factors, typically stored/trained at higher precision, plus declared extras	Quantizer, storage/compute precision, target matrices and optimizer configuration

Parameter-efficient fine-tuning (PEFT) is the broader family of methods training relatively few parameters for adaptation. An adapter in the original bottleneck-module sense inserts a small transformation: reduce a feature vector’s width, apply a nonlinearity, expand it back, and combine it with the original path. This is distinct from adding a linear low-rank weight increment. Libraries sometimes call LoRA weights “adapters” too; specify the mechanism.

The original Houlsby adapter recipe also trains layer-normalization parameters and a classifier. Thus “adapter training changes only newly inserted weights” is too broad a claim. Inspect the actual trainable list. The original adapter design explains its bottleneck and residual path.

39.4 Calculate an entire low-rank update

Example 39.1 — Four outputs from three inputs. Use a column input $x$ with three coordinates. A base matrix $W_0$ has four rows and three columns: each row combines the three input coordinates into one output coordinate. All values below are assigned, dimensionless teaching values. They are not trained weights from a PEFT run.

$$W_0= \begin{bmatrix} 1&0&0\\ 0&1&0\\ 0&0&1\\ 1&1&0 \end{bmatrix}, \qquad x=\begin{bmatrix}1\\2\\3\end{bmatrix}, \qquad W_0x=\begin{bmatrix}1\\2\\3\\3\end{bmatrix}.$$

The last output is $1\times1+1\times2+0\times3=3$. We want to represent a change without storing twelve independently adjustable increments.

First compress the three input coordinates to one number using a one-row matrix $A=[2,0,-1]$. Then expand that number to four coordinates using a one-column matrix $B=[1,0,-1,2]^{\mathsf T}$. The superscript $\mathsf T$ means transpose: write the listed row as a column. Here $A$ is $1\times3$ and $B$ is $4\times1$.

$$Ax=2\times1+0\times2-1\times3=-1, \qquad B(Ax)=\begin{bmatrix}-1\\0\\1\\-2\end{bmatrix}.$$

This is a change to the output, not a replacement for it. Call the four-coordinate combined output $h$. Add coordinate by coordinate:

$$h=W_0x+B(Ax) =\begin{bmatrix}1\\2\\3\\3\end{bmatrix} +\begin{bmatrix}-1\\0\\1\\-2\end{bmatrix} =\begin{bmatrix}0\\2\\4\\1\end{bmatrix}.$$

We can check the result along a second path. Multiply $B$ by $A$ first. Each row is A multiplied by the corresponding entry of B:

$$\Delta W=BA= \begin{bmatrix} 2&0&-1\\ 0&0&0\\ -2&0&1\\ 4&0&-2 \end{bmatrix}.$$

The third row is $-1[2,0,-1]=[-2,0,1]$, and the fourth is $2[2,0,-1]=[4,0,-2]$. This increment has the same $4\times3$ dimensions as $W_0$, so addition is valid:

$$(W_0+\Delta W)x= \begin{bmatrix} 3&0&-1\\ 0&1&0\\ -2&0&2\\ 5&1&-2 \end{bmatrix} \begin{bmatrix}1\\2\\3\end{bmatrix} =\begin{bmatrix}0\\2\\4\\1\end{bmatrix}.$$

The two paths agree. All increment rows are multiples of one row, so this change uses only one independent row direction. That is what rank one means here; rank is not a quality rating. The base matrix need not have rank one.

Low-Rank Adaptation (LoRA) keeps a base matrix fixed and trains factors for an increment of this kind. With $d_{\mathrm{in}}$ input coordinates, $d_{\mathrm{out}}$ output coordinates and rank bound $r$:

$$A\in\mathbb{R}^{r\times d_{\mathrm{in}}}, \qquad B\in\mathbb{R}^{d_{\mathrm{out}}\times r}, \qquad h=W_0x+\frac{\alpha}{r}BAx.$$

The symbol $\mathbb{R}$ means real-valued entries. The intermediate vector $Ax$ has $r$ coordinates; B restores $d_{\mathrm{out}}$ coordinates. The original LoRA convention scales the increment by $\alpha/r$, with $\alpha$ a chosen scale. Our example sets $r=1$ and $\alpha=1$, recovering the calculations above. The increment’s rank is at most $r$, possibly lower if the factors contain redundant or zero directions.

The original recipe initializes one factor randomly and the other to zero, so the initial increment is zero. Setting both factors to zero is different: each factor’s gradient through the product also contains the other zero factor, so the plain product path cannot get started. The supplied nonzero factors here demonstrate a possible increment, not an initialization or an observed training outcome. Hu et al. describe the parameterization and initialization.

Base path: 12 frozen entries

1. A list of numbersx = [1, 2, 3]ᵀ
2. Stored model numbersW₀: 4 × 3; frozen
3. An operationW₀x = [1, 2, 3, 3]ᵀ

Freezing prevents updates; it does not remove the forward computation or storage.

Increment path: 3 + 4 entries

1. Stored model numbersA = [2, 0, −1]; 1 × 3
2. An operationAx = 2 − 3 = −1
3. Stored model numbersB = [1, 0, −1, 2]ᵀ; 4 × 1
4. An operationBAx = [−1, 0, 1, −2]ᵀ

Add the paths: [0, 2, 4, 1]ᵀ. Scale α/r = 1. Values are assigned, not trained.

Figure 39.1 — A frozen base and seven adjustable numbers contribute to one output. Original hand-assigned matrix example · CC BY-SA 4.0

Figure 39.1 separates frozen base values, trainable factors and computed intermediate values. The input node holds the three numerical coordinates of $x$, which feed both paths. Freezing the base does not freeze the combined output; adding the factors can change behavior, including behavior that used to be correct.

39.5 Count savings before claiming them

The full $4\times3$ matrix has twelve parameters. Rank-one factors have $1\times3+4\times1=7$. This saves five trainable numbers for this matrix, but constrains the allowed increment. It does not reduce the base matrix’s stored twelve numbers.

In general the factor count is $r(d_{\mathrm{in}}+d_{\mathrm{out}})$, compared with $d_{\mathrm{in}}d_{\mathrm{out}}$ independent entries in a full update. Count the actual target matrices and any trained biases or heads; one example is not a whole-model memory estimate.

Table 39.3 — Low rank only saves parameters when the dimensions permit it.

Matrix shape	Rank	Full matrix entries	Factor entries	Factor count / full count
4 × 3	1	12	7	7/12
4 × 3	2	12	14	14/12
4096 × 4096	8	16,777,216	65,536	0.390625%

Rank two already uses more factor parameters than a full increment in our tiny matrix. For the large example, $8(4096+4096)=65{,}536$ and dividing by $4096^2=16{,}777{,}216$ gives 0.00390625, or 0.390625%. These are exact counts, not measured training memory or speed.

Optimizer states and parameter gradients can shrink when fewer numbers are trainable. Base storage, activations, temporary tensors and computation still remain. A small saved adapter file cannot be used as the total training-memory figure. Too small a rank or an unsuitable target-matrix selection can also restrict the task changes we need.

For an ordinary linear LoRA path, we can form a merged matrix $W_0+(\alpha/r)BA$ and use one matrix multiplication. In exact arithmetic this is the same function; finite precision can introduce differences. A saved adapter also depends on its particular base revision and target mapping. Matching dimensions alone does not make it compatible with arbitrary base weights. Keep the base and adapter separately when rollback or switching is required, and evaluate the actual merged artifact if that is what will run.

39.6 Quantizing the base is another change

QLoRA combines a frozen quantized base with trainable low-rank factors. The original work uses 4-bit NormalFloat storage, quantized scale information and paged optimizers, with higher-precision computation. It does not mean that every multiplication, activation and gradient is four-bit. During the forward computation, stored base weights are dequantized as needed; gradients can pass through the computation to the trainable factors while the base stays fixed. Dettmers et al. distinguish storage and computation types.

Example 39.2 — A quantization error survives an adapter addition. To see why a quantized base needs evaluation, use a separate scalar $w=0.93$, input 2, and an assigned quantizer rounding to the nearest multiple of 0.25 in [-1,1]. This is not NF4. The stored approximation is 1.00. The base output changes from $0.93\times2=1.86$ to $1.00\times2=2.00$, an error of 0.14.

Add the same assigned increment 0.05 to each path. The unquantized result is $(0.93+0.05)\times2=1.96$; the quantized-base result is $(1.00+0.05)\times2=2.10$. The difference remains 0.14. Training might learn a compensating increment under suitable data and capacity, but adding LoRA does not automatically erase quantization error. Requantizing a merged matrix introduces another rounding operation and requires another comparison.

Quantization reduces the base’s storage precision; LoRA restricts trainable increments. Neither specifies SFT versus continued pretraining. A uniform scalar example cannot establish QLoRA’s accuracy, resource use or suitability for a particular GPU. We performed the scalar and matrix exercises locally; full-model PEFT/QLoRA training remains optional and was not performed.

39.7 Evaluate the adapted system under its original contract

To compare full updates with LoRA, start from the same base and use the same training data, template, target mask, data split and evaluation rubric. Declare target modules, optimization settings, training-token budget and tuning budget. Equal learning rates or equal step counts alone do not ensure fairness: parameterizations and valid-token counts differ. Choose configurations on development data, then evaluate the retained test once under the same runtime budget.

Comparing continued pretraining with SFT asks a different question because data and objectives differ. Report those differences instead of attributing every result to parameter efficiency. Include the unadapted baseline, desired gains and regressions by language and task. Record sources and licenses for data, base, tokenizer, code and adapter; a new adapter license does not erase obligations associated with its base.

Our actual same-contract adaptation is Chapter 38’s runtime comparison, not a claimed PEFT experiment. Selected English remains at 2/6 robust semantic cases before and after formatting; Chinese remains at 0/6. The other checkpoint remains at 0/6 in both languages. The tiny LoRA calculation cannot override those measurements. A hypothetical SFT/LoRA job would have to improve the same historical, unknown, status and approval tests, retain existing correct cases, and then face untouched acceptance families.

For optional real PEFT work, the English practice README lists the exact evidence to record before selecting an external model or training stack. The core path uses original local fixtures and no downloaded weights. Frequently changing facts, such as the next lodging-policy revision, still call first for an auditable source-update path rather than a default parameter update.

39.8 Part checkpoint: identify the thing that changed

We track five axes: training data, training objective, trainable parameters, runtime input or external operations, and runtime computation budget.

Table 39.4 — Five axes prevent a false ranking of methods.

Change	Primary thing changed	Evidence needed
SFT demonstrations	Training data and response objective	Correct targets, generalization and retention
RLHF or DPO	Preference-derived training signal	Rubric validity, reference handling and independent behavior tests
RLVR	Reward from task checking	Verifier error audit and new-task results
More runtime attempts/search	Computation and selection with fixed parameters	Full cost, candidate diversity, selection quality and stopping
Prompt/RAG/tool use	Input or externally available information/operations	Delivery, relevance, faithful use and authorization where required
LoRA/QLoRA	Trainable parameter mechanism; QLoRA also changes base storage precision	Dimensions, targets, resource accounting and paired regressions

We can now ask a precise question about any proposed adaptation: does it change the evidence, the target, the update, or how much work we do before answering? Several can change together, but the report should say so. The next chapter studies the case where we keep parameters fixed and organize the input well.

39.9 Summary

Continued pretraining and SFT describe data/objective choices. Full updates, inserted adapters and LoRA describe trainable parameters; QLoRA also quantizes base storage. A low-rank increment can use fewer trainable numbers without making the base low rank or removing all training costs. Its arithmetic, quantization effects and compatibility can be checked locally; its usefulness requires the original behavioral evaluation and retention evidence.

39.10 Exercises

1. Reuse Example 39.1’s matrices but set $x=[1,0,0]^{\mathsf T}$. Compute the base, increment and combined outputs along both paths.
2. For a 6 × 4 matrix with rank two, count full and factor parameters. What changes at rank three? Explain why “low rank” is not sufficient to claim savings.
3. Keep Example 39.1’s factors, set $r=1$ and $\alpha=2$, and use the original input. Compute the new output. Why is this not evidence of a better model?
4. A student calls next-token training on domain documents “QLoRA,” with a frozen 4-bit base and trainable low-rank factors. Supply the missing data/objective description. Then change only the data/targets to desired assistant responses.
5. An adapter file is small and can be merged. List what must still be checked before calling it memory-efficient, compatible and useful.
6. A project proposes RAG, RLVR and LoRA simultaneously to repair a wrong current lodging cap. Use the part checkpoint to design a diagnosis and a comparison that can reveal which change helped.

39.11 Reference answers

1. The base gives $[1,0,0,1]^{\mathsf T}$. $Ax=2$, so the increment is $[2,0,-2,4]^{\mathsf T}$ and the sum $[3,0,-2,5]^{\mathsf T}$. Multiplying the merged matrix by this input selects its first column, giving the same result.
2. The full matrix has $6\times4=24$ entries. Rank two uses $2(6+4)=20$, saving four. Rank three uses 30, exceeding 24. Savings require the factor-count inequality, not just a rank below both dimensions.
3. Double the original increment: $[1,2,3,3]^{\mathsf T}+[-2,0,2,-4]^{\mathsf T}=[-1,2,5,-1]^{\mathsf T}$. These assigned coordinates have no task-quality labels or measured training effect.
4. The data/objective choice is continued causal-language pretraining on domain text; the update/storage mechanism is QLoRA-style adaptation. Switching to demonstrations and their response loss makes it SFT using the same update/storage mechanism.
5. Count base, factors, optimizer states, gradients, activations and temporary storage under declared precision/batch/sequence settings. Verify exact base/tokenizer revisions and target mappings, compare merged/unmerged outputs at the used precision, review licenses, and run the unchanged behavioral and retention evaluation.
6. First establish whether the current source is absent or supplied but ignored. Test source delivery separately from response training; specify any checkable rule and audit its verifier before RLVR. LoRA only chooses the trainable increment. Compare a retained baseline with separately declared interventions, keeping cases, criteria and budgets fixed; a combined gain alone cannot identify the responsible component.

39.12 References

• Gururangan et al., Don’t Stop Pretraining, 2020, sections 3–4. Continued domain/task language modeling before supervised downstream adaptation, with relevance controls.
• Houlsby et al., Parameter-Efficient Transfer Learning for NLP, 2019, sections 2–2.1. Inserted bottleneck adapters and the additional parameters trained in the original recipe.
• Hu et al., LoRA, 2021, sections 4.1–4.2. Frozen base matrices, low-rank factors, initialization, scaling, target selection and merging.
• Dettmers et al., QLoRA, 2023, sections 2–3. Quantized storage, higher-precision computation and memory mechanisms; no universal losslessness claim follows from our toy example.