Chapter 23 Pretraining Data Pipelines and Governance

The model card says that the model saw text about travel. That still leaves a decisive question: did the training stream retain the sentence that changed the fictional Beijing limit from 600 to 750 yuan? If a cleaning rule deleted that sentence, increasing the number of training passes would repeat the omission. Data quantity cannot compensate for information removed before training.

Chapter 8 taught us to preserve raw text when making a search key. Here we extend that discipline to a data pipeline: a sequence of recorded transformations from a source to the examples a training or retrieval program actually receives. Governance means assigning the rules, responsibility and evidence for that process, including who can approve use, what may be retained, and how a version can be corrected. It is more than a final “clean” filename.

23.1 Establish the basis for using a source

Before parsing, record where each item came from, who supplied it, the acquisition date, the applicable permission or license, intended uses and redistribution limits. An accessible URL answers where bytes can be obtained; it does not answer all those other questions.

Three records have different jobs. A robots.txt file expresses crawler rules under the Robots Exclusion Protocol. RFC 9309 explicitly distinguishes these from access authorization. Service terms describe the conditions imposed by a service; a content license addresses permissions for specified material. Neither the absence of a crawler restriction nor successful downloading supplies every right needed for training or republication. The answer can also depend on jurisdiction and the particular source. This chapter makes no claim that one general legal exception authorizes arbitrary web training.

Our practice avoids that ambiguity by using original fictional teaching records, offered as data under CC BY-SA 4.0. There is no web acquisition, real contact, real customer account or live order operation. The corpus and figures have their own license; the original implementation is Apache-2.0. These notices are not interchangeable. For example, redistributing adapted textbook material requires considering the attribution and ShareAlike conditions of its license, not merely attaching the code license. A license does not automatically grant unrelated privacy or publicity rights.

This is a narrow, verifiable basis for the local exercise. For another source, a missing permission field should remain a documented unresolved condition until the responsible review establishes the basis for use. Filling it with “public internet” would disguise the missing evidence.

23.2 Follow eight records through the first bad rule

Example 23.1 — Cleaning away the answer. The fixed input has the following eight records in order. Each is enclosed in a single <p>...</p> element; Table 23.1 shows the text inside it. IDs are source labels, not model token IDs. The parser accepts this deliberately limited wrapper, decodes HTML entities and otherwise preserves the text. It is not a general web-page extractor.

Table 23.1 — All original records in the processing exercise. The contact address is synthetic; the .invalid suffix is not a real support destination.

Source label	Parsed text	Role
new	Beijing: 750	Current policy fact
copy	Beijing: 750	Exact copy of new
old	Archived Beijing lodging limit: 600 CNY.	Historical fact
menu	Home | Help | Sign in	Navigation boilerplate
contact	Contact analyst@example.invalid for the fictional desk.	Synthetic contact example
red	The key is red. Answer: red.	Familiar key sentence
zh	新限额为750元。	Chinese current-limit sentence
canary	France capital? Paris.	Deliberate overlap with MG-2 validation

The simplest filter removes the exact navigation string. That helps: the menu contributes no policy fact in this declared collection. An overconfident second rule rejects any record whose decimal digits occupy at least 20% of its Unicode code points, counting spaces and punctuation in the denominator. The intended motivation is to remove numerical clutter.

For Beijing: 750, there are seven letters, one colon, one space and three digits: 12 code points in total. The fraction is $3/12=0.25$. Because $0.25\geq0.20$, the rule rejects the new limit. It rejects its copy too. The Chinese sentence has nine code points, of which three are digits, so $3/9=1/3$ also fails. The old, longer English sentence survives. The filter has therefore made it easier for a later model to encounter the old value while never encountering these two new-value sentences.

This is a complete counterexample to “fewer noisy-looking records means better data.” It is not a measured claim about how much model accuracy falls. We have shown what the declared rule removes, not trained a new model on its output. The input, bad decisions and revised decisions all remain in data/part-v/.

23.3 Repair the rule without erasing the audit trail

For this bounded exercise, the revised filter removes only the known navigation string. Numeric policy facts survive. A production policy would need a broader review than an exception for one favorite answer: sample accepted and rejected records, examine languages and document types, and test the rule on independently chosen numerical facts. Otherwise an exception could merely hide this example’s failure.

Exact deduplication then keeps the first occurrence of each parsed text and records which source it replaced. Thus copy points to new; the old 600 sentence remains because it is different text with a different historical role. Removing old versions merely because they are old would break historical questions in KA-0.

Next, replace the declared synthetic address pattern ending in @example.invalid with [CONTACT]. This illustrates a personally identifiable information (PII) processing step without using anyone’s actual information. It is a narrow pattern rule, not a detector that guarantees anonymity. Names, indirect identifying details and other address formats would need separate handling. Preserve the transformation reason; access to truly sensitive raw data would itself need controls and retention limits. Publishing a raw sensitive copy beside a redacted one would defeat the redaction.

A failed shortcut

1. Text piecesBeijing: 750 → 3 digits / 12 code points
2. An operation0.25 ≥ 0.20 → reject
3. Outside the modelCurrent limit disappears

A smaller file is not evidence that the retained corpus is better.

Revised processing

1. Outside the model8 records → remove 1 menu → 7
2. An operationExact duplicate: copy → new; 6 remain
3. Text piecesSynthetic contact → [CONTACT]; still 6

Keep source IDs, rule versions and the reason for every removal.

Split-aware check

1. Outside the modelCompare with the fixed validation texts
2. An operationQuarantine 1 exact overlap → 5
3. Text piecesOld 600 and new 750 both remain

This separate exercise never overwrites MG-2 or KA-0.

Figure 23.1 — Counts alone hide which information was removed. Original book diagram · CC BY-SA 4.0.

Figure 23.1 also shows why the order and denominator matter. Starting with eight parsed records, removing one menu leaves seven. Removing one duplicate leaves six. Replacing the address changes text but removes no record, so six remain. The final check compares texts with the existing MG-2 validation set. France capital? Paris. exactly matches its first validation record. We deliberately included this canary, a known check case, to ensure the check can fail. Quarantining it leaves five.

Table 23.2 — The revised pipeline’s complete count ledger. These are derived counts from original input, not industrial data-quality statistics.

Stage completed	Retained records	What the count alone omits
Parse	8	Wrapper assumptions and source permissions
Remove known navigation	7	Whether another filter would delete substantive text
Exact deduplication	6	Near-duplicates and historical distinctions
Synthetic contact replacement	6	Changed bytes and undetected identifying information
Quarantine exact validation overlap	5	Paraphrases, related templates and unknown evaluation sets

A real train/validation split should keep related source groups together before deriving windows or training the tokenizer. Otherwise two overlapping windows of one document can cross the split even when their file names differ. Exact hashes make exact equality convenient to check; they do not identify semantic equivalence. Near-duplicate methods compare longer shared spans or approximate textual similarity, with thresholds that can also remove legitimate shared language. The deduplication research in the references studies those tradeoffs; our five-record output does not implement that research system.

The check does not “clean” the accepted MG-2 bundle retroactively. MG-2 intentionally contains related templates and uses validation for snapshot selection. Its limitations remain. The new pipeline exercise is separately versioned; retraining on its output would be a new model experiment with a new data identity.

23.4 A mixture is a sampling rule, not a folder count

Example 23.2 — More exposures without more unique text. Suppose two approved pools contain 800 English and 200 Chinese tokens under one fixed tokenizer. These are assigned counts for a separate budget example, not counts of Table 23.1. We want 1,000 training-token exposures with a 60% English and 40% Chinese mixture.

English receives $1000\times0.6=600$ exposures; Chinese receives $1000\times0.4=400$. The totals add to 1000 because the two shares add to one. Sampling within each pool with replacement permits 400 Chinese exposures from only 200 available tokens. On average that is $400/200=2$ exposures per available Chinese token, while English receives $600/800=0.75$. It does not guarantee that every individual token is drawn exactly twice or three quarters of a time. Sampling varies; an actual packing implementation also has to respect record boundaries.

The distinction changes both scale estimates and interpretation. Repeated exposures consume compute. They do not add 200 newly authored Chinese tokens. Increasing a language’s share can help present it more often while also increasing repetition and overfitting pressure. Equal record counts would not imply equal token counts because record lengths and tokenization differ across languages. Chapter 24 will count processed tokens for compute, while retaining unique-data and mixture information as separate fields.

23.5 Keep two Data Cards for two different jobs

A Data Card makes these choices inspectable as a collection-level contract. It records provenance, content, preprocessing, splits, mixture, overlap, uses, updates and limitations. Our two expanded cards document existing bytes, rather than silently upgrading either project’s dataset.

Table 23.3 — The continuing projects have different data interfaces. Both data-card-v2.json files include the source file’s SHA-256 identity and verification date, 2026-09-13.

Field	Mini GPT corpus	Knowledge-assistant document pack
Source version	mini-gpt-corpus-v1	ka0-v1
Contents	24 train, 8 validation records	5 parallel bilingual documents, 9 queries
Model input	Raw text through frozen byte BPE	Index cards through search-key-v1 and lexical features
Mixture	22 English and 2 Chinese training records; equal sampling probability by record	Separate parallel locale collections; no learned translation
Labels/split	Shifted next-token targets; validation repeatedly used for step selection	Fixed relevance labels; not used as retrieval features
Versions retained	Old/new limit examples and original failed probes	travel-v1 600 and travel-v2 750, with effective dates
Known limit	Related templates; no independent product benchmark	Synonym and stale-premise failures; FAQ is not a live order lookup

The assistant’s raw document body and version metadata remain available even though KA-0 scores only its short index card. Changing the index is different from changing the source; both need their own identities. Chapter 32 will inherit all nine queries and all three lexical baselines, then add a separate answer-evaluation layer. A better answer score must not erase a retrieval failure.

Run python code/part-v/run.py after Primer-PY to reproduce the ledger and resource arithmetic. The command reads local files and prints results; --write deliberately updates only the derived Part V record. The five retained texts and all rejected decisions are statically inspectable in derived-run.json. The two project cards live beside their existing data under data/mini-gpt/ and data/knowledge-assistant/.

23.6 Summary

A useful pipeline preserves the basis for using each source and the meaning of every transformation. Filtering can help or erase the decisive fact. Exact duplicates, historical versions, sensitive details and evaluation overlap require different decisions. A mixture controls exposures, which can exceed unique content. Versioned Data Cards let later training and evaluation refer to the same choices. With those choices explicit, we can ask how parameters, processed tokens and compute should be budgeted together.

23.7 Exercises

1. Change the digit threshold to 30%, retaining the “at least” rule. Which of the two current-limit texts survives, and why is this still a language-dependent failure?
2. A deduplicator removes every document mentioning 600 because 750 is current. Which KA-0 question does this break? What relationship should it preserve instead?
3. A 2000-exposure budget uses 25% Chinese from a 200-token Chinese pool. Calculate its allocated exposures and average exposures per available token. Does that create new unique text?
4. Two windows from the same source article are placed in train and test. Their exact hashes differ. What risk remains, and where should grouping occur?
5. A repository has Apache-2.0 code beside CC BY-SA 4.0 data. A colleague labels the whole export Apache-2.0 because downloading worked. Identify the two missing distinctions.
6. The canary is detected and all exact-overlap checks pass after its removal. Can the author call the new evaluation independent of all training data? Explain the remaining evidence needed.

23.8 Reference answers

1. Beijing: 750 has 25% digits and survives. The Chinese sentence has one third and is rejected. The changed threshold still removes the same fact in one language because text length and representation differ; it has not established a language-neutral notion of quality.
2. The historical 600 query loses its relevant source. Retain version identity, effective dates and the superseded-by relationship, then select the appropriate version for the question. A current query and a historical query need different evidence.
3. Chinese receives 500 exposures, averaging 2.5 per available token. Sampling with replacement permits this, with variation among tokens. The unique pool remains 200 tokens; compute and repetition grow, not the amount of unique text.
4. Shared article content can leak despite different window bytes. Assign related source groups to splits before windowing, and investigate near-duplicates across groups. Exact inequality cannot prove independent information.
5. Acquisition success is not a permission grant. Code and data can also have different licenses within one repository. Inspect the applicable license and notices for each exported asset and any other rights or terms; the local exercise does not establish permission for an unrelated corpus.
6. No. The canary establishes that one exact-match case is detected. Paraphrases, common templates, related sources and unknown pretraining data can remain. Document the checks and exposure history, avoid using evaluation labels to fit the candidate, and limit the generalization claim to the actual evidence.

23.9 References

• Gebru et al., Datasheets for Datasets. The composition, collection and preprocessing questions inform the two original Data Cards.
• Raffel et al., Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Section 2.2 documents a particular C4 cleaning pipeline; its English/task-specific rules are not universal data-quality tests.
• Lee et al., Deduplicating Training Data Makes Language Models Better. Section 4 distinguishes substring and near-document duplication; the experiments show why thresholds and evaluation overlap matter.
• IETF, RFC 9309. Section 1 defines the crawler protocol’s scope and distinguishes it from access authorization.
• Creative Commons, BY-SA 4.0 legal code, and Apache Software Foundation, Apache License 2.0. These are the actual licenses applied to this book’s original content/data and implementation, respectively; they do not license unrelated third-party material.