Book roadmap

The book contains nine progressive parts, 57 core chapters, two foundation primers and one optional online multimodal extension. All 57 chapters and both primers are available in English and Chinese. Chapters 40–57 and the HTTP primer were released on September 15, 2026; earlier publication dates are unchanged. The multimodal extension remains planned and unpublished.

Part I: Opening the Black Box

• Chapter 1 Start by Observing: What an LLM Is—and Is Not
• Chapter 2 The Journey of a Token: Training Is Not Inference
• Chapter 3 How Text Becomes Tokens, IDs, and Embeddings
• Chapter 4 From N-grams to Bigrams: A Language Model by Hand

Part II: Essential Foundations

• Chapter 5 Batches, Shapes, Vectors, and Matrices
• Chapter 6 From Logits to Probability, Loss, and Perplexity
• Chapter 7 The Smallest Machine-Learning and Neural-Network Training Loop
• Chapter 8 Real Text Is Not a Clean String
• Chapter 9 Keywords, Bag of Words, and TF-IDF: Establishing a Sparse Baseline
• Chapter 10 From Sparse Representations to Word2Vec and Neural Language Models

Part III: Why Context Is Hard

• Chapter 11 RNNs: Carrying the Past into the Present
• Chapter 12 Why RNNs Forget: BPTT, LSTM/GRU, and Contextual Representations
• Chapter 13 Seq2Seq: Why Read-Then-Write Becomes a Bottleneck
• Chapter 14 Attention: Where Should the Current Output Look?

Part IV: Inside the Transformer

• Chapter 15 Self-Attention and Q/K/V
• Chapter 16 Masks and Multi-Head Attention
• Chapter 17 Assembling a Complete Transformer Block
• Chapter 18 Transformer Families: BERT, GPT, and T5
• Chapter 19 Tokenizers in Depth: BPE, WordPiece, SentencePiece, and Byte-Level Models
• Chapter 20 Building a Mini GPT from Scratch
• Chapter 21 Training, Saving, and Interrogating a Mini GPT

Part V: How Modern LLMs Are Built and Run

• Chapter 22 What Makes a Language Model an LLM
• Chapter 23 Pretraining Data Pipelines and Governance
• Chapter 24 Scaling: How Model, Data, and Compute Grow Together
• Chapter 25 Modern Dense LLM Blocks: RoPE, RMSNorm, and SwiGLU
• Chapter 26 A Map of Efficient, Sparse, and Alternative Architectures
• Chapter 27 Mapping and Estimating Large-Scale Training Systems
• Chapter 28 Mechanistic Interpretability: Decodability, Correlation, and Causality
• Chapter 29 Generation Strategies: From Logits to Sampling
• Chapter 30 Model Inference: Prefill, Decode, KV Cache, and Quantization
• Chapter 31 Serving Efficiency: Reuse, Kernels, Speculation, and Batching
• Chapter 32 Write the Model Evaluation Contract First
• Chapter 33 Choosing an API or Local Model from Evidence

Part VI: Turning Base Models into Useful Capabilities

• Chapter 34 SFT: From Completion to Instruction Following
• Chapter 35 Preference Optimization: RLHF, DPO, and the Limits of Alignment
• Chapter 36 Training Reasoning Capabilities: Supervision, RLVR, and Verifiers
• Chapter 37 Test-Time Compute: Sampling, Search, Verification, and Stopping
• Chapter 38 Diagnosing Capability Gaps: Prompt, RAG, Tools, or Parameter Updates
• Chapter 39 Continued Pretraining, Fine-Tuning, and LoRA/QLoRA

Part VII: Connecting Context, Tools, and Knowledge

• Chapter 40 Beyond Prompts: Context Engineering
• Chapter 41 Structured Output: A Machine-Checkable Data Contract
• Chapter 42 Function Calling and Tools: Who Actually Executes?
• Chapter 43 MCP: Stable Interoperability Concepts and Version Boundaries
• Chapter 44 Modern Retrieval: BM25, Dense, Hybrid, and Rerankers
• Chapter 45 Building a Cited RAG System from Scratch
• Chapter 46 RAG Engineering and Layered Evaluation

Part VIII: From Workflows to Trustworthy Agents

• Chapter 47 Before Agents: Workflows, State, and Memory
• Chapter 48 Agent Skills: Packaging Reusable Practices
• Chapter 49 Reliable Agents: Permissions, Budgets, Replay, and Recovery
• Chapter 50 From Model Evaluation to System Evaluation
• Chapter 51 Hallucination, Bias, Privacy, and Copyright: Continuous Governance
• Chapter 52 Prompt Injection, Jailbreaks, and Untrusted Input/Output
• Chapter 53 Secrets, Permissions, Tenant Isolation, and Supply Chain Security

Part IX: From Prototype to Maintainable Systems

• Chapter 54 From Notebook to Usable Service and Trustworthy UX
• Chapter 55 LLMOps I: Versions, Traces, Releases, and Rollbacks
• Chapter 56 LLMOps II: Caching, Resilience, SLOs, and Feedback
• Chapter 57 Capstone: An Evaluable Personal Knowledge Assistant

Foundation primers

• Primer-PY: Python, notebooks, NumPy/PyTorch tensors, and reading errors. Read after the core of Chapter 5 and before the programming exercises.
• Primer-HTTP: Client/server systems, HTTP, JSON, authentication, streaming, timeouts, and retries. Read before Chapter 42’s practice; Chapters 43 and 54 reuse it.

Online extension

• EXT-MM-01 From Text LLMs to Multimodal Systems.