A book in progress · written in public
Tokens to Transformers in Mojo
Build a GPT-2 from scratch — tokenizer to training loop — in a systems language for AI. From raw bytes to a working language model, all in Mojo, all from first principles, with no framework underneath you. Every chapter ends with code you can run. The book follows one thread from start to finish: text becomes token IDs, token IDs become vectors, vectors become contextual representations, and the model learns to predict the next token.
Contents
-
Chapter 1From Text to Token IDs
A language model reads integers, not text. This chapter builds the artifact that produces those integers: a minimal BPE tokenizer for learning, then the production tokenizer the rest of the book uses, checked against GPT-2's published token IDs.
Chapter status
- Chapter 1 — From Text to Token IDs Published
- Chapter 2 — From Token IDs to Vectors Complete
- Chapter 3 — Attention from Scratch Complete
- Chapter 4 — Assembling GPT-2 Complete
- Chapter 5 — Pretraining on TinyStories Complete
- Chapter 6 — Fine-tuning for Classification Complete
- Chapter 7 — Instruction Fine-tuning planned
- Chapter 8 — The BERT Encoder Side planned
How this book works
Nothing here is pseudo-code. The book uses three open-source repositories. Chapter 1 builds simple_bpe, a small tokenizer for learning, and leaves it behind. Everything the book carries forward stands on two production libraries — mbpe, a production BPE tokenizer, and Tenmo, a tensor library and neural network framework in Mojo. Every snippet is drawn from those repos at a pinned revision, and the test commands reproduce them. Each chapter names its revisions up front, in a “Code for this chapter” note where the building starts.