A book in progress · written in public

Tokens to Transformers in Mojo

Build a GPT-2 from scratch — tokenizer to training loop — in a systems language for AI. From raw bytes to a working language model, all in Mojo, all from first principles, with no framework underneath you. Every chapter ends with code you can run. The book follows one thread from start to finish: text becomes token IDs, token IDs become vectors, vectors become contextual representations, and the model learns to predict the next token.

Start reading → Chapter 1

Contents

  1. Chapter 1From Text to Token IDs

    A language model reads integers, not text. This chapter builds the artifact that produces those integers: a minimal BPE tokenizer for learning, then the production tokenizer the rest of the book uses, checked against GPT-2's published token IDs.

Chapter status

How this book works

Nothing here is pseudo-code. The book uses three open-source repositories. Chapter 1 builds simple_bpe, a small tokenizer for learning, and leaves it behind. Everything the book carries forward stands on two production libraries — mbpe, a production BPE tokenizer, and Tenmo, a tensor library and neural network framework in Mojo. Every snippet is drawn from those repos at a pinned revision, and the test commands reproduce them. Each chapter names its revisions up front, in a “Code for this chapter” note where the building starts.