About
The loss curve is a code-first course on building a language model. Every chapter gives you runnable code in the browser or on your machine, then shows what that code produced through a visualization tied to the same data the next chapter will pick up. By the end you have a working transformer training on your own machine.
It is the constructive counterpart to Step by Token. That site explains how an LLM works; this one shows how you build one. The two are designed to be read together, but neither requires the other.
Methodology
The book's organizing principle is artifact-first, code-first. Every chapter starts with runnable code: read the function, run it, look at what came out, then read the prose that explains the important pieces.
The chapters build cumulatively. The bigram model from chapter 1 is smoothed in chapter 2, gets a learned tokenizer in chapter 3, gets dense embeddings in chapter 4, gets attention in chapter 8, becomes a transformer block in chapter 10, then moves into local Python training and inference.
The pedagogical guarantee is nothing is a black box. Every line of code the reader meets is something they can open. The production implementation in lib/ml/ is available for cross-reference and is itself short, tested, and shaped to match the chapter.
Inspirations
The presentation owes a lot to The Nature of Code by Daniel Shiffman: manipulable sketches as pedagogical units, an open conversational voice, the patience to make a single concept interesting before moving on.
The architecture is informed by Andrej Karpathy's nanoGPT, Distill, Jay Alammar, and the public work that made scaling laws and mechanistic analysis legible outside frontier labs.
About the author
Dimitri Mérault
I have been building iOS apps since 2009 — almost since the App Store opened. The Loss Curve is a different kind of project: it came out of my interest in language models, and out of the conviction that the surest way to understand something is to build it.
When LLMs became part of my daily work, the vague metaphors stopped being enough. I wanted to know what was actually happening between a prompt and a reply. Writing every piece of this course — the tokenizer, the attention head, the training loop — is what forced me to find out. Nothing exposes a gap in your understanding faster than having to make the code run.
I care about well-made interfaces and honest explanations. This site tries to be both: no hand-waving where a number would do, and no claim that a 14-million-parameter model is more than it is.
I also write Step by Token, which explains how language models work without asking you to write any code. The two sites are the same curiosity approached from opposite ends — one takes an LLM apart, the other puts one together.
Credits
Built with Next.js, React, Tailwind CSS, MDX, KaTeX, Shiki, D3, and Radix UI.
Typography: Source Serif 4, Inter, JetBrains Mono, served via next/font.
The reference dataset for chapters 11-15 is TinyShakespeare (public domain). Tokenization in those chapters uses tiktoken with the GPT-2 vocabulary.
v0.1 · 21 chapters · stage 6