Language model
A model that assigns a probability to the next token given the tokens that came before. Generation is repeated sampling from that probability.
Every modern LLM is a language model: input is a sequence of tokens, output is a probability distribution over the vocabulary. The bigram in chapter 1, the transformer in chapter 10, and GPT-4 all share that interface — only the function in the middle differs.
Companion explanation in Step by Token, chapter 1.
Where this term gets built
- ch. 1The dumbest model that exists
- ch. 8An attention head by hand
- ch. 10The full transformer block
- ch. 11Prepare a dataset
- ch. 12The minimum code
- ch. 13The training loop
- ch. 14Generation and sampling
- ch. 15Load real weights
- ch. 16Why your model talks badly
- ch. 17Give your model instructions
- ch. 20Talk to your model
- ch. 21Ship a useful one
Continue