Skip to content

Chapters Lexicon AboutLanguage

GELU

Smooth approximation of ReLU used in transformers. Roughly x·Φ(x) where Φ is the Gaussian CDF.

Where this term gets built

ch. 10The full transformer block
ch. 15Load real weights

Continue

← All terms Browse chapters

Before you start

Before you start

Start the project

The dumbest model that exists
Counting isn't enough
Train your own tokens
Giving meaning to words

Make it learn

A neuron that learns
Stacking layers
Gradient descent live

Build the transformer

An attention head by hand
Multi-head and residuals
The full transformer block

Train and use the LLM

Prepare a dataset
The minimum code
The training loop
Generation and sampling
Load real weights
Why your model talks badly

Make it useful, cheaper, and usable

Give your model instructions
Fine-tuning with LoRA
Simple quantization
Talk to your model
Ship a useful one

Appendices

Appendix · Backprop by hand
Appendix · RLHF and DPO

Guides

Self-attention from scratch
Build a BPE tokenizer from scratch
Build an LLM from scratch — a code-first walkthrough
Build a GPT from scratch
LoRA fine-tuning, built from scratch
PyTorch LLM tutorial — from blank file to working GPT
Train a small language model — locally
Build a Transformer from scratch

Reference

All chapters
Lexicon
About

Constructive counterpart to Step by Token.RSS

v0.1 · stage 6