Autograd

Automatic differentiation: the machinery that records every operation in the forward pass and replays it backwards to compute gradients, so you never derive them by hand.

Each operation knows its own local derivative. Autograd chains them with the chain rule, in reverse order. `loss.backward()` in PyTorch is one call to that replay. The appendix builds a tiny version from scratch so the call stops being magic.

Companion explanation in Step by Token, chapter 6.