Optimizer

The rule that turns gradients into parameter updates. SGD subtracts the gradient; momentum accumulates it; Adam scales it per-parameter by its recent magnitude.

Companion explanation in Step by Token, chapter 6.