Momentum
Trick that accumulates a running velocity of recent gradients. Smooths the trajectory when consecutive steps reinforce, dampens it when they oscillate.
Companion explanation in Step by Token, chapter 6.
Where this term gets built
Continue
Trick that accumulates a running velocity of recent gradients. Smooths the trajectory when consecutive steps reinforce, dampens it when they oscillate.
Companion explanation in Step by Token, chapter 6.
Continue