Vocabulary
The set of all distinct tokens a model can read or produce. Size ranges from ~80 (character-level) to ~100,000 (modern subword tokenizers).
Companion explanation in Step by Token, chapter 2.
Where this term gets built
Continue
The set of all distinct tokens a model can read or produce. Size ranges from ~80 (character-level) to ~100,000 (modern subword tokenizers).
Companion explanation in Step by Token, chapter 2.
Continue