Tokenizer
The function that splits raw text into tokens. Ranges from naive whitespace + lowercase to learned schemes like BPE.
Companion explanation in Step by Token, chapter 2.
Where this term gets built
Continue
The function that splits raw text into tokens. Ranges from naive whitespace + lowercase to learned schemes like BPE.
Companion explanation in Step by Token, chapter 2.
Continue