Byte-Pair Encoding (BPE)
Tokenization scheme that starts with characters and iteratively merges the most-frequent adjacent pair. Produces subword tokens.
Originally a 1994 data-compression algorithm. Used in GPT-2/3/4 and most modern LLMs. The merges discover morphology — suffixes like 'ing' or 'ed' emerge naturally as common subwords.
Companion explanation in Step by Token, chapter 2.
Where this term gets built
Continue