Embedding

A dense low-dimensional vector representing a token. Words used in similar contexts end up with similar vectors.

Embeddings replace one-hot encodings of tokens with continuous vectors that capture semantic relationships. The geometry of the embedding space is meaningful: directions can encode features like gender, formality, register.