Positional embedding

A vector added to each token embedding to encode *where* the token sits in the sequence. Without it, attention would treat a sentence as an unordered bag of tokens.

Attention is permutation-equivariant: shuffle the input and the output shuffles the same way. Position has to be injected explicitly. GPT-2 learns one vector per position (a plain lookup table); modern models often use RoPE, which rotates queries and keys by an angle proportional to position and extends better to long contexts.

Companion explanation in Step by Token, chapter 5.