KV cache

The stored keys and values of every token already processed, so generating the next token costs one new query against the cache instead of recomputing the whole sequence.

Without a cache, generating n tokens is O(n³) work overall, because every step re-attends over the full prefix. With one, each step is O(n) — which is why the second token of a reply arrives much faster than the first. The cost is memory: the cache grows linearly with context length.