All termsTry it in the GPU / VRAM Calculator labExplore in the AI Universe
Glossary · Term
KV Cache
L4 · ExpertEvolving
In plain terms
Cached attention keys and values for fast generation.
Technical definition
The stored key and value tensors from previous tokens that let a Transformer generate each new token without recomputing attention over the whole sequence; its size grows with context length and dominates inference memory.
Example
A long chat slows and uses more memory as its KV cache grows.