KV cache
Key–value cache
Stored attention information from tokens a model has already processed. Reusing it can speed up text generation, but it uses memory.
By Mega AI Hub Editorial · Compute & hosting
5,108,791 AI resources indexedWorking toward the world’s largest AI resource index.
Key–value cache
Stored attention information from tokens a model has already processed. Reusing it can speed up text generation, but it uses memory.
By Mega AI Hub Editorial · Compute & hosting