Augmenting Attention with Exponentially Decaying Memory Improves Query-Aware KV Sparsity

Resource information

Authors

Xiuying Wei, Caglar Gulcehre

Research topics

RAT+, attention computation, KV-cache, dilated attention, sparse inference, Quest, MoBA, SnapKV, needle-in-a-haystack tasks, OLMo2-7B, memory module

Publication date

2026-05-27

Publication

arXiv

Last record update

Nov 3, 2026