A Survey of RWKV
Research paper by Zhiyuan Li and collaborators. arXiv record dated 2025-01-05. Research area: Language technology.
Resource information
- Authors
- Zhiyuan Li, Tingyu Xia, Yi Chang, Yuan Wu
- Research area
- Language technology
- Application domain
- General AI research
- AI methods
- Transformers, Computer vision
- Research topics
- cs.CL, cs.AI
- arXiv record date
- 2025-01-05
- Publication
- arXiv
- Upstream last modified
- Jan 7, 2025
- Last record update
- Nov 3, 2026
Abstract
The Receptance Weighted Key Value (RWKV) model offers a novel alternative to the Transformer architecture, merging the benefits of recurrent and attention-based systems. Unlike conventional Transformers, which depend heavily on self-attention, RWKV adeptly captures long-range dependencies with minimal computational demands. By utilizing a recurrent framework, RWKV addresses some computational inefficiencies found in Transformers, particularly in tasks with long sequences. RWKV has recently drawn considerable attention for its robust performance across multiple domains. Despite its growing popularity, no systematic review of the RWKV model exists. This paper seeks to fill this gap as the first comprehensive review of the RWKV architecture, its core principles, and its varied applications, such as natural language generation, natural language understanding, and computer vision. We assess how RWKV compares to traditional Transformer models, highlighting its capability to manage long sequences efficiently and lower computational costs. Furthermore, we explore the challenges RWKV encounters and propose potential directions for future research and advancement. We consistently maintain the related open-source materials at: https://github.com/MLGroupJLU/RWKV-Survey.