Cross-attention for State-based model RWKV-7

Research paper by Liu Xiao and collaborators. arXiv record dated 2025-04-19. Research area: Computer vision.

By Liu Xiao, Li Zhiyuan, Lin YueyuResearch paper

Resource information

Authors: Liu Xiao, Li Zhiyuan, Lin Yueyu
Research area: Computer vision
Application domain: General AI research
AI methods: Transformers
Research topics: cs.CV, cs.CL
arXiv record date: 2025-04-19
Publication: arXiv
Upstream last modified: Apr 22, 2025
License: CC BY 4.0
Last record update: Nov 4, 2026

Abstract

We introduce CrossWKV, a novel cross-attention mechanism for the state-based RWKV-7 model, designed to enhance the expressive power of text-to-image generation. Leveraging RWKV-7's linear-complexity Weighted Key-Value (WKV) architecture, CrossWKV integrates text and image modalities in a single pass, utilizing a generalized delta rule with vector-valued gating and low-rank adaptations (LoRA) to achieve superior cross-modal alignment. Unlike Transformer-based models, CrossWKV's non-diagonal, input-dependent transition matrix enables it to represent complex functions beyond the $\mathrmTC^0$ complexity class, including all regular languages, as demonstrated by its ability to perform state-tracking tasks like $S_5$ permutation modeling. Evaluated within the Diffusion in RWKV-7 (DIR-7) on datasets such as LAION-5B and ImageNet, CrossWKV achieves a Frechet Inception Distance (FID) of 2.88 and a CLIP score of 0.33 on ImageNet 256x256, matching state-of-the-art performance while offering robust generalization across diverse prompts. The model's enhanced expressivity, combined with constant memory usage and linear scaling, positions it as a powerful solution for advanced cross-modal tasks, with potential applications in high-resolution generation and dynamic state manipulation.Code at https://github.com/TorchRWKV/flash-linear-attention