5,542,141 AI resources indexedWorking toward the world’s largest AI resource index.

PDF

BlackGoose Rimer: Harnessing RWKV-7 as a Simple yet Superior Replacement for Transformers in Large-Scale Time Series Modeling

Research paper by Li weile and collaborators. Published 2025-03-08. Topics: Transformers, LSTMs, GRUs, RWKV-7, meta-learning.

Resource information

Authors
Li weile, Liu Xiao
Research area
Machine learning
Application domain
General AI research
AI methods
Large language models, Transformers
Research topics
cs.LG
arXiv record date
2025-03-08
Publication date
2025-03-08
Publication
arXiv
Upstream last modified
Mar 11, 2025
License
CC BY 4.0
Last record update
Nov 3, 2026

Abstract

Time series models face significant challenges in scaling to handle large and complex datasets, akin to the scaling achieved by large language models (LLMs). The unique characteristics of time series data and the computational demands of model scaling necessitate innovative approaches. While researchers have explored various architectures such as Transformers, LSTMs, and GRUs to address these challenges, we propose a novel solution using RWKV-7, which incorporates meta-learning into its state update mechanism. By integrating RWKV-7's time mix and channel mix components into the transformer-based time series model Timer, we achieve a substantial performance improvement of approximately 1.13 to 43.3x and a 4.5x reduction in training time with 1/23 parameters, all while utilizing fewer parameters. Our code and model weights are publicly available for further research and development at https://github.com/Alic-Li/BlackGoose_Rimer.

Research paper

Full text

Text edition from the author’s arXiv HTML. Figures and equations may display differently from the original.

Time series models face significant challenges in scaling to handle large and complex datasets, akin to the scaling achieved by large language models (LLMs). The unique characteristics of time series data and the computational demands of model scaling necessitate innovative approaches. While researchers have explored various architectures such as Transformers, LSTMs, and GRUs to address these challenges, we propose a novel solution using RWKV-7, which incorporates meta-learning into its state update mechanism. By integrating RWKV-7’s time mix and channel mix components into the transformer-based time series model Timer Liu et al. 2024, we achieve a substantial performance improvement of approximately 1.13x to 43.3x and a 4.5x reduction in training time with 1/23 parameters, all while utilizing fewer parameters. Our code and model weights are publicly available for further research and development. https://github.com/Alic−Li/BlackGoose_Rimer

1 Introduction

Figure 1: The benchmarks reveal that Rimer, with a significantly reduced parameter count of just 1.6 million, consistently outperforms or matches the performance of Timer, which relies on a much larger 37.8 million parameters, across multiple metrics.

Time series modeling stands as a fundamental pillar of machine learning Wen et al. 2022, driving a wide array of applications such as forecasting, anomaly detection, and decision-making across fields like finance Dong et al. 2024, healthcare, and environmental science11 1 This is an ongoing project. With the rapid expansion of data in these domains, scaling time series models to efficiently process large datasets has emerged as a critical challenge. This issue parallels the scalability hurdles encountered in large language models (LLMs), where computational efficiency and performance at scale are paramount. However, the distinct properties of time series data—namely, temporal dependencies, high dimensionality, and often the demand for real-time analysis—pose unique difficulties that traditional approaches struggle to address effectively. Over the years, researchers have investigated numerous architectures to tackle these challenges Zhang et al. 2025; Lang et al.; Wang et al.; Hou and Yu 2024, including transformers, Long Short-Term Memory networks (LSTMs), and Gated Recurrent Units (GRUs). While these models excel at capturing temporal patterns in smaller-scale settings, their efficiency and performance often degrade when applied to large datasets, highlighting the need for innovative solutions that can scale without compromising accuracy. This gap in capability has motivated the development of new methodologies tailored to the demands of large-scale time series modeling. In this paper, we introduce a novel approach to meet these needs through RWKV-7, an architecture that integrates meta-learning into its state update mechanism Grazzi et al. 2024. RWKV-7 features two core innovations—time mix and channel mix Peng et al. 2023 which enhance its ability to model complex temporal dynamics while maintaining computational efficiency. By embedding these components into the transformer-based time series model Timer, we achieve a remarkable 1.13x to 43.3x improvement in performance and a 4.5x reduction in training time compared to existing methods with 1/23 parameters. Furthermore, our approach leverages fewer parameters, offering a lightweight yet powerful solution for scaling time series models.

Our contributions in this work are twofold:

Revisiting RNNs for Time Series with RWKV-7: We reintroduce the recurrent neural network (RNN) paradigm to time series modeling by adopting RWKV-7, a novel architecture tailored for large-scale time series data. This approach harnesses the sequential processing strengths of RNNs while overcoming their traditional challenges in handling extensive datasets, resulting in improved scalability and performance.

Benchmarking Against Transformer-Based Models: We perform comprehensive benchmarks to evaluate our RWKV-7-based model against previous transformer-based time series models. Our experiments highlight significant improvements, showcasing the effectiveness of our approach in terms of predictive accuracy and computational efficiency.

2 Related Work

2.1 meta learning

Meta-learning Prudencio and Ludermir 2004, often described as ”learning to learn,” provides a powerful framework for enhancing large time series models by enabling them to quickly adapt to new tasks with limited data Khosravi et al. 2023. These models, commonly used for tasks like forecasting and anomaly detection, typically require significant computational resources and extensive retraining when faced with new datasets. By incorporating meta-learning, they can utilize insights gained from prior tasks to minimize retraining efforts, employing techniques such as few-shot learning, transfer learning, and hyperparameter optimization to generalize effectively across varied scenarios. This approach not only boosts efficiency, versatility, and scalability but also tackles challenges like the need for diverse training tasks and initial computational overhead, paving the way for more adaptable and practical time series modeling solutions in real-world applications.

2.2 Test time scaling

In the era of big data, time series models have become essential tools across industries like finance and healthcare for tasks such as forecasting and anomaly detection. However, as the volume and velocity of time series data increase, efficiently scaling these models during the test phase—when predictions are made on new, unseen data—poses a significant challenge. Traditional methods often struggle with heightened computational demands and potential accuracy losses when processing longer sequences, higher-frequency data, or additional features. This paper presents a novel framework to address test time scaling Sun et al. 2024 in time series models, utilizing adaptive computational techniques to maintain robust performance while preserving efficiency. Through extensive testing on diverse datasets, we showcase substantial gains in both speed and predictive accuracy, delivering a scalable solution for real-world applications. This work pushes forward the field of time series modeling and offers actionable insights for deploying these models in large-scale, real-time settings.

3 Methodology

3.1 PRELIMINARIES

The core structure of RWKV-7 (figure 2) is dynamic evolution WKV state, have time mix and channel mix two main components.

Figure 2: The RWKV-7 architecture is a RNN model that processes sequences using repeated RWKV blocks, each containing:1.A time mix block to blend current and past information.2.WKV heads for attention-like processing with an internal state to maintain memory.A channel mix block to transform the data further.

the core state update rule:

Statet=Statet−1​(d​i​a​g​(wt)−κ^tT​(at⋅κ^t))+vtT⋅k~t⋅at (1)

3.2 Implicit RWKV-7 layers

DEQ (Deep equilibrium models) Bai et al. 2019 models aim to define layers as fixed-point solutions, where the output (or state) of a layer is the equilibrium point of a function, rather than being computed through a traditional sequential update. This is particularly useful for recurrent or transformer-like architectures, as it allows for ”infinite-depth” behavior without explicitly unrolling the recurrence.

In a DEQ framework, our objective is to define Statet (or a related hidden state ht) as the solution to a fixed-point equation of the form:

z∗=f(z∗,x) (2)

where z∗ is the equilibrium state, x represents the input (including previous states and external inputs), and f is a function (often a neural network layer) parameterized by learnable weights. For a recurrent setting, this can be adapted to incorporate temporal dynamics, where the fixed-point equation depends on both the current input and the previous state.

Instead of explicitly computing Statet from Statet−1, we redefine the update implicitly. Let’s denote ht=Statet, and consider the original equation as part of a transformation that we want to express as a fixed-point problem. The original update can be written as:

ht=f⁡(ht,Statet−1,xt) (3)

by introducing a self-recurrent term,

ht=ϕ​(W​ht+V⁡(Statet−1⋅(diag​(wt)−κtT​(at⋅κ^t)))+U⁡(vtT⋅κt⋅at)) (4)

where W,V,U are Learnable weight matrices,Activation ϕ is relu to make non-linear.This DEQ form preserves the core dynamics state while reformulating it as an implicit layer,using latent-space iterations which can greatly improve expressive and efficiency.

4 Evaluation

To evaluate the performance of our proposed Rimer model against the baseline Timer, we conducted experiments across four diverse datasets,ELC, ETTH, Traffic, and Weather using metrics including RMSE, MAE, MAPE, and R², with datasets that are publicly available in our Rimer repository. Our Rimer model, featuring meta-learning in its state update rule and implemented with Triton operators, was trained on a Linux ROCm platform with AMD GPUs (Radeon Pro W7900 for training and Radeon RX6750XT for inference), achieving a 4.5x training time speedup compared to Timer due to its 1.6 million parameters versus the latter’s 37.8 million; this setup highlights our model’s unique strength in broad hardware compatibility, supporting training and inference across AMD GPUs, NVIDIA GPUs, and CPUs, a versatility enabled by Triton operators and developed on the ROCm platform.

RMSE MAE MAPE R2

Timer-37.8M 0.6488 0.2127 0.61% 0.9755

Rimer-1.6M 0.2409 0.0814 0.81% 0.9991

Table 1: ECL

RMSE MAE MAPE R2

Timer-37.8M 0.5770 0.4050 6.5% 0.9968

Rimer-1.6M 0.0133 0.0112 0.16% 0.9998

Table 2: ETTH

RMSE MAE MAPE R2

Timer-37.8M 0.0055 0.0015 19.94% 0.8955

Rimer-1.6M 0.0025 0.0006 4.01% 0.9838

Table 3: Traffic

RMSE MAE MAPE R2

Timer-37.8M 6.1765 3.6839 0.88% 0.8411

Rimer-1.6M 5.4311 1.3621 0.34% 0.8794

Table 4: Weather

The evaluation demonstrates that Rimer-1.6M , despite its significantly smaller size, delivers competitive or superior performance compared to the much larger Timer-37.8M model across multiple time series modeling tasks. This highlights its remarkable efficiency and effectiveness.

These results reveal that Rimer-1.6M, with just 1.6 million parameters, not only competes with but often outperforms Timer-37.8M, which relies on 37.8 million parameters. Its consistently higher R​² values across all datasets demonstrate a superior ability to capture temporal patterns, while its lower MAPE in ETTH and Traffic datasets suggests greater robustness to relative errors, potentially due to better handling of outliers or smaller values. The lightweight architecture of Rimer-1.6M over 23 times smaller than Timer-37.8M, also implies significant computational advantages, such as faster training and inference times, making it highly practical for large-scale applications.

In conclusion, the benchmark underscores Rimer-1.6M as a compelling alternative to traditional transformer-based models like Timer-37.8M. Its ability to achieve strong performance with a fraction of the parameters highlights the potential of RWKV-based architectures for efficient, effective, and scalable time series modeling, addressing both performance demands and resource constraints in real-world forecasting tasks.

5 Conclusion

The integration of RWKV-7 core components into the Timer architecture by replacing its transformer backbone has yielded a significant performance boost, as demonstrated through our experimental evaluations. This enhancement is attributed to the expressive and efficient nature of the RWKV-7 architecture, which, despite being fundamentally a type of recurrent neural network (RNN), effectively combines the strengths of traditional RNNs with the scalability and parallelization benefits typically associated with transformers. This hybrid design enables Rimer to outperform the original Timer across diverse datasets, leveraging its lightweight structure (1.6 million parameters versus 37.8 million) and broad hardware compatibility to achieve a 4.5x training time speedup. These results underscore the potential of RWKV-7 as a promising alternative for large-scale time series modeling, offering a compelling balance of performance, efficiency, and adaptability for future research and real-world applications.

5.1 Future Work

Looking ahead, we plan to focus our future efforts on several key directions to further advance the Rimer model. First, we aim to enhance its capability to handle long contexts by optimizing the state update mechanism to effectively capture extended temporal dependencies, potentially through advanced memory-augmented architectures. Second, we will explore the integration of latent space representations with state chains, allowing the model to learn compact, meaningful abstractions of the input sequence while maintaining state continuity across time steps, which could improve both efficiency and predictive accuracy. Finally, we intend to investigate hybrid models that combine RWKV-7 with other architectures Yueyu et al. 2025, such as transformers or convolutional neural networks, to leverage their complementary strengths for more robust time series forecasting. These explorations will pave the way for next-generation models capable of addressing complex, long-range dependencies and diverse application scenarios.

Acknowledgments

We extend our gratitude to Cao Qihuan for his generous guidance, as well as to Li Zhiyuan and the RWKV community for their contributions to RWKV-FLA.

References

Bai et al. [2019] Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in neural information processing systems, 32, 2019.

Dong et al. [2024] Chengqi Dong, Zhiyuan Cao, S Kevin Zhou, and Jia Liu. Dft: A dual-branch framework of fluctuation and trend for stock price prediction. arXiv preprint arXiv:2411.06065, 2024.

Grazzi et al. [2024] Riccardo Grazzi, Julien Siems, Jörg KH Franke, Arber Zela, Frank Hutter, and Massimiliano Pontil. Unlocking state-tracking in linear rnns through negative eigenvalues. arXiv preprint arXiv:2411.12537, 2024.

Hou and Yu [2024] Haowen Hou and F Richard Yu. Rwkv-ts: Beyond traditional recurrent neural network for time series tasks. arXiv preprint arXiv:2401.09093, 2024.

Khosravi et al. [2023] Maryam Khosravi, Ali Reza Khanteimouri, Seyed Mohammad Reza Azghani, Seyed Mohammad Mehdi Hosseini, and Seyed Mohammad Mehdi Hosseini. Using meta-learning to recommend an appropriate time-series forecasting model. BMC Public Health, 23(1):2432, 2023.

[6] Jinwei Lang, Li-Zhuang Yang, and Hai Li. Multi-modal dynamic brain graph representation learning for brain disorder diagnosis via temporal sequence model. Available at SSRN 5114041.

Liu et al. [2024] Yong Liu, Haoran Zhang, Chenyu Li, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Timer: Generative pre-trained transformers are large time series models, 2024.

Peng et al. [2023] Bo Peng, Bo Li, Wenhan Dai, Shujian Zhang, Jianzhong Qi, Wenjun Zeng, and Xuewei Li. Rwkv: Reinventing rnns for the transformer era, 2023.

Prudencio and Ludermir [2004] Ricardo B. C. Prudencio and Theresa B Ludermir. Meta-learning approaches to selecting time series models. Neurocomputing, 59:145–170, 2004.

Sun et al. [2024] Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, et al. Learning to (learn at test time): Rnns with expressive hidden states. arXiv preprint arXiv:2407.04620, 2024.

[11] Qingyong Wang, Guohao Lv, Lichuan Gu, et al. Chembr: A novel generative model based on bidirectional molecular ring constraints.

Wen et al. [2022] Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Transformers in time series: A survey, 2022.

Yueyu et al. [2025] Lin Yueyu, Li Zhiyuan, Peter Yue, and Liu Xiao. Arwkv: Pretrain is not what we need, an rnn-attention-based language model born from transformer. arXiv preprint arXiv:2501.15570, 2025.

Zhang et al. [2025] Xin Zhang, Liangxiu Han, Stephen White, Saad Hassan, Philip A Kalra, James Ritchie, Carl Diver, and Jennie Shorley. Tabulatime: A novel multimodal deep learning framework for advancing acute coronary syndrome prediction through environmental and clinical data integration. arXiv preprint arXiv:2502.17049, 2025.

By Li weile, Liu Xiao · Text edition under CC BY 4.0 · Original paper