DreamTest: World-Model Surrogates for Search-Based Testing of Deep Reinforcement Learning Agents

Resource information

Authors

Qinghua Xu, Guancheng Wang, Boxi Yu, Liting Lin, Lionel Briand

Research area

Machine learning

Application domain

Agent systems

AI methods

Reinforcement learning

Research topics

cs.SE, cs.LG

arXiv record date

2026-10-03

Publication

arXiv

Upstream last modified

Oct 6, 2026

License

CC BY 4.0

Last record update

Nov 4, 2026

Abstract

Testing deep reinforcement learning (DRL) agents in cyber-physical systems aims to uncover diverse failures before deployment, but each execution can be expensive. Surrogate-assisted testing reduces this cost by learning to predict which test configurations are likely to fail. Prior surrogates treat the system as a black box and predict pass or fail outcomes directly; we instead model how a test unfolds and estimate failure from an imagined episode. We introduce DreamTest, a world-model surrogate for testing DRL agents. DreamTest adapts a recurrent state-space model to learn agent behaviour and environment dynamics from the agent's training log. Given a candidate configuration, imagined rollouts produce a failure score that guides search without executing every candidate in a simulator or real system. We evaluate DreamTest for failure prediction, test generation, and failure diversity on Parking, Humanoid, and DonkeyCar. Mean area under the precision-recall curve (AUPRC) exceeds the strongest baseline by 97%, 12%, and 39%, respectively, and gains on five out-of-distribution test sets reach 145%, 29%, and 44%. Under the same simulator-validation budget, the best "DreamTest + search" combinations find 29%, 22%, and 79% more novel failures on average. Across clusterings with k = 2-40, failures generated with DreamTest cover the most behavioural clusters for almost all k, indicating that DreamTest consistently discovers behaviourally diverse failures.