5,545,781 AI resources indexedWorking toward the world’s largest AI resource index.

PDF

An Epistemic and Aleatoric Decomposition of Arbitrariness to Constrain the Set of Good Models

Research paper by Falaah Arif Khan and collaborators. arXiv record dated 2025-07-12. Research area: Machine learning.

Resource information

Authors
Falaah Arif Khan, Denys Herasymuk, Nazar Protsiv, Julia Stoyanovich
Research area
Machine learning
Application domain
General AI research
AI methods
Machine learning
Research topics
cs.LG, cs.AI, cs.CY
arXiv record date
2025-07-12
Publication
arXiv
Upstream last modified
Jul 15, 2025
Last record update
Nov 3, 2026

Abstract

Recent research reveals that machine learning (ML) models are highly sensitive to minor changes in their training procedure, such as the inclusion or exclusion of a single data point, leading to conflicting predictions on individual data points; a property termed as arbitrariness or instability in ML pipelines in prior work. Drawing from the uncertainty literature, we show that stability decomposes into epistemic and aleatoric components, capturing the consistency and confidence in prediction, respectively. We use this decomposition to provide two main contributions. Our first contribution is an extensive empirical evaluation. We find that (i) epistemic instability can be reduced with more training data whereas aleatoric instability cannot; (ii) state-of-the-art ML models have aleatoric instability as high as 79% and aleatoric instability disparities among demographic groups as high as 29% in popular fairness benchmarks; and (iii) fairness pre-processing interventions generally increase aleatoric instability more than in-processing interventions, and both epistemic and aleatoric instability are highly sensitive to data-processing interventions and model architecture. Our second contribution is a practical solution to the problem of systematic arbitrariness. We propose a model selection procedure that includes epistemic and aleatoric criteria alongside existing accuracy and fairness criteria, and show that it successfully narrows down a large set of good models (50-100 on our datasets) to a handful of stable, fair and accurate ones. We built and publicly released a python library to measure epistemic and aleatoric multiplicity in any ML pipeline alongside existing confusion-matrix-based metrics, providing practitioners with a rich suite of evaluation metrics to use to define a more precise criterion during model selection.