Efficient Data Selection at Scale via Influence Distillation
Research paper by Mahdi Nikdan and collaborators. Published 2025-05-25. Topics: Influence Distillation, second-order information, model-specific weights, Gradient Descent, Adam optimizers.
Resource information
- Authors
- Mahdi Nikdan, Vincent Cohen-Addad, Dan Alistarh, Vahab Mirrokni
- Research area
- Language technology
- Application domain
- General AI research
- AI methods
- Large language models
- Research topics
- cs.CL, cs.LG
- arXiv record date
- 2025-05-25
- Publication date
- 2025-05-25
- Publication
- arXiv
- Upstream last modified
- May 27, 2025
- Last record update
- Nov 3, 2026
Abstract
Effective data selection is critical for efficient training of modern Large Language Models (LLMs). This paper introduces Influence Distillation, a novel, mathematically-justified framework for data selection that employs second-order information to optimally weight training samples. By distilling each sample's influence on a target distribution, our method assigns model-specific weights that are used to select training data for LLM fine-tuning, guiding it toward strong performance on the target domain. We derive these optimal weights for both Gradient Descent and Adam optimizers. To ensure scalability and reduce computational cost, we propose a $landmark-based approximation$: influence is precisely computed for a small subset of "landmark" samples and then efficiently propagated to all other samples to determine their weights. We validate Influence Distillation by applying it to instruction tuning on the Tulu V2 dataset, targeting a range of tasks including GSM8k, SQuAD, and MMLU, across several models from the Llama and Qwen families. Experiments show that Influence Distillation matches or outperforms state-of-the-art performance while achieving up to $3.5\times$ faster selection.