Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMs
Research paper by Aly M. Kassem and collaborators. arXiv record dated 2025-02-09. Research area: Language technology.
Resource information
- Authors
- Aly M. Kassem, Omar Mahmoud, Niloofar Mireshghallah, Hyunwoo Kim, Yulia Tsvetkov, Yejin Choi, Sherif Saad, Santu Rana
- Research area
- Language technology
- Application domain
- General AI research
- AI methods
- Large language models
- Research topics
- cs.CL
- arXiv record date
- 2025-02-09
- Publication
- arXiv
- Upstream last modified
- Feb 11, 2025
- License
- CC BY 4.0
- Last record update
- Nov 4, 2026
Abstract
In this paper, we introduce a black-box prompt optimization method that uses an attacker LLM agent to uncover higher levels of memorization in a victim agent, compared to what is revealed by prompting the target model with the training data directly, which is the dominant approach of quantifying memorization in LLMs. We use an iterative rejection-sampling optimization process to find instruction-based prompts with two main characteristics: (1) minimal overlap with the training data to avoid presenting the solution directly to the model, and (2) maximal overlap between the victim model's output and the training data, aiming to induce the victim to spit out training data. We observe that our instruction-based prompts generate outputs with 23.7% higher overlap with training data compared to the baseline prefix-suffix measurements. Our findings show that (1) instruction-tuned models can expose pre-training data as much as their base-models, if not more so, (2) contexts other than the original training data can lead to leakage, and (3) using instructions proposed by other LLMs can open a new avenue of automated attacks that we should further study and explore. The code can be found at https://github.com/Alymostafa/Instruction_based_attack .