On Pretraining for Project-Level Code Completion
Research paper by Maksim Sapronov and collaborators. Published 2025-10-15. Topics: repository-level pretraining, code completions, OpenCoder, context window, rotary positional embedding (RoPE).
Resource information
- Authors
- Maksim Sapronov, Evgeniy Glukhov
- Research area
- Machine learning
- Application domain
- General AI research
- AI methods
- Large language models
- Research topics
- cs.SE, cs.LG
- arXiv record date
- 2025-10-15
- Publication date
- 2025-10-15
- Publication
- arXiv
- Upstream last modified
- Oct 16, 2025
- License
- CC BY 4.0
- Last record update
- Nov 3, 2026
Abstract
Repository-level pretraining is commonly used to enable large language models for code to leverage codebase-wide context. This enhances their ability to generate accurate and context-aware code completions. In this work, we investigate how different repository-processing strategies affect in-context learning in OpenCoder, a 1.5B-parameter model. We extend its context window from 4,096 to 16,384 tokens by training on additional 1B tokens of curated repository-level data. Despite relying on a smaller dataset than competing models (which often use hundreds of billions of tokens), our model achieves comparable performance on the Long Code Arena benchmark. We find that various repository-processing techniques yield similarly strong results, with the primary gain coming from adapting to a new rotary positional embedding (RoPE) scaling parameter. Finally, we show that a simpler file-level training approach at the original sequence length remains highly effective, opening up repository-level code completion research to settings with more constrained data and compute resources.