Extending LLMs' Context Window with 100 Samples
Research paper by Yikai Zhang and collaborators. Published 2024-01-13. Topics: rotary position embedding, attention entropy, attention scores, Position Interpolation (PI), YaRN.
Resource information
- Authors
- Yikai Zhang, Junlong Li, Pengfei Liu
- Research topics
- rotary position embedding, attention entropy, attention scores, Position Interpolation (PI), YaRN, attention logits, LLaMA, PaLM, GPT-NeoX, fine-tuning performance, data compositions, training curricula
- Publication date
- 2024-01-13
- Publication
- arXiv
- Last record update
- Nov 3, 2026