Llama 3.1 long-context tuning with ring context parallelism
A multi-GPU recipe for distributing long training sequences across GPUs with ring context parallelism.
Published by Axolotl
How to use it
The example uses a Llama 3.1 8B checkpoint and Alpaca with the Ringmaster attention path. Its comments document a PyTorch requirement and a launch on eight GPUs.
Substitute data with realistic long examples when testing a long-context use case. Track sequence length, communication overhead, and tokens per second before scaling the run.
- Framework
- Axolotl
- Dataset
- tatsu-lab/alpaca
- Hardware
- Eight GPUs and PyTorch 2.13 or newer per the example. Minimum VRAM unspecified.
- Access
- Public publisher configuration with launch instructions.