Gemma 2 2B distributed full fine-tuning
A torchtune example for updating the full Gemma 2 2B checkpoint on multiple CUDA GPUs.
Published by PyTorch torchtune
How to use it
The YAML loads a local Gemma 2 checkpoint and tokenizer and uses Alpaca Cleaned for training and validation. It enables activation checkpointing to reduce memory demand.
Set the checkpointer directory and tokenizer path before running full_finetune_distributed with the published configuration. Use a separate validation set if your work needs independent performance estimates.
- Framework
- PyTorch torchtune
- Dataset
- Alpaca Cleaned via torchtune.datasets.alpaca_cleaned_dataset.
- Hardware
- The publisher requires at least two GPUs for this configuration and shows a four-device CUDA launch.
- Access
- Public publisher configuration; local checkpoint required.