GPT-OSS 20B full fine-tuning across GPUs with FSDP2
A distributed supervised fine-tuning configuration for updating GPT-OSS 20B model weights.
Published by Axolotl
How to use it
The example shards the GPT-OSS 20B model across accelerators using FSDP2 and uses Multilingual-Thinking for supervised training. It configures MXFP4 model handling and the Cut Cross Entropy integration.
The accompanying GPT-OSS README documents the command and measured memory configurations. Reserve storage for checkpoints and run a short checkpoint-and-restart check before committing to a full training run.
- Framework
- Axolotl
- Dataset
- HuggingFaceH4/Multilingual-Thinking
- Hardware
- Publisher examples use 8×48 GB GPUs or 4×80 GB GPUs for this configuration.
- Access
- Public publisher configuration and documented launch command.