Distill Qwen2.5 14B into a Qwen2.5 7B adapter
A single-device teacher–student recipe that uses Qwen2.5 14B guidance to train a 7B student adapter.
Published by PyTorch torchtune
How to use it
The YAML points at separate student and teacher checkpoints and configures torchtune's Alpaca Cleaned dataset loader. The teacher supplies the signal used to train the student's LoRA parameters.
Set the checkpoint and tokenizer paths to your local copies. The header gives the tune run invocation for knowledge_distillation_single_device, along with a separate optional teacher fine-tuning step.
- Framework
- PyTorch torchtune
- Dataset
- Alpaca Cleaned via torchtune.datasets.alpaca_cleaned_dataset.
- Hardware
- Single CUDA device in the configuration. Minimum VRAM unspecified.
- Access
- Public publisher configuration; local checkpoints required.