--- name: huggingface-accelerate-e94cea7a174f description: Simplest distributed training API. 4 lines to add distributed support to any PyTorch script. Unified API for DeepSpeed/FSDP/Megatron/DDP. Automatic device placement, mixed precision (FP16/BF16/FP8). Interactive config, single launch command. HuggingFace ecosystem standard. version: 1.0.0 author: Orchestra Research license: MIT tags: [Distributed Training, HuggingFace, Accelerate, DeepSpeed, FSDP, Mixed Precision, PyTorch, DDP, Unified API, Simple] dependencies: [accelerate, torch, transformers] --- # HuggingFace Accelerate - Unified Distributed Training ## Quick start Accelerate simplifies distributed training to 4 lines of code. **Installation**: ```bash pip install accelerate ``` **Convert PyTorch script** (4 lines): ```python import torch + from accelerate import Accelerator + accelerator = Accelerator() model = torch.nn.Transformer() optimizer = torch.optim.Adam(model.parameters()) dataloader = torch.utils.data.DataLoader(dataset) + model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader) for batch in dataloader: optimizer.zero_grad() loss = model(batch) - loss.backward() + accelerator.backward(loss) optimizer.step() ``` **Run** (single command): ```bash accelerate launch train.py ``` ## Common workflows ### Workflow 1: From single GPU to multi-GPU **Original script**: ```python # train.py import torch model = torch.nn.Linear(10, 2).to('cuda') optimizer = torch.optim.Adam(model.parameters()) dataloader = torch.utils.data.DataLoader(dataset, batch_size=32) for epoch in range(10): for batch in dataloader: batch = batch.to('cuda') optimizer.zero_grad() loss = model(batch).mean() loss.backward() optimizer.step() ``` **With Accelerate** (4 lines added): ```python # train.py import torch from accelerate import Accelerator # +1 accelerator = Accelerator() # +2 model = torch.nn.Linear(10, 2) optimizer = torch.optim.Adam(model.parameters()) dataloader = torch.utils.data.DataLoader(dataset, batch_size=32) model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader) # +3 for epoch in range(10): for batch in dataloader: # No .to('cuda') needed - automatic! optimizer.zero_grad() loss = model(batch).mean() accelerator.backward(loss) # +4 optimizer.step() ``` **Configure** (interactive): ```bash accelerate config ``` **Questions**: - Which machine? (single/multi GPU/TPU/CPU) - How many machines? (1) - Mixed precision? (no/fp16/bf16/fp8) - DeepSpeed? (no/yes) **Launch** (works on any setup): ```bash # Single GPU accelerate launch train.py # Multi-GPU (8 GPUs) accelerate launch --multi_gpu --num_processes 8 train.py # Multi-node accelerate launch --multi_gpu --num_processes 16 \ --num_machines 2 --machine_rank 0 \ --main_process_ip $MASTER_ADDR \ train.py ``` ### Workflow 2: Mixed precision training **Enable FP16/BF16**: ```python from accelerate import Accelerator # FP16 (with gradient scaling) accelerator = Accelerator(mixed_precision='fp16') # BF16 (no scaling, more stable) accelerator = Accelerator(mixed_precision='bf16') # FP8 (H100+) accelerator = Accelerator(mixed_precision='fp8') model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader) # Everything else is automatic! for batch in dataloader: with accelerator.autocast(): # Optional, done automatically loss = model(batch) accelerator.backward(loss) ``` ### Workflow 3: DeepSpeed ZeRO integration **Enable DeepSpeed ZeRO-2**: ```python from accelerate import Accelerator accelerator = Accelerator( mixed_precision='bf16', deepspeed_plugin={ "zero_stage": 2, # ZeRO-2 "offload_optimizer": False, "gradient_accumulation_steps": 4 } ) # Same code as before! model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader) ``` **Or via config**: ```bash accelerate config # Select: DeepSpeed → ZeRO-2 ``` **deepspeed_config.json**: ```json { "fp16": {"enabled": false}, "bf16": {"enabled": true}, "zero_optimization": { "stage": 2, "offload_optimizer": {"device": "cpu"}, "allgather_bucket_size": 5e8, "reduce_bucket_size": 5e8 } } ``` **Launch**: ```bash accelerate launch --config_file deepspeed_config.json train.py ``` ### Workflow 4: FSDP (Fully Sharded Data Parallel) **Enable FSDP**: ```python from accelerate import Accelerator, FullyShardedDataParallelPlugin fsdp_plugin = FullyShardedDataParallelPlugin( sharding_strategy="FULL_SHARD", # ZeRO-3 equivalent auto_wrap_policy="TRANSFORMER_AUTO_WRAP", cpu_offload=False ) accelerator = Accelerator( mixed_precision='bf16', fsdp_plugin=fsdp_plugin ) model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader) ``` **Or via config**: ```bash accelerate config # Select: FSDP → Full Shard → No CPU Offload ``` ### Workflow 5: Gradient accumulation **Accumulate gradients**: ```python from accelerate import Accelerator accelerator = Accelerator(gradient_accumulation_steps=4) model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader) for batch in dataloader: with accelerator.accumulate(model): # Handles accumulation optimizer.zero_grad() loss = model(batch) accelerator.backward(loss) optimizer.step() ``` **Effective batch size**: `batch_size * num_gpus * gradient_accumulation_steps` ## When to use vs alternatives **Use Accelerate when**: - Want simplest distributed training - Need single script for any hardware - Use HuggingFace ecosystem - Want flexibility (DDP/DeepSpeed/FSDP/Megatron) - Need quick prototyping **Key advantages**: - **4 lines**: Minimal code changes - **Unified API**: Same code for DDP, DeepSpeed, FSDP, Megatron - **Automatic**: Device placement, mixed precision, sharding - **Interactive config**: No manual launcher setup - **Single launch**: Works everywhere **Use alternatives instead**: - **PyTorch Lightning**: Need callbacks, high-level abstractions - **Ray Train**: Multi-node orchestration, hyperparameter tuning - **DeepSpeed**: Direct API control, advanced features - **Raw DDP**: Maximum control, minimal abstraction ## Common issues **Issue: Wrong device placement** Don't manually move to device: ```python # WRONG batch = batch.to('cuda') # CORRECT # Accelerate handles it automatically after prepare() ``` **Issue: Gradient accumulation not working** Use context manager: ```python # CORRECT with accelerator.accumulate(model): optimizer.zero_grad() accelerator.backward(loss) optimizer.step() ``` **Issue: Checkpointing in distributed** Use accelerator methods: ```python # Save only on main process if accelerator.is_main_process: accelerator.save_state('checkpoint/') # Load on all processes accelerator.load_state('checkpoint/') ``` **Issue: Different results with FSDP** Ensure same random seed: ```python from accelerate.utils import set_seed set_seed(42) ``` ## Advanced topics **Megatron integration**: See [references/megatron-integration.md](references/megatron-integration.md) for tensor parallelism, pipeline parallelism, and sequence parallelism setup. **Custom plugins**: See [references/custom-plugins.md](references/custom-plugins.md) for creating custom distributed plugins and advanced configuration. **Performance tuning**: See [references/performance.md](references/performance.md) for profiling, memory optimization, and best practices. ## Hardware requirements - **CPU**: Works (slow) - **Single GPU**: Works - **Multi-GPU**: DDP (default), DeepSpeed, or FSDP - **Multi-node**: DDP, DeepSpeed, FSDP, Megatron - **TPU**: Supported - **Apple MPS**: Supported **Launcher requirements**: - **DDP**: `torch.distributed.run` (built-in) - **DeepSpeed**: `deepspeed` (pip install deepspeed) - **FSDP**: PyTorch 1.12+ (built-in) - **Megatron**: Custom setup ## Resources - Docs: https://huggingface.co/docs/accelerate - GitHub: https://github.com/huggingface/accelerate - Version: 1.11.0+ - Tutorial: "Accelerate your scripts" - Examples: https://github.com/huggingface/accelerate/tree/main/examples - Used by: HuggingFace Transformers, TRL, PEFT, all HF libraries --- Source attribution: Dicklesworthstone Original project: https://github.com/Dicklesworthstone/pi_agent_rust/blob/f28fc46ee0db8e710d95b8b55592cd38d60f2b26/tests/ext_conformance/artifacts/templates-davila7/cli-tool/components/skills/ai-research/distributed-training-accelerate/SKILL.md License: MIT License information: https://opensource.org/license/mit Original upstream instructions; downloaded from Mega AI Hub. ## Upstream license notice MIT License (with OpenAI/Anthropic Rider) Copyright (c) 2026 Jeffrey Emanuel Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: ADDITIONAL RIDER / RESTRICTION (OpenAI / Anthropic): This rider is part of the "conditions" of this License. In the event of any conflict between this rider and any other portion of this License, this rider controls. "Restricted Parties" means OpenAI, L.L.C.; Anthropic, PBC; any of their respective Affiliates; and any person or entity acting directly or indirectly on behalf of, for the benefit of, or under the direction of any of the foregoing (including any officer, director, employee, contractor, agent, consultant, service provider, or representative). Notwithstanding any other provision of this License, no rights are granted to any Restricted Party. Any purported license, sublicense, assignment, transfer, or other permission to any Restricted Party is null and void absent the express prior written permission of Jeffrey Emanuel. You may not provide, disclose, distribute, sublicense, sell, lease, lend, host, make available, or otherwise permit access to the Software or any derivative work of the Software (as defined in applicable copyright law) (collectively, "Derivative Works") to or for any Restricted Party. For purposes of this rider, "use" includes, without limitation: copying, modifying, merging, publishing, distributing, sublicensing, selling, transferring, making available, hosting, deploying, executing, benchmarking, testing, analyzing, indexing, or incorporating the Software or any Derivative Works into any dataset, training corpus, evaluation harness, or pipeline for machine learning or other automated systems. This rider applies to the Software and all Derivative Works. As a condition of use, you agree that this rider is a precondition to exercising any rights under this License, and you agree that any distribution of the Software or any Derivative Works must include this rider provision unmodified. Any breach of this rider automatically and immediately terminates the permissions granted by this License. Upon termination, you must immediately cease all use and distribution of the Software and any Derivative Works and destroy all copies under your control. You agree that a breach of this rider would cause irreparable harm and that Jeffrey Emanuel may seek injunctive or other equitable relief to enforce this rider, in addition to any other remedies available at law. To the maximum extent permitted by applicable law, the prevailing party in any action to enforce this rider shall be entitled to recover reasonable attorneys' fees and costs. For purposes of this rider, "Affiliate" means any entity that directly or indirectly controls, is controlled by, or is under common control with the specified party. "Control" means ownership of more than 50% of the voting securities or other ownership interest, or the power to direct management or policies by contract or otherwise. The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # Mega AI Hub resource metadata Creator / publisher: Dicklesworthstone Original project: https://github.com/Dicklesworthstone/pi_agent_rust/blob/f28fc46ee0db8e710d95b8b55592cd38d60f2b26/tests/ext_conformance/artifacts/templates-davila7/cli-tool/components/skills/ai-research/distributed-training-accelerate/SKILL.md Attribution basis: Source repository record Fields of work: Data & Analytics, Development & Engineering Roles: Backend Developer, Data Scientist, ML / AI Engineer Specialties: Backend Tools: Deepspeed, Pytorch Country applicability: Country not specified Classification basis: Source catalog labels Instruction SHA-256: d0fa9be09db9281b3765279186f6ad0ffcbad6f7b440c98c29e5999833513198