--- name: cleanrl-fbeb1df11743 description: Single-file deep reinforcement learning implementations (CleanRL). High-quality standalone implementations of PPO, DQN, C51, SAC, DDPG, TD3 with research-friendly features. Each algorithm is a self-contained file with ~300-500 lines. Includes Atari, MuJoCo, Procgen, PettingZoo multi-agent, and JAX variants. Use for RL algorithm reference, rapid prototyping, and understanding implementation details. license: MIT license tags: [ppo, dqn, sac, rl-reference-implementation, cleanrl] metadata: skill-author: K-Dense Inc. --------|------|----------------| | **PPO** | `ppo.py` | `python cleanrl/ppo.py --env-id CartPole-v1` | | **PPO Atari** | `ppo_atari.py` | `python cleanrl/ppo_atari.py --env-id BreakoutNoFrameskip-v4` | | **PPO Continuous** | `ppo_continuous_action.py` | `python cleanrl/ppo_continuous_action.py --env-id HalfCheetah-v4` | | **PPO Multi-Agent** | `ppo_pettingzoo_ma_atari.py` | `python cleanrl/ppo_pettingzoo_ma_atari.py --env-id pong_v3` | | **DQN** | `dqn.py` | `python cleanrl/dqn.py --env-id CartPole-v1` | | **DQN Atari** | `dqn_atari.py` | `python cleanrl/dqn_atari.py --env-id BreakoutNoFrameskip-v4` | | **C51 Atari** | `c51_atari.py` | `python cleanrl/c51_atari.py --env-id BreakoutNoFrameskip-v4` | | **SAC Continuous** | `sac_continuous_action.py` | `python cleanrl/sac_continuous_action.py --env-id HalfCheetah-v4` | | **SAC Atari** | `sac_atari.py` | `python cleanrl/sac_atari.py --env-id BreakoutNoFrameskip-v4` | | **DDPG** | `ddpg_continuous_action.py` | `python cleanrl/ddpg_continuous_action.py --env-id HalfCheetah-v4` | | **TD3** | `td3_continuous_action.py` | `python cleanrl/td3_continuous_action.py --env-id HalfCheetah-v4` | ### 3. PPO Training Workflow ```bash # Minimal PPO on CartPole python cleanrl/ppo.py \ --seed 1 \ --env-id CartPole-v1 \ --total-timesteps 50000 \ --track \ --wandb-project-name my-project # PPO on Atari (standard config) python cleanrl/ppo_atari.py \ --seed 1 \ --env-id BreakoutNoFrameskip-v4 \ --total-timesteps 10000000 \ --track \ --capture-video # PPO on MuJoCo continuous control python cleanrl/ppo_continuous_action.py \ --seed 1 \ --env-id HalfCheetah-v4 \ --total-timesteps 1000000 ``` **Key PPO Hyperparameters:** | Parameter | CartPole/Classic | Atari | MuJoCo | |-----------|-----------------|-------|--------| | `--total-timesteps` | 50K | 10M | 1M | | `--learning-rate` | 2.5e-4 | 2.5e-4 | 3e-4 | | `--num-envs` | 4 | 8 | 1 | | `--num-steps` | 128 | 128 | 2048 | | `--anneal-lr` | True | True | False | | `--gae-lambda` | 0.95 | 0.95 | 0.95 | | `--update-epochs` | 4 | 4 | 10 | | `--norm-adv` | True | True | True | | `--clip-coef` | 0.2 | 0.1 | 0.2 | | `--ent-coef` | 0.01 | 0.01 | 0.0 | ### 4. DQN Training ```bash # DQN on Atari python cleanrl/dqn_atari.py \ --seed 1 \ --env-id BreakoutNoFrameskip-v4 \ --total-timesteps 10000000 \ --buffer-size 100000 \ --learning-starts 80000 \ --target-network-frequency 1000 \ --batch-size 32 \ --track ``` ### 5. Multi-Agent RL with PettingZoo ```bash # PPO on multi-agent Atari Pong python cleanrl/ppo_pettingzoo_ma_atari.py \ --seed 1 \ --env-id pong_v3 \ --total-timesteps 10000000 \ --track # Available MA environments: # pong_v3, surround_v2, tennis_v3, space_invaders_v2, # warlords_v3, combat_plane_v2, combat_tank_v2 ``` ### 6. Logging and Monitoring ```bash # TensorBoard (runs in cleanrl/runs/) tensorboard --logdir runs # Weights & Biases (requires wandb login) python cleanrl/ppo.py --track --wandb-project-name my-project --wandb-entity my-entity # Video capture (every 100th evaluation) python cleanrl/ppo_atari.py --capture-video --env-id BreakoutNoFrameskip-v4 ``` ### 7. JAX-Accelerated Variants 5-10x faster training via JAX compilation + EnvPool: ```bash # Install JAX support pip install -r requirements/requirements-jax.txt # JAX PPO on Atari (ultra-fast) python cleanrl/ppo_atari_envpool_xla_jax.py \ --env-id BreakoutNoFrameskip-v4 \ --total-timesteps 10000000 # JAX DQN on Atari python cleanrl/dqn_atari_jax.py \ --env-id BreakoutNoFrameskip-v4 \ --total-timesteps 10000000 ``` ### 8. Docker and Cloud (AWS) ```bash # Build Docker image docker build -t cleanrl . # Submit to AWS Batch python cleanrl/ppo_atari.py \ --env-id BreakoutNoFrameskip-v4 \ --total-timesteps 10000000 \ --track \ --upload-model ``` ### 9. Algorithm Structure (Reading an Implementation) Each file follows a consistent structure: ```python # 1. Imports # 2. parse_args() — CLI arguments # 3. make_env() — Environment creation # 4. Agent class (if needed) — Neural network, usually simple MLP/CNN # 5. main(): # a. Setup: seeding, device, envs # b. Initialize agent, optimizer # c. Initialize storage (rollout buffer, replay buffer) # d. Training loop: # - Collect experience # - Compute returns/advantages # - Update policy/value/Q-network # - Log metrics # e. Save model, upload ``` Each file is ~300-500 lines and is meant to be read top-to-bottom. ### 10. Debugging and Development ```bash # Minimal test run (fewer steps, more frequent logging) python cleanrl/ppo.py \ --env-id CartPole-v1 \ --total-timesteps 5000 \ --num-envs 1 \ --num-steps 32 \ --track # Disable wandb (pure TensorBoard) python cleanrl/ppo.py --env-id CartPole-v1 --total-timesteps 50000 # Check available env IDs python -c "import gymnasium as gym; print([e for e in gym.envs.registry if 'CartPole' in e])" ``` ## Key Patterns 1. **CleanRL is NOT a library** — don't `import cleanrl`, run the scripts directly 2. **Each file is self-contained** — copy `ppo.py` and modify it for your research 3. **Use `--track` for W&B logging**, omit for plain TensorBoard 4. **`--capture-video` saves agent gameplay** — great for qualitative evaluation 5. **JAX variants are fastest** but require understanding of `jax.lax.scan` 6. **All implementations are benchmarked** — see https://benchmark.cleanrl.dev ## References - [CleanRL Documentation](https://docs.cleanrl.dev/) - [Algorithm Benchmarks](https://benchmark.cleanrl.dev/) - [JMLR Paper](https://www.jmlr.org/papers/volume23/21-1342/21-1342.pdf) - [CORL (offline RL fork)](https://github.com/corl-team/CORL) - [LeanRL (optimized PyTorch fork)](https://github.com/pytorch-labs/LeanRL) --- Source attribution: mkurman Original project: https://github.com/mkurman/zorai/blob/d06ee7597049567dab92db22d41b2764cf89094a/skills/scientific-skills/cleanrl/SKILL.md License: MIT License information: https://opensource.org/license/mit Original upstream instructions; downloaded from Mega AI Hub. ## Upstream license notice MIT License Copyright (c) 2026 zorai contributors Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # Mega AI Hub resource metadata Creator / publisher: mkurman Original project: https://github.com/mkurman/zorai/blob/d06ee7597049567dab92db22d41b2764cf89094a/skills/scientific-skills/cleanrl/SKILL.md Attribution basis: Source repository record Fields of work: Data & Analytics, Development & Engineering, Education & Research Roles: Data Scientist, ML / AI Engineer, Researcher Specialties: API Tools: Jax Country applicability: Country not specified Classification basis: Source catalog labels Instruction SHA-256: 2bdc240841257d70e9720f397ce9c829561f70b1425a0cdc80e68d90050a71db