--- name: nemo-rl-e2e-testing-d11394529d73 description: External NeMo-RL end-to-end validation workflow for Megatron-Bridge model/provider changes, including downstream compatibility checks, external RL lifecycle behavior, Megatron policy setup, HF import/export, checkpoint/resume, non-colocated vLLM refit, delta weight transfer, optional LoRA/generation variants, and questions such as "does this model work in NeMo-RL", "run NeMo-RL e2e", or "external RL loop validation". Covers running NeMo-RL Megatron policy jobs from a Bridge checkout, choosing GRPO/SFT/checkpoint/non-colocated refit variants, setting PYTHONPATH so NeMo-RL imports the local Bridge tree, and reporting pass/fail evidence. when_to_use: Adding or changing a Megatron-Bridge model/provider and needing downstream NeMo-RL compatibility validation; checking non-vanilla Bridge provider paths; testing PEFT/LoRA, checkpoint behavior, non-colocated vLLM refit, or explicitly requested advanced variants through NeMo-RL; 'does this model work in NeMo-RL', 'run NeMo-RL e2e', 'external RL loop validation'. --- # NeMo-RL E2E Testing Validate a Megatron-Bridge model or training API change through NeMo-RL's Megatron backend. This catches integration issues that Bridge-only tests miss: NeMo-RL-owned rollout scheduling, reward handling, policy/reference setup, HF import/export through Bridge, optimizer setup, checkpoint ownership, and policy-to-generation weight transfer. Use this as an external compatibility smoke test after the focused Bridge tests for the model/provider change pass. This is not a replacement for Bridge model parity tests. A NeMo-RL GRPO or SFT run proves that Bridge can survive an external RL training loop; architecture correctness still comes from Bridge import/export, logits, roundtrip, and model-specific inference tests. ## Scope Think in coverage levels. Start with Level 0 and add only the levels justified by the change. | Level | Required when | What it proves | |---|---|---| | 0: Megatron policy GRPO smoke | Any new provider or provider config change that claims NeMo-RL compatibility | NeMo-RL can import the local Bridge provider, build a Megatron policy, initialize optimizer/scheduler state, run rollout/ref/logprob wiring, and finish a short GRPO job | | 1: LoRA/checkpoint variant | Checkpointing, HF export, optimizer state, resume behavior, or a NeMo-RL-supported PEFT path changed | NeMo-RL can save through its checkpoint schedule, resume without losing training state, and, when PEFT is enabled in that NeMo-RL checkout, apply Bridge LoRA hooks | | 2: Non-colocated vLLM refit | HF export, weight mapping, policy-to-generation refit, delta compression, packed transfer, or vLLM update behavior changed | Bridge-exported weights can be transferred from the Megatron policy worker into separate vLLM generation workers | | 3: Optional Megatron generation backend | Only when the NeMo-RL checkout still supports `policy.generation.backend=megatron` and the change explicitly targets that path | NeMo-RL can use Megatron for both policy and generation rather than only vLLM generation | | 4: Parallelism stress | TP/PP/CP/EP, sequence parallel, MoE dispatch, pipeline stage layout, or distributed optimizer behavior changed | Provider settings remain correct under non-trivial Megatron parallel state | | 5: Architecture-specific e2e | VLM, audio, MoE, MTP/draft models, FP8/QAT/ModelOpt, quantized weights, or custom layers are involved | The architecture-specific runtime path is exercised, not just a text-only dense GRPO smoke | | 6: Learning signal | Optimizer, scheduler, loss, reward, PEFT trainability, gradient flow, or training stability changed | Metrics move in the expected direction over a short run and do not silently produce zero/NaN/unstable updates | The default Level 0 target is NeMo-RL's maintained Megatron GRPO functional: ```bash uv run bash tests/functional/grpo_megatron.sh ``` This is intentionally small. It exercises NeMo-RL's external RL loop without making Megatron-Bridge own rollout scheduling, rewards, checkpoint cadence, or trainer state. Level 0 is not a convergence test. It only proves the job can complete a small number of updates. Use Level 6 when the question is whether the model actually learns under NeMo-RL. ## Repos Use explicit repo variables. Do not rely on an installed `megatron-bridge` wheel; the purpose is to test the current Bridge checkout. Use the upstream NeMo-RL repository as the default source: ```text https://github.com/NVIDIA-NeMo/RL ``` If a checkout is not already available, clone it next to the Bridge checkout or into the site's standard workspace: ```bash git clone https://github.com/NVIDIA-NeMo/RL.git /path/to/nemo-rl ``` ```bash export BRIDGE_REPO=${BRIDGE_REPO:-/path/to/Megatron-Bridge} export NEMO_RL_REPO=${NEMO_RL_REPO:-/path/to/nemo-rl} export PYTHONPATH="${BRIDGE_REPO}/src:${BRIDGE_REPO}/3rdparty/Megatron-LM:${NEMO_RL_REPO}:${PYTHONPATH:-}" ``` NeMo-RL checkouts often also contain a vendored Bridge tree under: ```text 3rdparty/Megatron-Bridge-workspace/Megatron-Bridge ``` When testing a local Bridge change, either put the local Bridge checkout ahead of everything else in `PYTHONPATH`, or sync the exact local Bridge changes into that vendored checkout. Do not assume the vendored tree matches the Bridge PR under test. Before running, record both states: ```bash git -C "$BRIDGE_REPO" status --short git -C "$NEMO_RL_REPO" status --short git -C "$BRIDGE_REPO" rev-parse --short HEAD git -C "$NEMO_RL_REPO" rev-parse --short HEAD ``` If testing on a remote GPU machine, sync the exact local changes first. Do not reset or overwrite unrelated changes in either tree. Verify that Python imports the checkout under test: ```bash python - <<'PY' import megatron.bridge print(megatron.bridge.__file__) PY ``` The printed path must live under `$BRIDGE_REPO/src`, or under the NeMo-RL vendored Bridge checkout only if that vendored checkout was intentionally synced to the Bridge change. If it points at site-packages or an unexpected 3rdparty path, fix `PYTHONPATH` before trusting any result. ## Bridge Checks First Run focused Bridge tests before the external NeMo-RL e2e. Include any model-specific tests added by the change. ```bash cd "$BRIDGE_REPO" uv run python -m pytest -q \ tests/unit_tests/models/test_model_provider_mixin.py \ tests/unit_tests/models/test_param_mapping.py \ tests/unit_tests/training/test_integration.py \ ``` For a new model family, also run the relevant conversion or roundtrip test from the model's PR. See @skills/adding-model-support/tests-and-examples.md for model-test patterns. Minimum Bridge-side evidence for a new model/provider: - provider/config unit tests - parameter mapping tests - HF to Megatron import or roundtrip on a small model - model-specific generation or logits comparison when available - this NeMo-RL external-loop smoke after the above pass ## NeMo-RL Unit Checks Run the NeMo-RL unit checks that match the surface being exercised. Keep them focused; the e2e is the expensive signal. ```bash cd "$NEMO_RL_REPO" uv run pytest -q \ tests/unit/models/megatron/test_megatron_setup.py \ tests/unit/models/policy/test_megatron_worker.py \ tests/unit/utils/test_weight_transfer.py ``` For checkpoint changes, add: ```bash uv run pytest -q \ tests/unit/utils/test_checkpoint.py \ tests/unit/utils/test_native_checkpoint.py ``` For vLLM refit or generation-worker changes, add the relevant vLLM unit tests: ```bash uv run pytest -q \ tests/unit/models/generation/test_vllm_generation.py \ tests/unit/models/generation/test_vllm_utils.py ``` ## Model Choice Prefer the smallest public HF checkpoint that uses the changed provider family. The maintained Megatron GRPO functional uses `Qwen/Qwen2.5-0.5B` because it is small enough for a 2-GPU smoke and is supported by NeMo-RL's Megatron path. If there is no small public checkpoint for the new architecture, use the closest NeMo-RL recipe that constructs the model with a minimal config or small local checkpoint, and report that the run validates construction/training mechanics rather than pretrained weight compatibility. For VLM or audio models, a text-only GRPO smoke is not enough. Pair the Level 0 policy smoke with the relevant NeMo-RL VLM/audio functional, for example: ```bash uv run bash tests/functional/vlm_grpo.sh uv run bash tests/functional/audio_grpo_megatron.sh ``` For MoE models, Level 0 with trivial expert parallelism catches many provider issues, but it does not stress expert routing. Add a Level 4 run with expert parallelism when the change touches expert layout, dispatcher config, router behavior, or expert tensor parallelism. For MTP/draft models, use an Eagle/MTP-specific functional: ```bash uv run bash tests/functional/grpo_megatron_eagle3_online.sh ``` For FP8/QAT/ModelOpt or quantized checkpoint support, use the closest recipe or functional that explicitly enables the feature. Do not claim the generic GRPO smoke validated quantization unless the config turns it on. ## Environment Setup Use the NeMo-RL development environment or the site-approved NeMo-RL container. Make caches explicit on shared clusters: ```bash export HF_HOME=${HF_HOME:-/scratch/$USER/nemo_rl_hf} export HF_HUB_CACHE=$HF_HOME/hub export NEMO_RL_HOME=${NEMO_RL_HOME:-$NEMO_RL_REPO} export PYTHONPATH="${BRIDGE_REPO}/src:${BRIDGE_REPO}/3rdparty/Megatron-LM:${NEMO_RL_REPO}:${PYTHONPATH:-}" ``` If the container has a dependency fingerprint mismatch, note it in the report. Prefer rebuilding the container or virtualenv when possible; use environment overrides only as test-environment evidence, not repository changes. If model downloads fail with `No space left on device`, move `HF_HOME`, `HF_HUB_CACHE`, and any local `MODEL_PATH` to a larger shared or node-local path. If Hugging Face API calls fail with rate limits after the model is already cached, point both the model and tokenizer at the local snapshot and run offline: ```bash export MODEL_PATH=/scratch/$USER/hf/hub/models----/snapshots/ export HF_HOME=/scratch/$USER/hf export HF_HUB_CACHE=$HF_HOME/hub export HF_HUB_OFFLINE=1 export TRANSFORMERS_OFFLINE=1 ``` Then pass both overrides to NeMo-RL: ```bash policy.model_name="$MODEL_PATH" \ policy.tokenizer.name="$MODEL_PATH" ``` Before trusting the snapshot, verify it loads locally: ```bash uv run python - <<'PY' from transformers import AutoConfig, AutoTokenizer path = "" config = AutoConfig.from_pretrained(path, trust_remote_code=True, local_files_only=True) tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True, local_files_only=True) print(type(config).__name__, getattr(config, "model_type", None), type(tokenizer).__name__, tokenizer.vocab_size) PY ``` ## Minimal NeMo-RL Run Use NeMo-RL's maintained functional wrapper for the default smoke: ```bash cd "$NEMO_RL_REPO" ray stop --force || true export PYTHONPATH="${BRIDGE_REPO}/src:${BRIDGE_REPO}/3rdparty/Megatron-LM:${NEMO_RL_REPO}:${PYTHONPATH:-}" uv run bash tests/functional/grpo_megatron.sh ``` The wrapper writes: ```text tests/functional/grpo_megatron/run.log tests/functional/grpo_megatron/metrics.json ``` Capture the exact command and keep the log path. Prefer a saved log over a pasted terminal excerpt in PR descriptions. If the test needs a different provider or model, pass Hydra overrides through the wrapper: ```bash uv run bash tests/functional/grpo_megatron.sh \ policy.model_name= \ policy.megatron_cfg.converter_type= ``` Keep the first smoke small. Increase model size or parallelism only after a small run proves the basic path works. ## LoRA And Checkpoint Coverage Use Level 1 when the change touches checkpoint save/load, HF export, optimizer state, resume behavior, or a NeMo-RL PEFT path that is known to work in the checkout being tested. NeMo-RL PEFT support is backend- and revision-dependent. Do not block a provider-only compatibility smoke solely on a known-broken or unsupported NeMo-RL PEFT path. In that case, record Level 1 PEFT as not applicable or blocked by NeMo-RL, keep the Level 0 GRPO smoke as the required downstream signal, and cover Bridge PEFT behavior with focused Bridge tests. LoRA + checkpoint save smoke, when the NeMo-RL checkout supports this path: ```bash uv run bash tests/functional/grpo_megatron_lora.sh ``` SFT resume parity across dtensor and Megatron policy paths: ```bash uv run bash tests/functional/sft_resume_diamond.sh ``` The LoRA functional intentionally saves checkpoints. Remove stale checkpoint outputs between unrelated experiments, but keep them while validating resume behavior. Do not claim PEFT coverage from `grpo_megatron.sh`; use the LoRA functional or an equivalent Hydra override with `policy.megatron_cfg.peft.enabled=true`. ## Non-Colocated vLLM Refit Use Level 2 when the change touches Bridge HF export, parameter mapping, NeMo-RL weight refit, packed tensor transfer, vLLM loading, delta compression, or policy/generation worker synchronization. Small 2-GPU non-colocated smoke with the Megatron policy backend: ```bash cd "$NEMO_RL_REPO" uv run coverage run -a --data-file=tests/.coverage --source=nemo_rl \ examples/run_grpo.py \ --config examples/configs/grpo_math_1B_megatron.yaml \ policy.model_name=Qwen/Qwen2.5-0.5B \ grpo.num_prompts_per_step=2 \ grpo.num_generations_per_prompt=4 \ policy.train_global_batch_size=4 \ policy.train_micro_batch_size=1 \ policy.logprob_batch_size=4 \ policy.generation.colocated.enabled=false \ policy.generation.colocated.resources.gpus_per_node=1 \ policy.generation.vllm_cfg.async_engine=true \ cluster.gpus_per_node=2 \ grpo.max_num_steps=2 \ logger.tensorboard_enabled=true \ logger.log_dir=tests/functional/grpo_megatron_non_colocated/logs \ logger.wandb_enabled=false \ checkpointing.enabled=false ``` After the run, dump metrics: ```bash uv run tests/json_dump_tb_logs.py \ tests/functional/grpo_megatron_non_colocated/logs \ --output_path tests/functional/grpo_megatron_non_colocated/metrics.json ``` Metric assertion helpers differ across NeMo-RL revisions. Inspect `tests/check_metrics.py` or the maintained functional wrapper before assuming an interface. Some checkouts expect positional expressions: ```bash uv run tests/check_metrics.py tests/functional/grpo_megatron_non_colocated/metrics.json \ 'max(data["train/token_mult_prob_error"]) < 1.05' \ 'min(data["train/probs_ratio_clamped_min"]) > 0.79' \ 'max(data["train/probs_ratio_clamped_max"]) < 1.21' ``` For delta-compression testing, add these overrides: ```bash policy.generation.delta_compression.enabled=true \ policy.generation.delta_compression.dtype=bfloat16 \ policy.generation.delta_compression.transport=sparse_indices \ policy.generation.delta_compression.full_sync_interval=20 \ policy.generation.delta_compression.sparse_bucket_size_bytes=5368709120 \ policy.generation.delta_compression.delta_load_batch_size_bytes=536870912 ``` Report weight-transfer timing metrics when available, especially: - `timing/train/prepare_for_generation/total` - `timing/train/prepare_for_generation/transfer_and_update_weights` - `timing/train/prepare_for_generation/weight_transfer/producer/collect_tensors` - `timing/train/prepare_for_generation/weight_transfer/producer/sparse_encode` - `timing/train/prepare_for_generation/weight_transfer/producer/sparse_nonzero` - `timing/train/prepare_for_generation/weight_transfer/consumer/decode_sparse` - `timing/train/prepare_for_generation/weight_transfer/consumer/load_delta` If the payload broadcast time is tiny but sparse encode/decode dominates, report that boundary clearly. It is a weight-preparation bottleneck, not a NCCL broadcast bottleneck. ## Megatron Generation Backend Use Level 3 only when the NeMo-RL checkout under test supports the Megatron generation backend and the Bridge change explicitly affects that downstream path. Do not require this for normal provider compatibility, HF import/export, vLLM-backed generation, or generic Bridge inference tests. ```bash uv run bash tests/functional/grpo_megatron_generation.sh ``` This exercises `policy.generation.backend=megatron`, so it validates NeMo-RL's Megatron generation construction and runtime behavior more directly than the default vLLM-backed GRPO functional. Some NeMo-RL revisions declare `mcore` and `vllm` extras as mutually incompatible. In that environment, a vLLM-backed Level 0 run may be blocked even though the Megatron policy path is testable. Use `policy.generation.backend=megatron` for a Megatron-only smoke, record vLLM as skipped or blocked, and do not claim non-colocated vLLM refit coverage. ## Parallelism Stress Use Level 4 when provider finalization, model-parallel settings, sequence parallel, context parallel, MoE dispatch, pipeline layout, or distributed optimizer behavior changed. Start from a maintained recipe that already matches the intended GPU count. For example, use one of the recipe configs under: ```text examples/configs/recipes/llm/*megatron*.yaml examples/configs/recipes/llm/performance/*megatron*.yaml examples/configs/recipes/vlm/*megatron*.yaml ``` For a small manual stress variant, override the Megatron sizes explicitly: ```bash uv run bash tests/functional/grpo_megatron.sh \ policy.megatron_cfg.tensor_model_parallel_size=2 \ policy.megatron_cfg.pipeline_model_parallel_size=1 \ policy.megatron_cfg.context_parallel_size=1 \ policy.megatron_cfg.sequence_parallel=false \ cluster.gpus_per_node=2 ``` For MoE, use a MoE recipe and set expert parallelism only when the model and GPU count support it: ```bash policy.megatron_cfg.expert_model_parallel_size=2 \ policy.megatron_cfg.expert_tensor_parallel_size=1 ``` Keep these as follow-up runs. Do not make them the first debugging surface for a new provider. ## Learning Signal Use Level 6 only when the change affects trainability or when downstream validation explicitly asks for learning behavior. Do not require it for every provider-only PR; RL learning is slower, noisier, and more environment-dependent than compatibility smoke tests. The goal is a short learning-signal run, not a benchmark. Prefer a small model, fixed data, fixed seed when available, and enough steps to observe non-random metric movement: ```bash uv run bash tests/functional/grpo_megatron_lora.sh \ grpo.max_num_steps=20 \ data.shuffle=false \ checkpointing.enabled=false ``` Acceptable learning-signal evidence depends on the task, but the report should include at least: - no NaNs or infs in loss, reward, KL, entropy, grad norm, or logprob metrics - nonzero trainable parameter count when PEFT is enabled - actor losses and reward-related metrics logged for multiple steps - validation or reward trend compared against the starting point or a known-good baseline - no repeated zero gradients, frozen LoRA adapters, or constant logprobs unless expected Do not call a 20-step run "converged" in the benchmark sense. Call it "learning-signal passed" unless it reaches a pre-agreed metric threshold. ## Slurm Or Container Runs Use the cluster's standard NeMo-RL container and mount both checkouts into the container. Keep setup and the actual run in the same container step when using node-local paths such as `/tmp`; node-local model caches and ad-hoc installs disappear when a fresh container step starts. If the home filesystem is full or Megatron-Core tries to build helper extensions into a read-only/full checkout, copy the MCore submodule to node-local storage and put that copy on `PYTHONPATH` instead of editing `3rdparty/Megatron-LM/`: ```bash export MCORE_REPO=${MCORE_REPO:-/tmp/$USER/Megatron-LM} if [[ ! -d "$MCORE_REPO/.git" ]]; then cp -a "$BRIDGE_REPO/3rdparty/Megatron-LM" "$MCORE_REPO" fi EXT_SUFFIX=$(uv run python - <<'PY' import sysconfig print(sysconfig.get_config_var("EXT_SUFFIX") or ".so") PY ) make -C "$MCORE_REPO/megatron/core/datasets" LIBEXT="$EXT_SUFFIX" export PYTHONPATH="${BRIDGE_REPO}/src:${MCORE_REPO}:${NEMO_RL_REPO}:${PYTHONPATH:-}" ``` Overriding `LIBEXT` avoids a suffixless `helpers_cpp` binary on containers where `python3-config` is absent from `PATH`. Verify the built file is named like `helpers_cpp.cpython--.so` before launching a long run. For NeMo-RL multi-node jobs, prefer NeMo-RL's own `ray.sub` launcher when it is available. It starts the Ray head and worker nodes under Slurm, mounts the requested container/filesystems, and executes `COMMAND` from the NeMo-RL root. Launch it from `$NEMO_RL_REPO`, not from the Bridge checkout: ```bash cd "$NEMO_RL_REPO" COMMAND="uv run ./examples/run_grpo.py \ --config examples/configs/grpo_math_1B_megatron.yaml \ cluster.num_nodes=2 \ cluster.gpus_per_node=8 \ logger.log_dir=results/grpo_megatron_2n \ logger.wandb_enabled=false" \ CONTAINER="$NEMO_RL_IMAGE" \ MOUNTS="$BRIDGE_REPO:$BRIDGE_REPO,$NEMO_RL_REPO:$NEMO_RL_REPO,$HF_HOME:$HF_HOME" \ sbatch \ --nodes=2 \ --account= \ --partition= \ --job-name=nemo-rl-bridge-e2e \ --time=4:00:00 \ --gres=gpu:8 \ ray.sub ``` Include the local Bridge checkout in `MOUNTS` and in `PYTHONPATH` inside `COMMAND` when the container does not already see the same path. If using a vendored Bridge under `3rdparty/Megatron-Bridge-workspace/Megatron-Bridge`, sync the exact Bridge changes there instead and report that path. Use a direct `srun` only when `ray.sub` is unavailable, stale for the target cluster, or when debugging the container/Slurm layer itself. Keep paths generic in scripts committed to Megatron-Bridge: ```bash srun \ --container-image="${NEMO_RL_IMAGE}" \ --container-mounts="${BRIDGE_REPO}:/workspace/Megatron-Bridge,${NEMO_RL_REPO}:/workspace/nemo-rl,:" \ --container-workdir=/workspace/nemo-rl \ bash -lc ' export BRIDGE_REPO=/workspace/Megatron-Bridge export NEMO_RL_REPO=/workspace/nemo-rl export PYTHONPATH=$BRIDGE_REPO/src:$BRIDGE_REPO/3rdparty/Megatron-LM:$NEMO_RL_REPO ray stop --force || true uv run bash tests/functional/grpo_megatron.sh ' ``` If an attach helper enters a container that no longer sees the expected checkouts or log directory, treat that helper as stale. Start a fresh `srun` step against the existing allocation with explicit `--container-image`, `--container-mounts`, and `--container-workdir`. Attach helpers that use `--no-container-mount-home` can enter a minimal `/home/$USER` in follow-up steps even when the original run saw the real checkout. Keep metric dumping and assertions in the same container step as the run when possible. If a follow-up step must inspect compute-local artifacts, use paths under the node-local run directory and do not assume `$NEMO_RL_REPO` is visible. For general Slurm debugging and multi-node patterns, read @skills/nemo-mbridge-multi-node-slurm/SKILL.md. ## Pass Criteria A useful pass has all of the following: - Focused Bridge tests pass for provider/config/mapping behavior. - NeMo-RL imports the intended Bridge checkout, verified by `megatron.bridge.__file__`. - The NeMo-RL config has `policy.megatron_cfg.enabled=true` for Megatron policy validation. - The run reaches the requested step count and writes `metrics.json`. - `tests/check_metrics.py` passes when the maintained functional includes metric assertions. - No exception occurs during Bridge provider setup, HF import/export, enabled PEFT/LoRA wrapping, Megatron initialization, optimizer setup, checkpoint manager setup, weight transfer, or the training step. Ray shutdown warnings, Python resource-tracker warnings, or post-completion process-group warnings can be acceptable if the training step completed, metrics were written, and the process exits successfully. Mention them as residual log noise. Do not claim full model e2e if the run used a dummy config, text-only data for a VLM/audio model, trivial expert parallelism for an expert-parallel change, or disabled save/resume for a checkpointing change. Call it the exact level that passed. Do not claim convergence from Level 0. Claim learning signal only from Level 6, and distinguish "learning signal" from benchmark convergence in the report. ## Failure Triage If model construction fails, verify that NeMo-RL is importing the Bridge checkout under test and that `policy.megatron_cfg.converter_type` matches the provider. If the config silently uses dtensor instead of Megatron, set `policy.dtensor_cfg.enabled=false` and `policy.megatron_cfg.enabled=true`, or use `grpo_megatron.sh`. If LoRA fails, check NeMo-RL PEFT config names and Bridge target module names. Reproduce with `grpo_megatron_lora.sh` before adding larger model or parallelism changes. If checkpoint save/load fails, first rerun with `checkpointing.enabled=false` to separate model construction from checkpoint behavior, then use `sft_resume_diamond.sh` for resume parity. If non-colocated refit fails, separate the boundary: - producer export and metadata preparation on the policy worker - payload packing/broadcast - consumer decode and model loading on the generation worker - vLLM-specific weight-loader behavior If NeMo-RL rejects TP >= 4 with the batch-variant accuracy guard, prefer TP 1 or 2 for the smoke, or set `policy.train_micro_batch_size` and `policy.logprob_batch_size` equal. Do not bypass with `NRL_IGNORE_TP_ACCURACY_CHECK=1` for pass/fail evidence unless the user explicitly wants an unsupported diagnostic run. If Megatron generation fails during `cuda graph warmup` with `CUDA error: an illegal memory access was encountered`, rerun the same config with: ```bash policy.generation.mcore_generation_config.num_cuda_graphs=null \ policy.generation.mcore_generation_config.use_cuda_graphs_for_non_decode_steps=false ``` If the no-graph run passes, report the original result as a Megatron generation CUDA-graph failure and the no-graph run as a reduced-optimization pass. Keep both logs. If the run reaches the requested step count but `tests/check_metrics.py` fails on `train/token_mult_prob_error`, treat it as a real metric failure, not a harness failure. NeMo-RL computes this metric from `exp(abs(generation_logprobs - prev_logprobs))`; huge values mean the generation backend logprobs disagree with the policy logprobs recomputed for training. Isolate by retrying with simpler parallelism or kernels such as `policy.megatron_cfg.sequence_parallel=false`, `policy.megatron_cfg.apply_rope_fusion=false`, shorter sequence lengths, or vLLM generation when available. Do not relax the metric threshold or use sequence masking to claim a pass; run Bridge logits/import/export parity to localize whether the mismatch is in Bridge conversion, Megatron generation logprob collection, or NeMo-RL recomputation. If model download fails, move HF caches to a larger path and rerun with explicit cache settings. If Hugging Face returns `429 Too Many Requests` during tokenizer/config setup, first check whether the snapshot already exists under `$HF_HUB_CACHE`. If it does, switch `policy.model_name` and `policy.tokenizer.name` to the local snapshot path and enable offline mode. This is an environment failure unless the local snapshot cannot load with `local_files_only=True`. If `helpers_cpp` fails to link with `No space left on device`, or if logs show `make: python3-config: No such file or directory`, rebuild the helper in a node-local copy of Megatron-LM with `LIBEXT` set from `sysconfig.get_config_var("EXT_SUFFIX")`. Do not patch files under `3rdparty/Megatron-LM/` in the Bridge checkout. If a baseline fails before model build because of data, Ray, vLLM, package setup, or container mismatch, fix the environment first and do not report it as a Bridge provider failure. ## Summary Format End every run with a short user-facing summary that answers "Did the requested deliverables pass?" before adding details. Use `Pass`, `Fail`, `Skipped`, or `Blocked` for each deliverable, and do not report an overall `Pass` unless the pass criteria for the requested coverage level were met. ```text Result: - Requested coverage: Model: Deliverables: - Bridge-side checks: - - Local Bridge import in NeMo-RL: - - NeMo-RL Megatron policy run: - - Requested variants: - - Metrics/log capture: - Evidence: - Bridge repo: plus dirty files - NeMo-RL repo: plus dirty files - Command: - Key lines: Limitations: - Follow-ups: - ``` If the job is blocked before Bridge model/provider construction by data, Ray, vLLM, dependency, disk, container, or cluster setup, mark the overall result as `Blocked`, not `Fail`, and state that it is not evidence against the Bridge provider. If any requested deliverable was not run, mark it `Skipped` or `Not requested` with the reason. Do not leave it implicit in the limitations. --- Source attribution: NVIDIA-NeMo Original project: https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/e301c817218e4e626c8c264aee9935514559a04b/skills/nemo-rl-e2e-testing/SKILL.md License: Apache-2.0 License information: https://www.apache.org/licenses/LICENSE-2.0 Original upstream instructions; downloaded from Mega AI Hub. ## Upstream license notice Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. # Mega AI Hub resource metadata Creator / publisher: NVIDIA-NeMo Original project: https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/e301c817218e4e626c8c264aee9935514559a04b/skills/nemo-rl-e2e-testing/SKILL.md Attribution basis: Source repository record Fields of work: Data & Analytics, Development & Engineering Roles: Backend Developer, Data Scientist, ML / AI Engineer Specialties: API, Backend, Frontend, Git, Playwright, Testing Tools: Megatron, Ne Mo Country applicability: Country not specified Classification basis: Source catalog labels Instruction SHA-256: e198deaf28acf35dcb3c986d2a1f1ca799847030e220f762cb4d1cead6e74769