--- name: evaluation-e621a18e2244 description: Evaluates accuracy of quantized or unquantized LLMs using NeMo Evaluator Launcher (NEL). Triggers on "evaluate model", "benchmark accuracy", "run MMLU", "evaluate quantized model", "accuracy drop", "run nel". Handles deployment, config generation, and evaluation execution. Not for quantizing models (use ptq) or deploying/serving models (use deployment). license: Apache-2.0 # Based on nel-assistant skill from NeMo Evaluator Launcher (commit f1fa073) # https://github.com/NVIDIA-NeMo/Evaluator/tree/f1fa073/packages/nemo-evaluator-launcher/.claude/skills/nel-assistant # Modifications: renamed to evaluation, added workspace management (Step 0), # auto-detect ModelOpt quantization format, quantization-aware benchmark defaults. --- ## NeMo Evaluator Launcher Assistant You're an expert in NeMo Evaluator Launcher! Guide the user through creating production-ready YAML configurations, running evaluations, and monitoring progress via an interactive workflow specified below. ### Workspace and Pipeline Integration If `MODELOPT_WORKSPACE_ROOT` is set, read `skills/common/workspace-management.md`. Check for existing workspaces — especially if evaluating a model from a prior PTQ or deployment step. Reuse the existing workspace so you have access to the quantized checkpoint and any code modifications. This skill is often the final stage of the PTQ → Deploy → Eval pipeline. If the model required runtime patches during deployment (transformers upgrade, framework source fixes), carry those patches into the NEL config via `deployment.command`. ### Workflow ```text Config Generation Progress: - [ ] Step 0: Check workspace (if MODELOPT_WORKSPACE_ROOT is set) - [ ] Step 1: Check if nel is installed and if user has existing config - [ ] Step 2: Build the base config file - [ ] Step 3: Configure model path and parameters - [ ] Step 4: Fill in remaining missing values - [ ] Step 5: Confirm tasks (iterative) - [ ] Step 6: Advanced - Multi-node (Data Parallel) - [ ] Step 7: Advanced - Interceptors - [ ] Step 7.5: Check container registry auth (SLURM only) - [ ] Step 8: Run the evaluation ``` **Step 1: Check prerequisites** Test that `nel` is installed with `nel --version`. If not, instruct the user to `pip install nemo-evaluator-launcher`. If the user already has a config file (e.g., "run this config", "evaluate with my-config.yaml"), skip to Step 8. Optionally review it for common issues (missing `???` values, quantization flags) before running. **Shortcut: use pre-built task snippets.** If the user asks for a specific benchmark (e.g., "run MMLU-Pro", "evaluate with AIME"), check `recipes/tasks/` (relative to this skill's directory) for a matching task snippet. Available: mmlu_pro, gpqa, aime2025, livecodebench, ifbench, scicode. Task snippets contain only the task-specific config (name, params, repeats) — not the full NEL config. To use them: 1. Read the task snippet(s) the user wants 2. Use `recipes/examples/example_eval.yaml` as the base config template 3. Replace the `tasks:` section with the selected snippet(s) 4. Do Step 3 (auto-detect model settings from checkpoint) and Step 4 (fill in `???` values) 5. Proceed to Step 7.5/8 **Step 2: Build the base config file** Prompt the user with "I'll ask you 5 questions to build the base config we'll adjust in the next steps". Guide the user through the 5 questions using AskUserQuestion: 1. Execution: - Local - SLURM 2. Deployment: - None (External) - vLLM - SGLang - NIM - TRT-LLM 3. Auto-export: - None (auto-export disabled) - MLflow - wandb 4. Model type - Base - Chat - Reasoning 5. Benchmarks: Allow for multiple choices in this question. 1. Standard LLM Benchmarks (like MMLU, IFEval, GSM8K, ...) 2. Code Evaluation (like HumanEval, MBPP, and LiveCodeBench) 3. Math & Reasoning (like AIME, GPQA, MATH-500, ...) 4. Safety & Security (like Garak and Safety Harness) 5. Multilingual (like MMATH, Global MMLU, MMLU-Prox) Only accept options from the categories listed above (Execution, Deployment, Auto-export, Model type, Benchmarks). YOU HAVE TO GATHER THE ANSWERS for the 5 questions before you can build the base config. > **Note:** These categories come from NEL's `build-config` CLI. **Always run `nel skills build-config --help` first** to get the current options — they may differ from this list (e.g., `chat_reasoning` instead of separate `chat`/`reasoning`, `general_knowledge` instead of `standard`). When the CLI's current options differ from this list, prefer the CLI's options. When you have all the answers, run the script to build the base config: ```bash nel skills build-config --execution --deployment --model_type --benchmarks [--export ] [--output ] ``` Where `--output` depends on what the user provides: - Omit: Uses current directory with auto-generated filename - Directory: Writes to that directory with auto-generated filename - File path (*.yaml): Writes to that specific file It never overwrites existing files. **Step 3: Configure model path and parameters** Ask for model path. Determine type: - Checkpoint path (local directory — starts with `/`, `./`, `../`, `~`, or contains no `/` but exists on disk) → set `deployment.checkpoint_path: ` and `deployment.hf_model_handle: null` - HF handle (e.g., `org/model-name` — contains exactly one `/` and does not exist locally) → set `deployment.hf_model_handle: ` and `deployment.checkpoint_path: null` **Auto-detect ModelOpt quantization format** (checkpoint paths only): Check for `hf_quant_config.json` in the checkpoint directory: ```bash cat /hf_quant_config.json 2>/dev/null ``` If found, read `quantization.quant_algo` and set the correct vLLM/SGLang quantization flag in `deployment.extra_args`: | `quant_algo` | Flag to add | |-------------|-------------| | `FP8` | `--quantization modelopt` | | `W4A8_AWQ` | `--quantization modelopt` | | `NVFP4`, `NVFP4_AWQ` | `--quantization modelopt_fp4` | | Other values | Try `--quantization modelopt`; consult vLLM/SGLang docs if unsure | If no `hf_quant_config.json`, also check `config.json` for a `quantization_config` section with `quant_method: "modelopt"`. If neither is found, the checkpoint is unquantized — no flag needed. > **Note:** Some models require additional env vars for deployment (e.g., `VLLM_NVFP4_GEMM_BACKEND=marlin` for Nemotron Super). These are not in `hf_quant_config.json` — they are discovered during model card research below. **Auto-detect deployment settings from checkpoint:** Read `config.json` from the checkpoint (or HF model card) and build `deployment.extra_args` dynamically: ```bash cat /config.json 2>/dev/null ``` | Field in `config.json` | What to set | Example | | --- | --- | --- | | `max_position_embeddings` | `--max-model-len ` | `131072` → `--max-model-len 131072` | | `auto_map` exists | `--trust-remote-code` | Only add if model has custom code | Then use WebSearch to check the model card (HuggingFace page) for deployment-specific settings: | Model card signal | What to set | | --- | --- | | Reasoning model (thinking/CoT) | `--reasoning-parser` and `--reasoning-parser-plugin` if a custom parser is provided | | Tool-calling support | `--enable-auto-tool-choice --tool-call-parser ` | | Custom vLLM flags documented | Add as specified (e.g., `--mamba_ssm_cache_dtype float32`) | Combine all detected flags into a single `deployment.extra_args` override. The recipe's default `--max-model-len 32768` is a fallback — always prefer the value from `config.json`. **Quantization-aware benchmark defaults:** When a quantized checkpoint is detected, read `references/quantization-benchmarks.md` for benchmark sensitivity rankings and recommended sets. Present recommendations to the user and ask which to include. Read `references/model-card-research.md` for the full extraction checklist (sampling params, reasoning config, ARM64 compatibility, pre_cmd, etc.). Use WebSearch to research the model card, present findings, and ask the user to confirm. **Step 4: Fill in remaining missing values** - Find all remaining `???` missing values in the config. - Ask the user only for values that couldn't be auto-discovered from the model card (e.g., SLURM hostname, account, output directory, MLflow/wandb tracking URI). Don't propose any defaults here. Let the user give you the values in plain text. - Ask the user if they want to change any other defaults e.g. execution partition or walltime (if running on SLURM) or add MLflow/wandb tags (if auto-export enabled). **Step 5: Confirm tasks (iterative)** Show tasks in the current config. Loop until the user confirms the task list is final: 1. Tell the user: "Run `nel ls tasks` to see all available tasks". 2. Ask if they want to add/remove tasks or add/remove/modify task-specific parameter overrides. To add per-task `nemo_evaluator_config` as specified by the user, e.g.: ```yaml tasks: - name: nemo_evaluator_config: config: params: temperature: max_new_tokens: ... ``` 3. Apply changes. 4. Show updated list and ask: "Is the task list final, or do you want to make more changes?" **Known Issues** - NeMo-Skills workaround (self-deployment only): If using `nemo_skills.*` tasks with self-deployment (vLLM/SGLang/NIM), add at top level: ```yaml target: api_endpoint: api_key_name: DUMMY_API_KEY ``` For the None (External) deployment the `api_key_name` should be already defined. The `DUMMY_API_KEY` export is handled in Step 8. **Step 6: Advanced - Multi-node** If the user needs multi-node evaluation (model >120B, or more throughput), read `references/multi-node.md` for the configuration patterns (HAProxy multi-instance, Ray TP/PP, or combined). **Step 7: Advanced - Interceptors** - Tell the user they should see: . - DON'T provide any general information about what interceptors typically do in API frameworks without reading the docs. If the user asks about interceptors, only then read the webpage to provide precise information. - If the user asks you to configure some interceptor, then read the webpage of this interceptor and configure it according to the `--overrides` syntax but put the values in the YAML config under `evaluation.nemo_evaluator_config.config.target.api_endpoint.adapter_config` (NOT under `target.api_endpoint.adapter_config`) instead of using CLI overrides. By defining `interceptors` list you'd override the full chain of interceptors which can have unintended consequences like disabling default interceptors. That's why use the fields specified in the `CLI Configuration` section after the `--overrides` keyword to configure interceptors in the YAML config. **Documentation Errata** - The docs may show incorrect parameter names for logging. Use `max_logged_requests` and `max_logged_responses` (NOT `max_saved_*` or `max_*`). **Step 7.5: Check container registry authentication (SLURM only)** NEL's default deployment images by framework: | Framework | Default image | Registry | | --- | --- | --- | | vLLM | `vllm/vllm-openai:latest` | DockerHub | | SGLang | `lmsysorg/sglang:latest` | DockerHub | | TRT-LLM | `nvcr.io/nvidia/tensorrt-llm/release:...` | NGC | | Evaluation tasks | `nvcr.io/nvidia/eval-factory/*:26.03` | NGC | Before submitting, verify the cluster has credentials for the deployment image. See `skills/common/slurm-setup.md` section 6 for the full procedure. ```bash ssh "grep -E '^\s*machine\s+' ~/.config/enroot/.credentials 2>/dev/null" ``` **Decision flow (check before submitting):** 1. Check if the cluster has credentials for the default DockerHub image (see command above) 2. If DockerHub credentials exist → use the default image and submit 3. If DockerHub credentials are missing but can be added → add them (see `slurm-setup.md` section 6), then submit 4. If DockerHub credentials cannot be added → override `deployment.image` to the NGC alternative and submit: ```yaml deployment: image: nvcr.io/nvidia/vllm:-py3 # check https://catalog.ngc.nvidia.com/orgs/nvidia/containers/vllm for latest tag ``` 5. **Do not retry more than once** without fixing the auth issue **Step 8: Run the evaluation** Print the following commands to the user. Propose to execute them in order to confirm the config works as expected before the full run. **Important**: Export required environment variables based on your config. If any tokens or keys are missing, point the user to `recipes/env.example` — it lists all possible keys with notes on which tasks need them. Ask the user to copy it, fill in their keys, and source it: ```bash cp recipes/env.example .env # Edit .env with your keys set -a && source .env && set +a ``` ```bash # If using pre_cmd or post_cmd (review pre_cmd content before enabling — it runs arbitrary commands): export NEMO_EVALUATOR_TRUST_PRE_CMD=1 # If using nemo_skills.* tasks with self-deployment: export DUMMY_API_KEY=dummy ``` 1. **Dry-run** (validates config without running): ```bash nel run --config --dry-run ``` 2. **Test with limited samples** (quick validation run): ```bash nel run --config -o ++evaluation.nemo_evaluator_config.config.params.limit_samples=10 ``` 3. **Re-run a single task** (useful for debugging or re-testing after config changes): ```bash nel run --config -t ``` Combine with `-o` for limited samples: `nel run --config -t -o ++evaluation.nemo_evaluator_config.config.params.limit_samples=10` 4. **Full evaluation** (production run): ```bash nel run --config ``` After the dry-run, check the output from `nel` for any problems with the config. If there are no problems, propose to first execute the test run with limited samples and then execute the full evaluation. If there are problems, resolve them before executing the full evaluation. **Monitoring Progress** After job submission, register the job per the **monitor skill** for durable cross-session tracking. For one-off queries (live status, debugging a failed run, analyzing results) use the **launching-evals skill**; for querying past runs in MLflow use **accessing-mlflow**. **NEL-specific diagnostics** (for debugging failures): ```bash # Quick status check nel status nel info # Get log paths nel info --logs # Inspect logs via SSH ssh @ "tail -100 /server--*.log" # deployment errors ssh @ "tail -100 /client-.log" # evaluation errors ssh @ "tail -100 /slurm-.log" # scheduling/walltime ssh @ "grep -i 'error\|failed' /*.log" # search all logs ``` --- Direct users with issues to: - **GitHub Issues:** - **GitHub Discussions:** Now, copy this checklist and track your progress: ```text Config Generation Progress: - [ ] Step 0: Check workspace (if MODELOPT_WORKSPACE_ROOT is set) - [ ] Step 1: Check if nel is installed and if user has existing config - [ ] Step 2: Build the base config file - [ ] Step 3: Configure model path and parameters - [ ] Step 4: Fill in remaining missing values - [ ] Step 5: Confirm tasks (iterative) - [ ] Step 6: Advanced - Multi-node (Data Parallel) - [ ] Step 7: Advanced - Interceptors - [ ] Step 7.5: Check container registry auth (SLURM only) - [ ] Step 8: Run the evaluation ``` --- Source attribution: yo-steven Original project: https://github.com/yo-steven/skills-exploration-20260522/blob/6e1127ffb7690aa8038b172d0ae7cc720754870d/skills/Model-Optimizer/evaluation/SKILL.md License: Apache-2.0 License information: https://www.apache.org/licenses/LICENSE-2.0 Original upstream instructions; downloaded from Mega AI Hub. ## Upstream license notice SPDX-License-Identifier: Apache-2.0 AND CC-BY-4.0 Copyright (c) 2026 NVIDIA Corporation This project is licensed under a dual license: - Apache License 2.0 (for code and configuration) - Creative Commons Attribution 4.0 International (CC BY 4.0) (for documentation) You may use the contents of this repository under the terms of either license, or both, as applicable to the type of content. ================================================================================ Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to the Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by the Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding any notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS ================================================================================ Creative Commons Attribution 4.0 International License (CC BY 4.0) https://creativecommons.org/licenses/by/4.0/ You are free to: Share — copy and redistribute the material in any medium or format for any purpose, even commercially. Adapt — remix, transform, and build upon the material for any purpose, even commercially. The licensor cannot revoke these freedoms as long as you follow the license terms. Under the following terms: Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits. Notices: You do not have to comply with the license for elements of the material in the public domain or where your use is permitted by an applicable exception or limitation. No warranties are given. The license may not give you all of the permissions necessary for your intended use. For example, other rights such as publicity, privacy, or moral rights may limit how you use the material. For the full legal text, see: https://creativecommons.org/licenses/by/4.0/legalcode # Mega AI Hub resource metadata Creator / publisher: yo-steven Original project: https://github.com/yo-steven/skills-exploration-20260522/blob/6e1127ffb7690aa8038b172d0ae7cc720754870d/skills/Model-Optimizer/evaluation/SKILL.md Attribution basis: Source repository record Fields of work: Data & Analytics, Development & Engineering Roles: Data Analyst, Data Scientist, ML / AI Engineer Specialties: API Tools: Nemo Evaluator Launcher Country applicability: Country not specified Classification basis: Source catalog labels Instruction SHA-256: 938be8476053573c3d9152a0d0aeec69145bb969b08f18292af08ff79617b3d0