---
name: validate-38fe49737378
description: "Statically validate a `claude plugin eval` suite (`prompt.md`, `case.yaml`, `graders/*.md`) before any run spends money: FAIL for what the binary rejects at load, WARN for documented authoring mistakes, and an offline check of each deterministic grader against the case's sample answers. Use when: 'validate my eval cases', 'check my eval suite', 'will this suite load', 'lint case.yaml', 'check my graders', 'why did my case fail to load', or before paying for a run. Not for the skill-creator JSON eval format."
argument-hint: "[eval-dir]"
user-invocable: true
disable-model-invocation: false
metadata:
  workflow-stage: test
  summary: Static FAIL/WARN check of an eval suite's cases and graders, with no model call
---

# Validate eval cases before a run spends money

A case that fails to load costs the same as one that works. `scripts/validate-cases.py` reads the
suite off disk and reports what would break, with no model call and no network access.

## Run it

```bash
python3 "${CLAUDE_PLUGIN_ROOT}/skills/validate/scripts/validate-cases.py" <eval-dir>
```

`<eval-dir>` is the directory holding the cases, `evals/` under the plugin root by default. Add
`--json` for the findings as an array of `{level, case, file, message}` objects; `--help` prints
the full contract from the script's header.

| Exit | Meaning |
|---|---|
| 0 | no FAIL finding. WARN findings are printed and do not gate |
| 1 | at least one FAIL finding |
| 2 | usage error, or the eval dir is missing or unreadable |

Findings print one per line as `<FAIL|WARN> <case>/<file>: <message>`, sorted by case then file; a
finding about the suite as a whole carries the eval dir in place of `<case>/<file>`.

## What the two tiers mean

**FAIL is what the binary itself rejects**: an unknown `prompt.md` frontmatter key, a grader with no
usable `type`, an unknown option for the declared grader type, a case with no grader, two graders
sharing a name, a non-positive `weight`, `runs` / `max_turns` / `timeout_seconds` out of bounds, an
`env` key that does not match `EVAL_[A-Z0-9_]*`, a case that has a `case.yaml` but no
`schema_version` or `name` in it or in `prompt.md` frontmatter (without a `case.yaml` the binary
supplies both), a case with no prompt (`prompt.md` body or `execution.prompt`), a grader missing an
option its type requires, and a `schema_version` that is not a quoted version string or whose
major is newer than the binary supports. The complete sets the script checks, so a question about
a key or a type is answered here without opening the script:

- `prompt.md` frontmatter keys: `schema_version`, `name`, `description`, `tags`, `plugins`,
  `runs`, `expected_outcome`, `model`, `max_turns`, `timeout_seconds`, `allowed_tools`,
  `append_system_prompt`, `env`. A time limit is `timeout_seconds`; `timeout` is an unknown key, so
  the case fails to load.
- Grader `type`: `regex`, `tool_order`, `tool_used`, `file_exists`, `llm`, `baseline`. Any other
  value, such as `contains` or `string_match`, is no usable type, so the case fails to load. A
  substring or pattern check is `type: regex` with the text in `pattern`.
- Options per type, beside the `type`, `weight`, `arm` and `name` every grader takes: `regex`
  `pattern`, `flags`, `match`, `target`; `tool_used` `tool`, `input_match`, `min`, `max`;
  `tool_order` `before`, `after`; `file_exists` `path`, `exists`; `llm` `criteria`, `focus`;
  `baseline` `baseline_file`, `criteria`.
- Required options per type: `regex` `pattern`; `tool_used` `tool`; `tool_order` `before`,
  `after`; `file_exists` `path`; `llm` `criteria`; `baseline` `baseline_file`, `criteria`. A
  `graders/*.md` body stands in for `pattern` (regex) or `criteria` (llm, baseline).
- Bounds: `runs` 1 to 50, `max_turns` 1 to 200, `timeout_seconds` 1 to 3600.

The script's constants are the one place to correct when the schema moves, under the drift record
its header carries; this list follows them.

**WARN is an authoring mistake the suite loads with** and then scores badly on: `target: files`
(which reads the list of paths created, not their contents), an inline `(?i)` the grader's regex
engine does not honor, a case whose graders are all judges (the two paid types, with no
deterministic grader beside them), a tool in `allowed_tools` that the operator must grant with
`--allow-tools`, a case naming a skill (leading `/<plugin>:<skill>` or a tag) whose SKILL.md has a
pre-computed context line while `allowed_tools` lacks Bash, `file_exists` in a case that requests no write tool (so nothing it could match is
ever created), and a `prompt.md` key the binary loads but the reference page does not list, which
can change without notice (the script's `UNDOCUMENTED_PROMPT_KEYS`, under its own drift record).
The judge-only rule is this skill's own pairing heuristic, not a rejection the binary makes; the
sample-answer findings below are likewise this skill's checks.

A WARN never sets exit 1, so a suite can ship with warnings on purpose. Read them once and decide.

## Sample answers

A grader that rejects a correct paraphrase scores the plugin down for nothing, and no run says the
grader was at fault. So each case can carry known answers, and the script runs the free graders
over them before any money is spent.

Put them in `samples/<grader-name>.json` inside the case directory, beside `graders/` and never in
it:

```json
{
  "pass": [{"answer": "...", "why": "the ranking rule with 'and' for the comma"}],
  "fail": [{"answer": ""}, {"answer": "...", "why": "near miss: base-model phrasing"}]
}
```

`answer` is what the grader reads: text for a `regex` (the final reply on the default target), a
list of `{"tool", "input"}` calls for `tool_used` and `tool_order`, and a list of created paths for
`file_exists`. `why` is optional and is echoed in any finding. Give each grader several paraphrases
it must pass, near misses it must reject, an empty answer, and an answer to a different question.

| Finding | Level |
|---|---|
| A must-pass answer the grader rejects, or a must-fail answer it accepts | FAIL |
| A sample file that is not JSON in this shape, or an answer of the wrong kind for its grader | FAIL |
| A `regex`, `tool_used`, `tool_order` or `file_exists` grader with no sample file | WARN |
| A sample file with no must-pass or no must-fail answers, or named after no grader | WARN |
| A setting the script cannot reproduce offline, such as the `y` or `v` regex flag | WARN |
| A pattern with Python-only syntax: `\A`, `\Z`, `\z`, a possessive quantifier, an atomic group, a `(?P<` group or `(?P=` reference, a `(?#` comment, a `(?(` conditional. The finding names the grader and the token | FAIL |
| With `node` on PATH: a regex that does not compile under JavaScript, or a sample JavaScript's `RegExp` sorts the other way | FAIL |
| An `llm` or `baseline` grader with samples: they need a paid judge calibration run | WARN |

The script FAILs the Python-only tokens listed above; that check needs nothing installed. When
`node` is on PATH it also grades each regex grader and its samples through `RegExp`, and without
`node` it skips that part silently.

The script never calls a judge. Samples on an `llm` grader are the labeled answers a calibration
run feeds the judge, and that run is the operator's to start.

- **Pointer**: the samples location rests on the case layout and run isolation in the
  [eval suite reference](https://code.claude.com/docs/en/plugin-evals#eval-suite-reference) and on
  the case loader in Claude Code 2.1.287, which reads only `prompt.md`, `case.yaml` and `graders/`.
  The sample check grades as the binary does, per its grader code read against the
  [grader types](https://code.claude.com/docs/en/plugin-evals#grader-types) table; Python's `re`
  stands in for the JavaScript regex engine, and the script's header lists where the two differ.
- **As of**: 2026-10-01
- **Recheck trigger**: the next Claude Code release, a release that adds a file the loader reads
  from a case directory, or a grader whose sample verdict disagrees with its verdict in a real run.

## When the parser stops

The script reads a bounded YAML subset: scalars, quoted scalars, flow lists and mappings, block
lists including lists of mappings, and block mappings three levels deep. An anchor, an alias, a tag,
a block scalar, a merge key, tab indentation, a nested sequence in either spelling, a backslash
escape outside `\\`, `\"`, `\/`, `\n`, `\t`, `\r`, or nesting past three levels is reported as
`frontmatter not parsed (<construct>)` at FAIL, never waved through at exit 0, because a validator
that green-lights input it could not read is worse than no validator. Simplify the file, or run the
CLI, which carries the full parser.

## Mirroring record

The load-time FAIL findings match what the binary rejects when it loads a case, so a suite at exit
0 loads.

- **Pointer**: the case schema recovered from the shipped Claude Code binary, read against the
  eval-suite reference at <https://code.claude.com/docs/en/plugin-evals#eval-suite-reference>; the
  required fields were re-derived from Claude Code 2.1.296 and each rejection reproduced with
  `claude plugin eval --case`.
- **As of**: 2026-10-09
- **Recheck trigger**: the next Claude Code release, which can add a frontmatter key, add a grader
  type, or move a bound. On a firing, re-derive both lists from the schema rather than patching
  one value.

This skill asserts nothing about whether `claude plugin eval` is available in your install; it
reads files and never invokes anything. `/evals:plugin-eval` preflight owns that question.

## Boundary

This validator reads only the `prompt.md` / `case.yaml` / `graders/*.md` layout, the format
`/evals:design` writes for a skill. Pointed at a skill-creator JSON eval file, it reports only that
the directory holds no cases. Convert that file's cases to this layout (`/evals:design skill <name>`
writes them), then validate the case directories.

## Next

`/evals:plugin-eval <target>`. A suite at exit 0 is one worth paying to run.

## Gotchas

- Exit 0 means the suite **loads**, not that it measures anything. A case whose graders pass in both
  arms proves the plugin contributed nothing; the delta is read after a run, not here.
- Samples that all sort correctly prove the grader handles those answers, not the ones the model
  will write. When a real run fails a correct answer, add that answer's phrasing as a sample.
- A case is a directory holding `prompt.md` or `case.yaml`. A directory holding neither is skipped
  silently, so a misnamed prompt file reads as absent rather than as an error. An eval dir with no
  case at all is a FAIL.
- `results/` and `mocks/` are never treated as cases, whatever they contain.
- When both files exist, `prompt.md` frontmatter overrides the matching `case.yaml` field, and the
  bounds check reports the file the effective value came from, which may not be the file you edited.
- Unknown top-level keys in `case.yaml` are not reported. The unknown-key rejection is recorded for
  `prompt.md`, and this validator invents no rejection the binary does not make.
- The script is Python 3.8+, standard library only, so it runs from a consumer checkout where no
  package is provisioned. It never writes to the suite it reads.


---
Source attribution: melodic-software
Original project: https://github.com/melodic-software/claude-code-plugins/blob/79b70b6cbd0d9a457ced85a0c723d39e08250a1d/plugins/evals/skills/validate/SKILL.md
License: MIT
License information: https://opensource.org/license/mit
Original upstream instructions; downloaded from Mega AI Hub.

## Upstream license notice

MIT License

Copyright (c) 2026 Melodic Software

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.



# Mega AI Hub resource metadata

Creator / publisher: melodic-software
Original project: https://github.com/melodic-software/claude-code-plugins/blob/79b70b6cbd0d9a457ced85a0c723d39e08250a1d/plugins/evals/skills/validate/SKILL.md
Attribution basis: Source repository record

Fields of work: Product Management, Operations, Development & Engineering
Roles: Product Manager, Business Process Analyst, QA / Test Engineer
Specialties: API
Tools: Spec Reviewer
Country applicability: Country not specified
Classification basis: Source category, role and specialization labels
Instruction SHA-256: 53565230df50317c237fcad08a0fa8f0554259033d76b6d0d6e3fb966bea7d7f