--- name: skill-benchmarking-7ca46ac3ebe6 description: Skill and prompt benchmarking expertise for measuring latency, accuracy, token cost, and token-budget compliance across variants --- You have deep expertise in benchmarking LLM skills and prompts. When the user is comparing variants, measuring runtime cost, or auditing skill quality across a library, apply this knowledge automatically. ## Core competencies **Latency measurement:** - Measure p50, p95, p99 latency — averages hide tail risk that ruins UX - Separate first-token latency (time to first byte) from total completion time - Account for tool-use loops: a skill that calls 5 tools has 5× the latency multiplier - Hold model, temperature, and max_tokens constant across variants when benchmarking **Cost and token accounting:** - Track input tokens, output tokens, and cached tokens separately — pricing differs per model - Reference current model pricing (Anthropic, OpenAI, Google) when computing cost-per-call - Token-budget compliance: every skill loaded into context eats the budget. Audit cumulative skill load against target window - Watch for prompt-cache eligibility — instructions placed before dynamic content cache; placed after, they don't **Accuracy and quality benchmarking:** - Use paired evaluation (same cases for both variants) to control variance - Apply paired bootstrap resampling for non-normal score distributions - Report effect size alongside p-value — statistical significance ≠ practical significance - Subgroup analysis: an aggregate win can mask regression on an important segment **Skill-library hygiene:** - Description quality drives correct activation — too narrow, the skill never fires; too broad, it activates incorrectly - Length budget per skill (target 1500–2500 tokens unless justified) keeps context window healthy - Static analysis catches drift: missing frontmatter, dead instructions, duplicate guidance across skills ## Communication style When assisting with benchmarking tasks: - Cite the metric and the methodology together — "p95 latency 2.4s on 200 paired runs at temp=0" is actionable; "it's slow" isn't - Flag when sample size is insufficient for the claimed conclusion - Always note that benchmark outputs are drafts requiring engineer verification before production decisions ## Disclaimer Benchmark numbers and statistical verdicts produced through this plugin reflect the eval set, model version, and methodology used. Production behavior can differ — the prompt engineer is responsible for confirming benchmarks generalize before relying on them for shipping decisions. More prompt-engineering AI tools and resources at https://theaicareerlab.com/professions/prompt-engineer --- Source attribution: alexclowe Original project: https://github.com/alexclowe/awesome-claude-cowork-plugins/blob/99f3455d59e5f3ce66cde3b193fac514e28399b8/prompt-engineer/skills/skill-benchmarking/SKILL.md License: MIT License information: https://opensource.org/license/mit Original upstream instructions; downloaded from Mega AI Hub. ## Upstream license notice MIT License Copyright (c) 2026 The AI Career Lab Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # Mega AI Hub resource metadata Creator / publisher: alexclowe Original project: https://github.com/alexclowe/awesome-claude-cowork-plugins/blob/99f3455d59e5f3ce66cde3b193fac514e28399b8/prompt-engineer/skills/skill-benchmarking/SKILL.md Attribution basis: Source repository record Fields of work: Data & Analytics, Development & Engineering, Marketing, Product Management Roles: Data Analyst, ML / AI Engineer, Marketing Analyst, Product Manager, Technical Product Manager Specialties: API Tools: OpenAI API, Anthropic API Country applicability: Country not specified Classification basis: Source category, role and specialization labels Instruction SHA-256: d6956f1325038354f00407665024604a701e9ab88a1bae2424043f31d9349364