--- name: lighteval-b9e04f195e69 description: "All-in-one LLM evaluation toolkit (HuggingFace LightEval). 1000+ tasks with multi-backend support: Accelerate, vLLM, SGLang, Nanotron, TGI, LiteLLM, inference providers, and custom models. Sample-by-sample result exploration, custom task/metric creation. Used by HuggingFace's Leaderboard and Evals team. For pure GPT-style model eval, also consider lm-evaluation-harness." license: MIT license tags: [multilingual-benchmarks, backend-flexible-eval, sample-level-analysis, custom-metrics, lighteval] metadata: skill-author: K-Dense Inc. ------|---------|----------| | `inspect-ai` | `lighteval eval` | Preferred, modern backend | | Accelerate | `lighteval accelerate` | Single/multi-GPU local models | | vLLM | `lighteval vllm` | Fast batched inference | | SGLang | `lighteval sglang` | Structured generation evals | | Nanotron | `lighteval nanotron` | Distributed training evaluation | | TGI | `lighteval endpoint tgi` | Locally served models | | LiteLLM | `lighteval endpoint litellm` | Any API (OpenAI, Anthropic, etc.) | | HF Providers | `lighteval endpoint inference-providers` | HuggingFace's hosted API | | Inference Endpoints | `lighteval endpoint inference-endpoint` | HF Dedicated Endpoints | | Custom | `lighteval custom` | Anything with a Python API | ### 4. Task Domains **Knowledge Tasks:** ```bash lighteval accelerate "model-name" mmlu # 57-subject knowledge lighteval accelerate "model-name" mmlu-pro # Harder MMLU lighteval accelerate "model-name" gpqa # Graduate-level science lighteval accelerate "model-name" triviaqa # Trivia QA lighteval accelerate "model-name" humanitys_last_exam # Very hard questions ``` **Math and Code:** ```bash lighteval accelerate "model-name" gsm8k # Grade school math lighteval accelerate "model-name" math # Competition math lighteval accelerate "model-name" aime24 # AIME 2024 lighteval accelerate "model-name" lcb # LiveCodeBench ``` **Chat Model Evaluation:** ```bash lighteval accelerate "model-name" ifeval # Instruction following lighteval accelerate "model-name" mt_bench # Multi-turn dialogue lighteval accelerate "model-name" musr # Multi-step reasoning lighteval accelerate "model-name" ruler # Long context ``` **Multilingual:** ```bash lighteval accelerate "model-name" mgsm # Math in 10+ languages lighteval accelerate "model-name" flores200 # 200-language translation lighteval accelerate "model-name" mmlu_arabic # Arabic MMLU lighteval accelerate "model-name" cmmlu # Chinese MMLU lighteval accelerate "model-name" russian_squad # Russian QA ``` ### 5. Custom Tasks ```python from lighteval.tasks.lighteval_task import LightevalTask from lighteval.metrics.metrics import SampleLevelMetric class MyCustomTask(LightevalTask): def __init__(self, *args, **kwargs): super().__init__( name="my_custom_task", version=0, metrics=["my_metric"], *args, **kwargs ) def get_prompt(self, sample): return f"Question: {sample['question']}\nAnswer:" def process_output(self, output, sample): # Extract answer from model output return output.strip() def get_gold(self, sample): return sample["answer"] ``` ### 6. Custom Metrics ```python from lighteval.metrics.metrics import SampleLevelMetric import numpy as np class F1Metric(SampleLevelMetric): def __init__(self, *args, **kwargs): super().__init__(metric_name="f1", *args, **kwargs) def compute(self, golds, predictions, **kwargs): # golds and predictions are lists scores = [] for gold, pred in zip(golds, predictions): # Compute per-sample F1 gold_tokens = set(gold.lower().split()) pred_tokens = set(pred.lower().split()) tp = len(gold_tokens & pred_tokens) fp = len(pred_tokens - gold_tokens) fn = len(gold_tokens - pred_tokens) precision = tp / (tp + fp + 1e-10) recall = tp / (tp + fn + 1e-10) f1 = 2 * precision * recall / (precision + recall + 1e-10) scores.append(f1) return np.mean(scores) ``` ### 7. Multi-Backend Configuration ```python # Evaluate same tasks across backends backends = { "vllm": "lighteval vllm", "sglang": "lighteval sglang", "accelerate": "lighteval accelerate", } for backend, cmd in backends.items(): print(f"Running {backend}...") subprocess.run(f"{cmd} meta-llama/Meta-Llama-3-8B-Instruct mmlu gsm8k", shell=True) ``` ### 8. Pushing Results to HuggingFace Hub ```bash lighteval accelerate "model-name" mmlu \ --push-to-hub \ --push-results-dir my-org/eval-results \ --results-org my-org # Results appear at: https://huggingface.co/my-org/eval-results ``` ### 9. Task Discovery ```bash # List all tasks lighteval list-tasks # Filter by domain lighteval list-tasks --domain math lighteval list-tasks --domain multilingual # Search lighteval list-tasks --query mmlu ``` **Open Benchmark Index (web UI):** - Browse: https://huggingface.co/spaces/OpenEvals/open_benchmark_index - Find tasks by domain, language, difficulty ### 10. Detailed Result Analysis ```python from lighteval.logging.evaluation_tracker import EvaluationTracker tracker = EvaluationTracker(output_dir="./results") # After evaluation: for task_name, task_results in tracker.results.items(): print(f"\n=== {task_name} ===") print(f" Score: {task_results['score']:.3f}") print(f" Samples: {len(task_results['samples'])}") # Inspect failures failures = [s for s in task_results['samples'] if not s['correct']] for f in failures[:5]: print(f" Q: {f['input']}") print(f" Predicted: {f['prediction']}") print(f" Expected: {f['gold']}\n") ``` ## Key Patterns 1. **Use `lighteval eval` as preferred entrypoint** — inspect-ai backend is most modern 2. **vLLM for speed**, Accelerate for simplicity, LiteLLM for API access 3. **Push to Hub** for sharing results and comparing models 4. **Sample-level analysis** for debugging eval failures 5. **Custom metrics** are first-class — no need to fork the library 6. **Open Benchmark Index** for discovering available tasks ## References - [LightEval Documentation](https://huggingface.co/docs/lighteval/) - [Open Benchmark Index](https://huggingface.co/spaces/OpenEvals/open_benchmark_index) - [Adding Custom Tasks Guide](https://huggingface.co/docs/lighteval/adding-a-custom-task) - [Adding Custom Metrics Guide](https://huggingface.co/docs/lighteval/adding-a-new-metric) - [Backend Configuration](https://huggingface.co/docs/lighteval/installation) --- Source attribution: mkurman Original project: https://github.com/mkurman/zorai/blob/d06ee7597049567dab92db22d41b2764cf89094a/skills/scientific-skills/lighteval/SKILL.md License: MIT License information: https://opensource.org/license/mit Original upstream instructions; downloaded from Mega AI Hub. ## Upstream license notice MIT License Copyright (c) 2026 zorai contributors Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # Mega AI Hub resource metadata Creator / publisher: mkurman Original project: https://github.com/mkurman/zorai/blob/d06ee7597049567dab92db22d41b2764cf89094a/skills/scientific-skills/lighteval/SKILL.md Attribution basis: Source repository record Fields of work: Data & Analytics, Development & Engineering Roles: Backend Developer, Data Scientist, ML / AI Engineer Specialties: API, Backend Tools: Huggingface Country applicability: Country not specified Classification basis: Source catalog labels Instruction SHA-256: 787b2f5d0fcd394f20e4105ebeaaa8063fd3a9ed5a3c448f6c0c9f9d9a0a9cbe