AI Evaluation Tools

17 resources

• lm-eval

  Run reproducible language model benchmarks against standard or custom tasks with a common scoring interface.

  eleutherai · Local Python runner. The chosen model backend determines model access.

  Compatibility

  Python, Hugging Face Transformers, vLLM

  Evaluation areas

  Model benchmarks, Custom tasks, Batch evaluation

  Setup notes

  Choose the model backend and task, then fix the prompt format, shot count, and decoding settings before running a comparison. Keep the exact task and model revisions with the result.

  Install the extra for your intended backend, such as Hugging Face Transformers or vLLM. Start with a limited sample to check task formatting and scoring before spending time on the full benchmark.

• lighteval

  Evaluate language models with configurable tasks, metrics, and model backends.

  huggingface · Local Python runner with supported inference backends.

  Compatibility

  Python, Hugging Face Transformers, vLLM

  Evaluation areas

  Model benchmarks, Custom tasks, Batch evaluation

  Setup notes

  Pick a supported task and metric or define your own task. Save the prompt configuration and per-example predictions so a score change can be traced to individual cases.

  Use a backend such as Transformers or vLLM when testing a local model. Keep the same examples and generation limits when comparing backends.

• evaluate

  Compute reusable evaluation metrics from predictions and references in Python.

  HuggingFace Inc. · Local Python library. Some metric modules or resources are fetched on first use.

  Compatibility

  Python, Hugging Face Datasets

  Evaluation areas

  Metric library, Prediction scoring

  Setup notes

  Load a metric that matches your prediction and reference formats, then calculate it over the same examples for each model. Check the metric card for expected inputs and how to interpret the score.

  This library is useful when predictions already exist. For a full language-model benchmark run with model and task configuration, Hugging Face directs users to Lighteval.

• crfm-helm

  Compare foundation models using standardized scenarios and several kinds of performance measures.

  stanford-crfm · Python evaluation framework. Some model or dataset choices require credentials.

  Compatibility

  Python, Model APIs

  Evaluation areas

  Model benchmarks, Batch evaluation, Comparative reporting

  Setup notes

  Pick a scenario and a model configuration, then retain the run specification with the results. Standardized scenarios make it easier to tell whether a score change comes from the model or a test setup change.

  Review metrics by scenario before relying on a summary score. Accuracy, efficiency, and other measures answer different questions and can move in different directions.

• inspect-ai

  Define evaluation tasks with a dataset, execution steps, and scorers for models or agents.

  UK AI Security Institute · Local Python framework. Model provider and sandbox requirements depend on the task.

  Compatibility

  Python, Model APIs, Docker

  Evaluation areas

  Agent evaluation, Custom tasks, Run logs

  Setup notes

  Write a task using a dataset and solver, then choose a scorer aligned with the behavior you want to measure. Multi-step solvers can exercise a tool-using agent or a longer conversation.

  Keep logs and score individual samples so you can distinguish wrong answers from tool or infrastructure failures. Use an isolated execution environment when the evaluation includes code execution.

• promptfoo/promptfoo

  Test prompts and model configurations against repeatable assertions from a command line or CI job.

  promptfoo · Local command line. The selected provider can be a local Ollama model.

  Compatibility

  Node.js, YAML, Ollama

  Evaluation areas

  Prompt regression tests, Assertions, CI integration

  Setup notes

  Define cases, candidate prompts, providers, and assertions in a test configuration. Start with checks for output format and required fields, then add reference-based or model-graded checks for cases that need judgment.

  Run the same suite after a prompt or model change. Keep failures and representative output samples in the review so a single aggregate pass rate does not hide a regression.

• deepeval

  Write repeatable tests for LLM application behavior, retrieval quality, and output regressions.

  Jeffrey Ip · Local Python tests. Some metrics require a configured judge model.

  Compatibility

  Python, pytest, Model APIs

  Evaluation areas

  Prompt regression tests, RAG evaluation, Model-graded scoring

  Setup notes

  Build test cases from representative requests, application outputs, and any known references or retrieved context. Choose a metric that tests the intended behavior and set a pass threshold from a labeled sample.

  Run tests after changes to a prompt, retrieval setting, or model. Record the judging model and rubric when using a model-based metric, and review borderline cases with a person.

• Ragas

  Evaluate retrieval and generated answers using test datasets and metrics for RAG applications.

  Vibrant Labs · Local Python evaluator. Judge and embedding requirements vary by metric.

  Compatibility

  Python, Model APIs

  Evaluation areas

  RAG evaluation, Test datasets, Model-graded scoring

• trulens

  Trace LLM application runs and evaluate the steps that produced an answer.

  Snowflake Inc. · Local Python instrumentation and evaluation. Scoring providers are configured separately.

  Compatibility

  Python, OpenTelemetry, Model APIs

  Evaluation areas

  RAG evaluation, Agent evaluation, Run logs

  Setup notes

  Instrument an application and record spans for retrieval, generation, or tool use. Attach feedback functions to the pieces of a trace that contain the information they need.

  Review groundedness and relevance scores alongside individual traces. Comparing the same inputs across versions helps identify where retrieval or generation behavior changed.

• Arize-ai/phoenix

  Evaluate an LLM application's traces, retrieved context, and generated responses in a local analysis workspace.

  Arize-ai · Self-hostable evaluation and tracing workspace. Judge model access depends on the selected metric.

  Compatibility

  Python, OpenTelemetry, Docker

  Evaluation areas

  RAG evaluation, Run logs, Test datasets

  Setup notes

  Record a representative group of traces or create a versioned dataset of test examples. Add response or retrieval evaluations to the relevant spans and inspect examples behind the scores.

  Use experiments to compare a model, prompt, or retrieval change on the same dataset. Keep evaluation inputs and judge settings consistent and review cases where metrics disagree.

• comet-ml/opik

  Run evaluation experiments over a dataset and inspect scores alongside application traces.

  comet-ml · Self-hostable workspace with local SDKs. Model-based metrics may need a provider.

  Compatibility

  Python, pytest, Docker

  Evaluation areas

  Test datasets, Run logs, CI integration

  Setup notes

  Create a dataset of representative cases, define the task that runs your application, and attach scoring metrics. Use the experiment record to compare model or prompt versions against the same examples.

  When a score drops, inspect the stored traces and test individual cases. A pytest integration can run repeatable checks on selected cases before changes are merged.

• opencompass

  Run a reproducible model evaluation workflow from dataset and model configuration through scoring and reporting.

  OpenCompass Contributors · Local Python runner for supported local models and API models.

  Compatibility

  Python, Hugging Face Transformers, vLLM

  Evaluation areas

  Model benchmarks, Batch evaluation, Comparative reporting

  Setup notes

  Choose compatible datasets, model backends, and evaluator settings, then save the configuration used to produce the report. A local model can be run through a supported inference backend.

  Inspect task-level results before comparing aggregate scores. Check that prompt templates, model revisions, and dataset splits were equivalent across experiments.

• OpenEvals

  Add reusable scoring functions to existing tests for prompts and LLM application outputs.

  LangChain · Local evaluation libraries. Model-graded scorers need a configured judge.

  Compatibility

  Python, pytest, Vitest

  Evaluation areas

  Assertions, Model-graded scoring, CI integration

• evalscope

  Evaluate models on benchmark tasks and measure inference performance against a defined request workload.

  ModelScope team · Local Python runner. Dataset, backend, and model access depend on the selected task.

  Compatibility

  Python, Model APIs

  Evaluation areas

  Model benchmarks, Multimodal evaluation, Performance testing

  Setup notes

  Choose a benchmark and model interface, then save the execution settings with the task results. The available evaluation paths cover language, multimodal, embedding, and reranking models.

  Run performance tests separately from capability tests. Record concurrency, input length, output length, and the target hardware so latency and throughput figures can be reproduced.

• LMMs-Eval

  Evaluate multimodal models on tasks that use images, video, audio, or text inputs.

  LMMs Lab · Local Python runner with compatible model adapters and task data.

  Compatibility

  Python, Hugging Face Transformers

  Evaluation areas

  Multimodal evaluation, Model benchmarks, Batch evaluation

• MLflow

  Store evaluation inputs, outputs, scores, and traces alongside versioned machine learning experiments.

  MLflow Project · Local Python evaluation and tracking. Optional integrations have separate requirements.

  Compatibility

  Python, Model APIs

  Evaluation areas

  Test datasets, Run logs, Comparative reporting

• TorchMetrics

  Compute machine learning metrics over PyTorch tensors, with state that can be accumulated across batches.

  Lightning AI · Local Python library for PyTorch workloads.

  Compatibility

  Python, PyTorch

  Evaluation areas

  Metric library, Prediction scoring, Batch evaluation