AI Monitoring

29 resources

• langfuse/langfuse

  An observability platform for tracing model calls, tools, and other steps in AI applications.

  langfuse · Self-hosted deployment is available. Application instrumentation is required.

  Compatibility

  Python, JavaScript, OpenTelemetry, Docker

  Monitoring areas

  LLM tracing, Latency analysis, Evaluation scores

  Setup notes

  Create a trace for each application request and add observations for retrieval, model generation, and tool use. Use those boundaries to identify the step that changed when response time or quality regresses.

  Run a small sample through the self-hosted deployment before connecting production. Check what the tracing library captures and redact sensitive input or output before exporting it.

• Arize-ai/phoenix

  An open-source platform for inspecting AI traces and connecting them with evaluation results.

  Arize-ai · Local installation and self-hosted deployment are available.

  Compatibility

  Python, OpenTelemetry, OpenInference

  Monitoring areas

  LLM tracing, Retrieval inspection, Evaluation

  Setup notes

  Capture a trace around an answer request and mark the retrieval, reranking, and generation spans. Compare the retrieved evidence with the output before attributing a failure to the model.

  Start a local Phoenix instance and send a small set of OpenTelemetry spans. Select a test dataset and repeat the same queries after making a pipeline change.

• OpenLIT

  An OpenTelemetry-based observability stack and instrumentation library for AI applications.

  OpenLIT · Local instrumentation and a self-hosted stack are available.

  Compatibility

  Python, OpenTelemetry, Docker

  Monitoring areas

  LLM instrumentation, Token metrics, OpenTelemetry export

• OpenLLMetry

  An OpenTelemetry instrumentation project for adding traces to calls made by AI frameworks and SDKs.

  Traceloop · Local instrumentation. A compatible collector or tracing backend is needed to inspect spans.

  Compatibility

  Python, OpenTelemetry, OTLP

  Monitoring areas

  LLM instrumentation, Distributed tracing, OpenTelemetry export

• MLflow Tracing

  MLflow’s tracing interface for recording and inspecting steps inside generative AI applications.

  MLflow Project · Local Python library and tracking UI. Remote storage is optional.

  Compatibility

  Python, MLflow Tracking

  Monitoring areas

  LLM tracing, Experiment tracking, Trace-linked evaluation

• evidently

  A Python toolkit for measuring data quality, drift, and model behavior over reference and current data.

  evidentlyai · Local Python library. Reports can be generated locally.

  Compatibility

  Python, Tabular data

  Monitoring areas

  Data drift, Data quality, Model monitoring reports

  Setup notes

  Compare a known reference period with a current sample. Track missing values, feature distributions, and prediction changes so you can distinguish upstream data problems from model behavior.

  Build the report locally and verify what each threshold catches with a known shift. If labels arrive later, add the outcome metrics when that ground truth is available.

• nannyml

  A Python library for monitoring tabular models, including performance estimation when labels arrive late.

  Niels Nuyttens · Local Python library for supported tabular classification or regression monitoring tasks.

  Compatibility

  Python, Tabular data

  Monitoring areas

  Performance estimation, Data drift, Tabular model monitoring

  Setup notes

  Split model outputs into time chunks and fit an estimator with a labeled reference period. Compare estimated production performance with actual performance as labels become available.

  Review covariate and prediction drift alongside the estimate. Validate the assumptions of the selected estimator on a period where you know the true outcomes.

• deepchecks

  A Python testing toolkit for checking machine-learning data and model behavior before and after changes.

  deepchecks · Local Python library. Supported checks vary by data type.

  Compatibility

  Python, Tabular data, Computer vision

  Monitoring areas

  Data validation, Model checks, Distribution comparison

  Setup notes

  Run a suite on reference and candidate datasets to check schema problems, leakage, and distribution changes. Reuse the checks when new training or validation snapshots arrive.

  Set thresholds based on known behavior and inspect failed checks with sample rows or features. Keep each report connected to the data and model versions that produced it.

• frouros

  A Python library with detectors for drift in batch comparisons and streaming data.

  Jaime Céspedes Sisniega · Local Python library.

  Compatibility

  Python, Batch data, Streaming data

  Monitoring areas

  Data drift, Concept drift, Streaming monitoring

  Setup notes

  Choose a batch detector for scheduled snapshot comparisons or a streaming detector for incremental observations. Use a clean baseline and introduce a known change to see how quickly the detector reacts.

  Record detector settings, window size, and the time of each signal. Review alerts against operational changes such as a deployment or a new input source.

• Alibi Detect

  An open-source library for drift and outlier detection in model inputs and related signals.

  Seldon · Local Python library. Dependencies vary by detector and backend.

  Compatibility

  Python, Tabular data, Embeddings

  Monitoring areas

  Drift detection, Outlier detection, Model input monitoring

• OpenInference instrumentation

  OpenTelemetry conventions and instrumentation packages for representing model, retrieval, and tool activity in traces.

  Arize AI · Local instrumentation packages. Use a compatible OpenTelemetry backend for viewing traces.

  Compatibility

  Python, JavaScript, OpenTelemetry, OTLP

  Monitoring areas

  LLM instrumentation, Trace conventions, Framework connectors

• Helicone/helicone

  An observability gateway for tracking AI request latency, errors, usage, and associated metadata.

  Helicone · Self-hosted deployment is available. Model access depends on the application’s configured providers.

  Compatibility

  Docker, HTTP APIs, Model gateways

  Monitoring areas

  AI request logging, Latency monitoring, Usage monitoring

  Setup notes

  Route a small workload through the gateway and attach a request or session identifier. Use those identifiers to compare error and latency patterns across application paths.

  For a self-hosted deployment, verify request routing and retention settings in a test environment. Decide what payload data should be captured and use bounded labels for dashboards.

• Agenta-AI/agenta

  An open-source platform for tracing and evaluating language-model applications.

  Agenta-AI · Self-hosted deployment is available.

  Compatibility

  Python, OpenTelemetry, Docker

  Monitoring areas

  LLM tracing, Prompt evaluation, Version comparisons

  Setup notes

  Instrument application steps to see where errors occur and tie trace samples to a particular prompt or application version. Inspect tool arguments and retrieved context when the output changes unexpectedly.

  Use a small fixed test set to compare versions before rollout. Run the self-hosted stack if evaluation and trace data need to stay in your environment.

• comet-ml/opik

  An open-source platform for tracing AI applications and evaluating outputs against a test set.

  comet-ml · Local and self-hosted options are available.

  Compatibility

  Python, JavaScript, Docker

  Monitoring areas

  LLM tracing, Dataset evaluations, Trace inspection

  Setup notes

  Capture spans around model and tool calls and record the evaluation score alongside each test example. Use the trace to inspect why a result changed before deciding whether the change is acceptable.

  Use a local deployment to inspect a prototype, then select a set of repeated checks for prompt or model updates. Retain the input set and configuration with each result.

• TensorBoard

  A local dashboard for visualizing training metrics, experiment comparisons, and model diagnostics.

  TensorFlow Project · Local Python package and dashboard.

  Compatibility

  Python, TensorFlow, PyTorch

  Monitoring areas

  Training monitoring, Scalar metrics, Experiment comparison

• NVIDIA DCGM Exporter

  A Prometheus exporter for collecting NVIDIA GPU telemetry from supported hosts and clusters.

  NVIDIA · Local exporter. Requires supported NVIDIA hardware and drivers.

  Compatibility

  NVIDIA GPUs, Prometheus, Docker, Kubernetes

  Monitoring areas

  GPU telemetry, Training infrastructure metrics, Prometheus export

• river

  Streaming drift detectors in River for monitoring observations or online prediction errors.

  online-ml · Local Python library.

  Compatibility

  Python, Streaming data

  Monitoring areas

  Online drift detection, Streaming monitoring, Prediction error monitoring

  Setup notes

  Feed one observation at a time to a detector and record the time of each drift signal. A prediction-error stream needs labels, which may arrive after the model request.

  Compare false alerts on a stable replay with detection delay on a replay containing a known change. Save the detector parameters alongside the replay results.

• OpenInference instrumentation for LangChain

  An OpenTelemetry instrumentor for following LangChain application steps in a trace.

  Arize AI · Local Python instrumentation. A compatible OpenTelemetry receiver is needed to view traces.

  Compatibility

  Python, LangChain, OpenTelemetry

  Monitoring areas

  Chain tracing, Retrieval spans

• OpenInference instrumentation for LlamaIndex

  An instrumentation package for tracing LlamaIndex retrieval and query execution.

  Arize AI · Local Python instrumentation. A compatible OpenTelemetry receiver is needed to view traces.

  Compatibility

  Python, LlamaIndex, OpenTelemetry

  Monitoring areas

  RAG tracing, Query engine monitoring

• OpenInference instrumentation for DSPy

  An instrumentation package for recording traces from DSPy modules and model calls.

  Arize AI · Local Python instrumentation. A compatible OpenTelemetry receiver is needed to view traces.

  Compatibility

  Python, DSPy, OpenTelemetry

  Monitoring areas

  Program tracing, Module latency

• OpenInference instrumentation for MCP

  An instrumentation package for following Model Context Protocol activity in application traces.

  Arize AI · Local Python instrumentation. A compatible OpenTelemetry receiver is needed to view traces.

  Compatibility

  Python, MCP, OpenTelemetry

  Monitoring areas

  Tool protocol tracing, MCP monitoring

• OpenInference instrumentation for Ollama

  An instrumentation package for observing calls made through the Ollama Python client.

  Arize AI · Local Python instrumentation. A compatible OpenTelemetry receiver is needed to view traces.

  Compatibility

  Python, Ollama, OpenTelemetry

  Monitoring areas

  Local model tracing, Request latency

• OpenInference instrumentation for Pydantic AI

  An instrumentation package for recording the activity of Pydantic AI applications.

  Arize AI · Local Python instrumentation. A compatible OpenTelemetry receiver is needed to view traces.

  Compatibility

  Python, Pydantic AI, OpenTelemetry

  Monitoring areas

  Agent tracing, Tool-call monitoring

• OpenInference instrumentation for Pipecat

  An instrumentation package for observing Pipecat applications that process conversational audio or video.

  Arize AI · Local Python instrumentation. A compatible OpenTelemetry receiver is needed to view traces.

  Compatibility

  Python, Pipecat, OpenTelemetry

  Monitoring areas

  Conversational pipeline tracing, Voice application timing

• OpenInference instrumentation for smolagents

  An instrumentation package for tracing runs of Hugging Face smolagents applications.

  Arize AI · Local Python instrumentation. A compatible OpenTelemetry receiver is needed to view traces.

  Compatibility

  Python, smolagents, OpenTelemetry

  Monitoring areas

  Agent tracing, Tool execution tracing

• OpenLLMetry instrumentation for ChromaDB

  An OpenTelemetry instrumentor for ChromaDB operations used in retrieval pipelines.

  Traceloop · Local Python instrumentation. A compatible OpenTelemetry receiver is needed to view traces.

  Compatibility

  Python, ChromaDB, OpenTelemetry

  Monitoring areas

  Vector-store tracing, Retrieval latency

• OpenLLMetry instrumentation for Qdrant

  An OpenTelemetry instrumentor for calls made through the Qdrant Python client.

  Traceloop · Local Python instrumentation. A compatible OpenTelemetry receiver is needed to view traces.

  Compatibility

  Python, Qdrant, OpenTelemetry

  Monitoring areas

  Vector-client tracing, Search request monitoring

• OpenLLMetry instrumentation for LiteLLM

  An OpenTelemetry instrumentor for observing calls from applications that use LiteLLM.

  Traceloop · Local Python instrumentation. A compatible OpenTelemetry receiver is needed to view traces.

  Compatibility

  Python, LiteLLM, OpenTelemetry

  Monitoring areas

  Model-call tracing, Application error monitoring

• OpenLLMetry instrumentation for Transformers

  An OpenTelemetry instrumentor for supported Hugging Face Transformers calls.

  Traceloop · Local Python instrumentation. A compatible OpenTelemetry receiver is needed to view traces.

  Compatibility

  Python, Transformers, OpenTelemetry

  Monitoring areas

  Local model tracing, Model workload timing