Local Inference Engines

17 resources

• ONNX Runtime

  Run exported ONNX models with a common inference interface and hardware-specific execution providers.

  Microsoft · Runs locally. Install a build for the intended execution provider.

  Hardware

  CPU, NVIDIA CUDA, Apple Core ML, DirectML

  Model formats

  ONNX, ORT model

  Operating systems

  Linux, Windows, macOS, Android, iOS

• llama.cpp

  Run compatible language and multimodal models locally from GGUF files with a native command line or server.

  ggml.org · Runs locally. Supply a compatible GGUF model.

  Hardware

  CPU, NVIDIA CUDA, AMD HIP, Apple Metal, Vulkan

  Model formats

  GGUF

  Operating systems

  Linux, Windows, macOS

• OpenVINO Runtime

  Run compatible vision, language, and other neural networks on local Intel hardware.

  Intel · Runs locally. Install the runtime and drivers appropriate to the device.

  Hardware

  CPU, Intel GPU, Intel NPU

  Model formats

  OpenVINO IR, ONNX

  Operating systems

  Linux, Windows

• TensorRT

  Optimize supported networks and execute the resulting inference engines on NVIDIA GPUs.

  NVIDIA · Runs locally on a supported NVIDIA GPU with compatible drivers.

  Hardware

  NVIDIA CUDA

  Model formats

  ONNX, TensorRT plan

  Operating systems

  Linux, Windows

• tensorrt-llm

  Run supported large language models on NVIDIA GPUs with local generation and serving runtimes.

  NVIDIA Corporation · Runs locally on supported NVIDIA GPUs.

  Hardware

  NVIDIA CUDA

  Model formats

  TensorRT-LLM engine, Hugging Face checkpoint

  Operating systems

  Linux

  Setup notes

  Use TensorRT-LLM when a model and GPU are on the supported list and you need control over generation settings, batching, or multi-GPU execution. Follow the model-specific setup for the selected backend.

  Compare the runtime's outputs against a reference before enabling quantization or changing parallelism. Measure time to the first token and total generation speed with prompt sizes similar to your workload.

• vLLM

  Run supported language models for offline batch generation or behind a locally operated inference server.

  vLLM Project · Runs locally. Install the build for your hardware backend.

  Hardware

  NVIDIA CUDA, AMD ROCm

  Model formats

  Hugging Face checkpoint, Safetensors

  Operating systems

  Linux

• sglang

  Run supported language and vision-language models on local GPU hardware.

  sgl-project · Runs locally. Driver, model, and backend support varies by accelerator.

  Hardware

  NVIDIA CUDA, AMD ROCm

  Model formats

  Hugging Face checkpoint, Safetensors

  Operating systems

  Linux

  Setup notes

  Use the runtime to load a supported model checkpoint and run generation behind your own server. Start with the recommended backend for your GPU and test representative prompts before enabling additional optimizations.

  Prefix reuse and batching can help repeated or concurrent requests. Benchmark those features with your own workload and check GPU memory and latency under the planned request load.

• executorch

  Run exported PyTorch models inside mobile and other resource-constrained applications.

  pytorch · Runs on device. The application supplies a compiled runtime and exported model.

  Hardware

  CPU, Apple Core ML, Qualcomm QNN

  Model formats

  ExecuTorch PTE

  Operating systems

  Android, iOS, Linux

  Setup notes

  Export a supported PyTorch model, lower it for the intended backend, and package the resulting PTE program with your application. Keep a simple reference example so you can compare the runtime output with the original model.

  Choose backend delegates for each target build. A model that runs on the CPU may need a different export or operator coverage review before it runs on a phone accelerator.

• LiteRT

  Run TensorFlow Lite models on phones and other edge devices with a lightweight inference runtime.

  Google AI Edge · Runs on device. Package the runtime and the TFLite model with the application.

  Hardware

  CPU, Mobile GPU delegate

  Model formats

  TensorFlow Lite

  Operating systems

  Android, iOS, Linux

• ncnn

  Run converted neural networks in native desktop and mobile applications with CPU or Vulkan execution.

  nihui · Runs locally. Build or install ncnn and supply a converted model.

  Hardware

  CPU, Vulkan

  Model formats

  ncnn param and bin

  Operating systems

  Android, iOS, Linux, Windows, macOS

  Setup notes

  Convert a supported model with the project's conversion tools and load the resulting parameter and weight files into an ncnn application. Validate image preprocessing and output shapes against the original model.

  A Vulkan build can accelerate supported layers on compatible devices. Test the CPU build as a baseline and use a representative device to measure memory and startup costs.

• MNN

  Run converted neural networks in local mobile and desktop applications using the MNN runtime.

  Alibaba · Runs locally. Supply a compatible MNN model and target-specific runtime build.

  Hardware

  CPU, OpenCL, Vulkan, Apple Metal

  Model formats

  MNN

  Operating systems

  Android, iOS, Linux, Windows, macOS

• MLC LLM

  Deploy compatible language models locally using a runtime and compiled model artifacts for the target device.

  MLC AI · Runs locally with target-specific compiled model artifacts.

  Hardware

  NVIDIA CUDA, Vulkan, Apple Metal

  Model formats

  MLC weights, MLC compiled model library

  Operating systems

  Linux, Windows, macOS, Android, iOS

• ctranslate2

  Run compatible Transformer models for translation, speech transcription, and text generation in a native runtime.

  OpenNMT · Runs locally. Supply a converted CTranslate2 model.

  Hardware

  CPU, NVIDIA CUDA

  Model formats

  CTranslate2 model

  Operating systems

  Linux, Windows, macOS

  Setup notes

  Convert a supported model to the CTranslate2 format, or use an existing compatible conversion. Load it with the interface for the selected model family and use the matching tokenizer or preprocessing.

  Choose a supported compute type on the target CPU or NVIDIA GPU. Compare quality and memory before using lower precision, especially for speech and translation workflows.

• mlx-lm

  Generate text with supported language models on Apple silicon through MLX.

  ml-explore · Runs locally on Apple silicon.

  Hardware

  Apple silicon, Apple Metal

  Model formats

  Hugging Face checkpoint, MLX Safetensors

  Operating systems

  macOS

  Setup notes

  Install the local Python package and load a supported checkpoint, including a compatible MLX conversion. Use the command-line generator for a quick test before connecting the runtime to an application.

  Pick a quantization and context length that fit your Mac's unified memory. Benchmark using typical prompts and check the resulting output quality before committing to a model.

• ollama

  Download, configure, and run compatible language models through a local command line and service.

  ollama · Runs locally with downloaded or imported compatible model weights.

  Hardware

  CPU, NVIDIA CUDA, AMD ROCm, Apple Metal

  Model formats

  Ollama model, GGUF

  Operating systems

  Linux, Windows, macOS

  Setup notes

  Start the local Ollama service, then select a compatible model for its local runner. A Modelfile can define configuration and import a supported GGUF file.

  Match the model size and context setting to available memory. Select a local model for offline inference after its files are downloaded, and verify that any application is connected to your local service.

• whisper.cpp

  Transcribe speech locally with compatible Whisper model weights and a native inference engine.

  ggml.org · Runs locally. Supply compatible Whisper weights and local audio.

  Hardware

  CPU, NVIDIA CUDA, Apple Metal, Apple Core ML, Vulkan

  Model formats

  Whisper GGML weights

  Operating systems

  Linux, Windows, macOS, Android, iOS

• tensorflow/serving

  Serve versioned TensorFlow SavedModel exports from a local inference process or container.

  tensorflow · Runs locally through the serving binary or a compatible container.

  Hardware

  CPU, NVIDIA CUDA

  Model formats

  TensorFlow SavedModel

  Operating systems

  Linux

  Setup notes

  Export a TensorFlow model with a serving signature and put the export in a versioned model directory. Run the serving binary or container on a machine you control and verify a prediction against the original export.

  Test how new model versions load and how the serving signature handles input validation before replacing a running model. CPU and GPU images have different runtime dependencies.