llama.cpp
Run compatible language and multimodal models locally from GGUF files with a native command line or server.
Published by ggml.org
How to use it
Choose a GGUF model whose architecture the runtime supports. Use the command-line runner for a first prompt and the local server when an application needs an HTTP endpoint on your machine.
Select a release build or compile for your accelerator. Quantization, context length, and GPU layer offloading change memory use. Record those settings and check whether the model fits before comparing response speed.
- Hardware
- CPU, NVIDIA CUDA, AMD HIP, Apple Metal, Vulkan
- Model formats
- GGUF
- Operating systems
- Linux, Windows, macOS
- Access
- Runs locally. Supply a compatible GGUF model.