vLLM
Run supported language models for offline batch generation or behind a locally operated inference server.
Published by vLLM Project
How to use it
Use vLLM's local Python runner for batch jobs, or start its server on a machine you control. Choose a supported model architecture and use the installation instructions for your hardware backend.
Benchmark with the context lengths, output lengths, and concurrency you plan to support. Watch memory usage as context and request concurrency increase.
- Hardware
- NVIDIA CUDA, AMD ROCm
- Model formats
- Hugging Face checkpoint, Safetensors
- Operating systems
- Linux
- Access
- Runs locally. Install the build for your hardware backend.