Higher performance OpenAI LLM service than vLLM serve: A pure C++ high-performance OpenAI LLM service implemented with GPRS+TensorRT-LLM+Tokenizers.cpp, supporting chat and function call, AI agents, distributed multi-GPU inference, multimodal capabilities, and a Gradio chat interface.
Experimental MoE expert pruning and router analysis for safetensors checkpoints. CPU probing, keep plans, dry runs, and checkpoint rewriting with synthetic tests.
Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning (COLM 2026): ExpertCondenser: SFT for sparse MoE LLMs with gated condenser experts. Official code.