Research paper
Evaluating Quantized Large Language Models
Post-training quantization (PTQ) has emerged as a promising technique to reduce the cost of large language models (LLMs). Specifically, PTQ can effectively mitigate memory consumption and reduce computational overhead in LLMs. To meet the requirements of both high efficiency and performance across diverse scenarios, a comprehensive evaluation of quantized LLMs is essential to guide the selection of quantization methods. This paper presents a thorough evaluation of these factors by evaluating the effect of PTQ on Weight, Activation, and KV Cache on 11 model families, including OPT, LLaMA2, Falcon, Bloomz, Mistral, ChatGLM, Vicuna, LongChat, St
Li, Shiyao, Ning, Xuefei, Wang, Luning, Liu, Tengxuan, Shi, Xiangsheng, Yan, Shengen, Dai, Guohao, Yang, Huazhong, Wang, Yu
arXiv.org perpetual, non-exclusive licenseView resource