---
name: model-compression-ff82197bda54
description: "Model compression techniques: pruning, knowledge distillation, and quantization. Covers compression ordering (P-KD-Q), tools, and evaluation metrics."
tags: [model-compression, pruning, distillation, quantization, llm, optimization, zorai]
---

## Overview

Model compression reduces LLM size and inference cost through three main techniques: **pruning** (removing parameters), **knowledge distillation** (training smaller student models), and **quantization** (lower precision weights). A 2025 study on Qwen2.5 3B found the optimal ordering: **Pruning → Knowledge Distillation → Quantization** (P-KD-Q) achieves 3.68x compression while preserving instruction-following and language understanding.

## The Three Techniques

| Technique | What it does | Compression | Quality Impact |
|---|---|---|---|
| **Quantization** | Lower precision (FP16→INT4/FP8) | Highest standalone | Low with PTQ, recoverable with QAT |
| **Structured Pruning** | Remove attention heads, layers, neurons | Moderate | Moderate degradation, recoverable with KD |
| **Knowledge Distillation** | Train smaller model on larger model outputs | Architecture-dependent | Best quality recovery post-pruning |

## Compression Ordering (P-KD-Q)

The paper shows ordering matters critically:

1. **Prune first** — removes redundant parameters, creating a smaller base
2. **Distill** — recover quality by training the pruned model on the original model's outputs
3. **Quantize last** — reduces precision on the already-optimized model

Applying quantization early causes irreversible information loss that impairs subsequent training.

## Available Toolkits

| Toolkit | Features | Use Case |
|---|---|---|
| `angelslim` (Tencent) | PTQ/QAT, pruning, Eagle3 speculative decoding | Full compression pipeline, 1.25-bit to FP8 |
| `nvidia-model-optimizer` | PTQ, QAT, pruning, distillation, speculative decoding | NVIDIA ecosystem, TensorRT-LLM deployment |
| `intel-neural-compressor` | INT8/FP8/INT4 quantization, pruning, distillation | Intel hardware, ONNX Runtime |
| `peft` + `unsloth` | QLoRA fine-tuning | Training-time efficiency, adapter-based |

## Evaluation Metrics

- **Perplexity**: language modeling quality
- **G-Eval / Clarity**: instruction-following quality
- **Compression ratio**: original size / compressed size
- **Inference speed**: tokens/second
- **Accuracy on benchmarks**: MMLU, GSM8K, HumanEval

---
Source attribution: mkurman
Original project: https://github.com/mkurman/zorai/blob/d06ee7597049567dab92db22d41b2764cf89094a/skills/scientific-skills/model-compression/SKILL.md
License: MIT
License information: https://opensource.org/license/mit
Original upstream instructions; downloaded from Mega AI Hub.

## Upstream license notice

MIT License

Copyright (c) 2026 zorai contributors

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.



# Mega AI Hub resource metadata

Creator / publisher: mkurman
Original project: https://github.com/mkurman/zorai/blob/d06ee7597049567dab92db22d41b2764cf89094a/skills/scientific-skills/model-compression/SKILL.md
Attribution basis: Source repository record

Fields of work: Data & Analytics, Development & Engineering, Product Management
Roles: Data Scientist, ML / AI Engineer, Product Manager
Specialties: Specialty not specified
Tools: Angelslim, Intel Neural Compressor, Nvidia Model Optimizer, Peft
Country applicability: Country not specified
Classification basis: Source catalog labels
Instruction SHA-256: 89b39c216e2f08fc2a2ae6c70eb8c61f267caf8709c696c83cce289c9d15655e