Model compression techniques: quantization, distillation, pruning, and sparsity
Model compression techniques are a suite of methods-quantization (reducing numerical precision), distillation (training a smaller model to mimic a larger one), pruning (removing redundant weights/neurons), and sparsity (inducing and leveraging zero weights)-designed to reduce the computational, memory, and storage footprint of large neural networks without proportional loss in task performance.
This skill is highly valued because it directly reduces the total cost of ownership (TCO) for deploying AI models at scale, lowering cloud compute bills and enabling inference on resource-constrained edge devices. Mastering compression techniques accelerates time-to-market for production AI and is critical for building competitive, cost-effective AI products.
1Careers
1Categories
9.0 Avg Demand
15%
Avg AI Risk
How to Learn Model compression techniques: quantization, distillation, pruning, and sparsity
Focus on: 1) Understanding the fundamental compression taxonomy (quantization vs. pruning vs. distillation) and the core trade-off between model size/speed and accuracy. 2) Grasping the basic math behind post-training quantization (PTQ) and its primary limitation: calibration dataset dependency. 3) Familiarizing yourself with the output artifacts: ONNX, TorchScript, and core inference engines like TensorRT and ONNX Runtime.
Move to practice by: 1) Implementing quantization-aware training (QAT) instead of just PTQ to understand gradient estimation through the quantization function (Straight-Through Estimator). 2) Applying structured pruning (entire channels/heads) and comparing its hardware efficiency to unstructured (weight-level) pruning. 3) Executing a knowledge distillation pipeline using KL-divergence loss between a teacher and student logits, a common mistake is ignoring temperature scaling in the softmax.
Master the domain by: 1) Architecting mixed-precision pipelines (e.g., FP16/INT8) with sensitivity analysis per layer, using frameworks like NVIDIA's AMP. 2) Designing a hybrid compression strategy (e.g., distill -> quantize -> prune) and analyzing the cascading impact on accuracy vs. latency. 3) Optimizing for specific hardware backends (GPUs with INT8 Tensor Cores, ARM NPUs) by aligning sparsity patterns (e.g., 2:4) with hardware capabilities, and mentoring teams on avoiding common pitfalls in model retraining for compression.
Practice Projects
Beginner
Project
Post-Training Quantization Pipeline
Scenario
You have a pre-trained ResNet-50 model (FP32) from the torchvision library for an image classification task. Your goal is to reduce its size and latency for CPU deployment using Post-Training Quantization (PTQ).
How to Execute
1) Load the pre-trained FP32 model and a calibration dataset (e.g., a subset of ImageNet val). 2) Use PyTorch's `torch.quantization.quantize_dynamic` for a simple approach or implement a custom calibration loop to compute activation ranges. 3) Serialize the quantized model to TorchScript. 4) Benchmark the FP32 vs. INT8 model on CPU using `torch.utils.benchmark.Timer` to measure latency and calculate size reduction.
Intermediate
Project
Knowledge Distillation for Task-Specific Model
Scenario
Deploy a large BERT-Large teacher model for a text classification task. Your objective is to train a smaller, faster BERT-Base student model that retains high accuracy while drastically reducing inference cost.
How to Execute
1) Prepare your labeled dataset and fine-tune the BERT-Large teacher model on the target task. 2) Implement a distillation training loop for the BERT-Base student. The loss function must combine: a) standard cross-entropy on ground truth labels, and b) KL-divergence between the softmax outputs of teacher and student, scaled by a temperature parameter (e.g., T=3). 3) Experiment with different loss weighting (alpha) between the two losses. 4) Evaluate the student's accuracy, F1 score, and inference throughput compared to the teacher.
Advanced
Project
Hybrid Compression Pipeline for Edge Deployment
Scenario
You must compress a modern transformer-based model (e.g., a Vision Transformer) to run real-time (30+ FPS) on an NVIDIA Jetson Orin (an edge GPU) for a robotics application, with strict power constraints.
How to Execute
1) Perform structured pruning (e.g., using movement pruning or L1-norm based channel pruning) to remove 30-40% of the model's parameters, focusing on entire attention heads and FFN layers for hardware-friendly sparsity. 2) Apply Quantization-Aware Training (QAT) to the pruned model to calibrate for INT8, using PyTorch's QAT modules and a Straight-Through Estimator. 3) Export the final model to ONNX and optimize it with TensorRT, specifying the target platform (Jetson) and enabling layer fusion and kernel auto-tuning. 4) Validate accuracy on a test set and measure real-time latency and power consumption on the Jetson device, iterating on pruning/QAT if latency targets are missed.
Tools & Frameworks
Software & Platforms (Core Frameworks)
PyTorch (torch.quantization, torch.nn.utils.prune)TensorFlow Model Optimization Toolkit (tfmot)ONNX Runtime (with quantization and graph optimizations)NVIDIA TensorRT (for advanced INT8/FP16 optimization)
Use PyTorch/TensorFlow for implementing and training compressed models (QAT, PTQ, pruning). ONNX Runtime and TensorRT are essential for final deployment, providing platform-specific optimizations, layer fusion, and efficient kernel execution on target hardware (CPU, GPU, edge).
Leverage these for domain-specific or hardware-specific compression. Hugging Face Optimum simplifies transformer model compression workflows. Intel's toolkit is critical for maximizing performance on Intel CPUs. TVM provides a compiler stack for automatic optimization and kernel generation for diverse hardware.
Mandatory for measuring the true impact of compression. Use these to profile latency, memory footprint, FLOPs, and accuracy before and after compression. W&B is crucial for experiment tracking when iteratively tuning pruning sparsity or distillation temperature.
Interview Questions
Answer Strategy
Frame the answer around accuracy/effort trade-off and the 'why' behind QAT's superiority. Key points: PTQ is fast but accuracy-sensitive models (transformers) often degrade; QAT simulates quantization during training, recovering accuracy. The core technical challenge in QAT is accurate gradient estimation for the non-differentiable quantization function, requiring techniques like the Straight-Through Estimator (STE). For transformers, layer-wise sensitivity analysis is critical.
Answer Strategy
The interviewer is testing diagnostic methodology and knowledge of modern LLM compression techniques. Strategy: Diagnose by evaluating perplexity on specific problematic vs. general datasets, and analyze the pruning mask (is it removing critical attention heads?). Address by: 1) Switching to a more structured pruning method that preserves functional subnetworks, 2) Implementing a distillation step post-pruning to recover lost knowledge, 3) Using gradual pruning with a learning rate warm-up schedule to allow the model to adapt. Reference techniques like SparseGPT or Wanda for LLM-specific pruning.
Careers That Require Model compression techniques: quantization, distillation, pruning, and sparsity